A double-blind, crossover, non-inferiority randomised controlled trial where primary care providers and patients compare human-generated and AI-generated digital health messages: the AI-CARE study protocol
Clinical Snapshot
PICO Framework
| P — Population | Primary care providers (n=24) and patients (n=24) evaluating digital health messages |
| I — Intervention | Digital health messages generated by GenAI (ChatGPT) |
| C — Comparator | Digital health messages created by primary care and public health experts |
| O — Outcomes | Primary: Clarity and Understandability score; Secondary: Quality of information, adaptation to target audience, relevance and usefulness, readiness to share with patients |
Bottom Line
The AI-CARE study protocol describes a methodologically sound approach to evaluating whether AI-generated health messages are non-inferior to expert-created content. The double-blind, crossover design with comprehensive evaluation criteria represents a rigorous framework for assessing AI utility in patient education. However, as a protocol paper, actual effectiveness cannot be determined. Key limitations include undefined non-inferiority margins and limited consideration of AI-related risks. The study addresses a clinically relevant question given increasing AI adoption in healthcare and the need for efficient, high-quality patient education materials. Results will inform evidence-based decisions about integrating AI tools into primary care practice, though clinicians should await actual outcomes before implementation. The Canadian primary care setting enhances relevance to similar healthcare systems globally, including Australia's primary care landscape.
Key Findings
P Value: Not yet available - protocol stage
Effect Size: Not yet available - protocol stage
Primary Outcome: Clarity and Understandability score (results pending)
Nnt Or Sensitivity: Non-inferiority margin not specified in protocol
Confidence Interval: Not yet available - protocol stage
Clinical Application
High feasibility given existing digital health infrastructure and GenAI accessibility Highly relevant to Australian primary care facing similar workforce pressures and health misinformation challenges; aligns with RACGP digital health initiatives and TGA considerations for AI in healthcare Primary care providers and patients requiring health education materials
Abstract
INTRODUCTION: Primary care is facing multiple crises, including an increase in health misinformation. Digital health messaging by primary care providers has been shown to reach a diverse patient population. With the uptake of Generative Artificial Intelligence (GenAI) usage in healthcare, there is an important opportunity to rapidly create messages that are tailored to different populations and conditions. However, thoroughly assessing artificial intelligence (AI)-generated content is essential, as GenAI raises concerns regarding its accuracy, understandability, actionability and bias perpetuation. We aim to investigate whether digital health messages created by GenAI are evaluated as non-inferior compared with those created by human experts. METHODS AND ANALYSIS: The AI-CARE (AI to Create Accessible and Reliable patient Education materials) study is a double-blind, crossover, non-inferiority randomised controlled trial. Data collection began on 30 May 2025, and is expected to be completed at the end of May 2026. Over 12 months, 192 messages on 48 topics will be written: half by primary care and public health experts and half by a GenAI tool (OpenAI's ChatGPT). Review Panels composed of 24 primary care providers and 24 patients will evaluate these messages using an Evaluation Grid developed to assess the messages' quality of information, adaptation to the target audience, relevance and usefulness, and readiness to be shared with patients. Evaluations will be completed via online REDCap (Research Electronic Data Capture) surveys and the order in which the 192 messages appear will be randomised and will vary between individuals. Participants and analysts will be blinded to the generation source. The primary outcome will be the Clarity and Understandability score. ETHICS AND DISSEMINATION: The Research Ethics Boards of the Hôpital Montfort (24-25-11-038) and the University of Ottawa (S-12-24-11153) formally approved this study in December 2024. Reported data will be grouped and anonymised for dissemination in peer-reviewed scientific journals and conferences. TRIAL REGISTRATION NUMBER: NCT06997107.
References
- 1.Lemieux, A., Kutcher, S. A., Galani Tietcheu, B. R., Seitz, G., Trickovic, J., Archibald, D., Grosjean, S., Hogg, W., & Johnston, S. (2026). A double-blind, crossover, non-inferiority randomised controlled trial where primary care providers and patients compare human-generated and AI-generated digital health messages: the AI-CARE study protocol. BMJ Open, 16(4), e115673. https://doi.org/10.1136/bmjopen-2025-115673
Related Research
The Journal of rheumatology
Examining the Role of Wearables in Inflammatory Arthritis Care: A Narrative Literature Review
3 Aug 2026
Journal of medical Internet research
Advancing Human-Centered AI in Clinical Decision Support: Sociocognitive Human-in-the-Loop Study in HIV Care
2 Aug 2026
Journal of psychiatric research
Systematic review of machine learning and deep learning models for EEG-based detection of depression.
2 Aug 2026
This content is for educational purposes for healthcare professionals only and does not constitute clinical advice. Clinical decisions should be based on individual patient assessment, current guidelines, and appropriate specialist consultation. Editorial Standards · Privacy Policy · Terms of Service