Research Appraisals
Evidence-based critical appraisals of the latest medical research, systematically evaluated using Oxford CEBM methodology.
Showing 14 appraisals
Acta obstetricia et gynecologica Scandinavica
ChatGPT in urogynecology: Comparing large language model responses to human experts
INTRODUCTION: Large language models (LLMs) are increasingly used in healthcare, including urogynecology, where stigma may limit open discussion. LLM-based chat platforms may provide a less intimidating and more accessible way for patients to obtain information, but their reliability requires evaluation. This study compared the quality of ChatGPT-generated responses in urogynecology with those provided by a consultant urogynecologist, focusing on understandability, helpfulness, and reassurance. MATERIAL AND METHODS: A cross-sectional survey was conducted among urogynecology patients. After informed consent, participants reviewed responses to six common questions, each answered by ChatGPT and a single consultant. A blinded third-party consultant verified clinical accuracy. Patients rated responses using a 5-point Likert scale across three domains (maximum score 15 per response). Wilcoxon signed-rank tests were used for comparison. RESULTS: A total of 203 patients participated (median age 56 years, interquartile range 46-66). ChatGPT responses received higher total ratings than consultant responses (76 [67-85] vs. 72 [63-80], p < 0.01). Scores were higher for understandability, helpfulness, and reassurance (all p < 0.01). ChatGPT was preferred in four of six questions, one showed no difference, and one favored the consultant. Subgroup analyses showed no significant variation based on patient characteristics. CONCLUSIONS: In this exploratory study, women rated ChatGPT's responses as clearer and more reassuring than consultant answers. These findings reflect patient perceptions in a limited setting and should be interpreted with caution. While LLMs may have a supportive role in patient education, their use must remain secondary to expert clinical care and subject to careful oversight.
3 Aug 2026
Read appraisal →Journal of medical Internet research
Trusting Generative AI for Health Advice: Preregistered Survey Experiment
BACKGROUND: Generative artificial intelligence (AI) systems are increasingly used for health information seeking, yet it remains unclear how the public evaluates AI-generated health advice relative to guidance from credentialed clinicians in digital environments. Understanding the conditions under which AI is perceived as credible is critical as these systems become integrated into digital health ecosystems. OBJECTIVE: This study examined how source type (a human nurse in an online portal, a health care-specialized generative artificial intelligence system framed as an "AI nurse," or ChatGPT [OpenAI], a general-purpose chatbot), message characteristics, contextual risk, values framing, and individual differences in medical skepticism and experience with AI shape credibility evaluations of the provided advice and its purported source. METHODS: In a preregistered online experiment, a national sample of US adult participants (N=1502) was randomly assigned to 1 of 3 source conditions and evaluated health advice across 3 scenarios: low risk (dietary advice for cholesterol), high risk (chest pain triage), and a morally sensitive scenario (egg freezing). Advice type (intuitive vs counterintuitive) was manipulated in the risk scenarios, and ideological framing (neutral, conservative-leaning, and liberal-leaning) was manipulated in the morally sensitive scenario. Primary outcomes included participants' perceived credibility of the advice and beliefs about whether the patient should follow it. Source-level perceptions of competence and benevolence were also assessed. Medical skepticism and prior AI experience were examined as moderators. RESULTS: Advice attributed to a human nurse was rated as more credible than advice attributed to either AI source. Message intuitiveness showed effects comparable to, and sometimes larger than, the effects of source: intuitive advice was perceived as more credible than counterintuitive advice, with this difference amplified in high-risk contexts. In the morally sensitive scenario, ideological framing influenced perceived bias but did not interact significantly with source. Medical skepticism moderated source evaluations: higher skepticism was associated with greater perceived competence of the AI nurse and lower perceived competence of the human nurse. CONCLUSIONS: Generative AI is evaluated within existing credibility frameworks rather than dismissed outright as inferior to human expertise. While licensed clinicians retain a credibility advantage, AI-generated advice is generally perceived as competent and legitimate. Importantly, individuals skeptical of traditional medical authority may evaluate AI-based guidance more favorably, suggesting that AI systems may redistribute-rather than uniformly erode-trust in health advice. As AI tools become embedded in patient-facing health platforms, message design and audience characteristics may shape acceptance more strongly than source labeling alone.
24 June 2026
Read appraisal →BMJ open
Self-management experiences and needs in patients with chronic obstructive pulmonary disease: a meta-synthesis of qualitative studies
OBJECTIVES: To systematically review qualitative studies on the self-management experiences and needs of patients with chronic obstructive pulmonary disease (COPD) and to provide evidence for developing needs-based self-management intervention strategies. DESIGN: A systematic review and meta-aggregation of qualitative studies conducted in accordance with the Enhancing Transparency in Reporting the Synthesis of Qualitative Research statement. DATA SOURCES: PubMed, Cochrane Library, Web of Science, Scopus, CINAHL, Embase, CNKI, Wanfang Database, VIP Database and CBM were searched from database inception to March 2025 for qualitative studies on COPD self-management experiences and needs. ELIGIBILITY CRITERIA: Qualitative studies involving adults (≥18 years) with a confirmed diagnosis of COPD that reported patients' real feelings, experiences and needs during the self-management process in any setting (hospital, community or home). Studies were excluded if the full text could not be obtained or data were incomplete, if they were duplicate publications or if they were not published in Chinese or English. DATA EXTRACTION AND SYNTHESIS: Two reviewers independently screened titles, abstracts and full texts, extracted data using a standardised form and assessed methodological quality using the 2016 Joanna Briggs Institute (JBI) Critical Appraisal Checklist for Qualitative Research. Findings (participants' quotations and authors' interpretive statements) were synthesised using a JBI meta-aggregation approach to develop categories and overarching synthesised findings. RESULTS: 15 qualitative studies were included, comprising 251 patients with COPD. A total of 52 findings were identified, grouped into 13 categories and synthesised into 4 overarching findings: (1) physical symptom burden strongly shapes psychological well-being during COPD self-management; (2) patients face multiple and interrelated barriers to effective self-management; (3) some patients develop active coping strategies and self-management resources; and (4) patients express multidimensional and continuing support needs. CONCLUSIONS: Patients with COPD experience substantial physical and psychological distress and face multiple challenges in the self-management process. Healthcare professionals should prioritise patients' lived experiences, establish comprehensive and coordinated support systems and develop diversified, needs-based intervention strategies to address patients' multidimensional needs, thereby enhancing self-management capacity and improving quality of life. PROSPERO REGISTRATION NUMBER: CRD42024607051.
29 May 2026
Read appraisal →JMIR perioperative medicine
AI-Generated Avatar Videos for Postoperative Patient Education Among Health Care Workers: Pilot Randomized Controlled Trial
BACKGROUND: Effective postoperative communication is vital for patient recovery, yet traditional text-based discharge instructions often lead to poor comprehension and adherence, particularly among patients with limited health literacy. Although educational videos improve understanding and retention, their widespread use has been hampered by high production costs. Generative artificial intelligence (AI) offers a scalable solution for creating engaging video content. OBJECTIVE: The primary objective of this pilot study was to assess the feasibility of creating and deploying AI-generated, avatar-led videos for postoperative instruction delivery. Secondary objectives included comparing knowledge retention, engagement, perceived clarity, and user experience between AI-generated video and traditional text-based handout formats among health care workers. METHODS: In this randomized pilot study, 38 health care worker volunteers were recruited as a convenience sample to pilot-test the intervention before patient implementation. Participants were assigned to either a text handout group (n=19, 50%) or an AI-generated video group (n=19, 50%). Both groups received information on 10 common postoperative topics. The primary outcome was objective knowledge, assessed via a 10-item quiz. Secondary outcomes, measured through surveys with 5-point Likert scales, included engagement time, subjective engagement, perceived clarity, usefulness, confidence in understanding, and information retention. Qualitative feedback was also collected. RESULTS: Objective knowledge quiz scores did not differ significantly between groups (mean 8.89, SD 1.20 for the AI-generated video group vs mean 8.21, SD 1.78 for the text handout group; P=.17; Cohen d=0.45). Participants in the AI-generated video group demonstrated significantly higher engagement time (mean 15.11, SD 7.78 minutes vs mean 8.84, SD 4.03 minutes; P=.004; Cohen d=1.04). They also rated instructions as significantly clearer (mean 4.63, SD 0.50 vs mean 4.00, SD 0.82; P=.007; Cohen d=0.93), more engaging (mean 4.05, SD 0.78 vs mean 3.32, SD 1.00; P=.02; Cohen d=0.81), and more effective for retention (mean 4.42, SD 0.84 vs mean 3.37, SD 0.68; P<.001; Cohen d=1.38). Qualitative feedback highlighted the engaging nature of AI-generated videos but noted areas for avatar refinement. CONCLUSIONS: In this pilot study with health care workers, AI-generated avatar videos did not improve objective knowledge scores but significantly enhanced engagement, perceived retention and perceived clarity (Cohen d=0.81-1.38). Future studies in actual patient populations with diverse health literacy levels are needed to determine whether these engagement advantages translate into improved knowledge outcomes.
28 May 2026
Read appraisal →Scientific reports
Impact of nurse-led postoperative education on health outcomes following endovascular aortic repair: a randomized trial
The purpose of this randomized controlled clinical trial was to investigate the health effects of an education-based intervention performed at a nurse-led clinic, for patients who underwent endovascular treatment for abdominal aortic aneurysm. Fifty-four adult patients were randomized to the intervention group (n = 26) or the control group (n = 28). The control group received standard care, and the intervention group received standard care alongside two consultations at a specialist surgical nurse-led clinic. The consultations focused on individual motivational support and education about abdominal aortic aneurysm disease. Data was collected prior to surgery (baseline) and at one, six, and 12 months postoperatively for each group. Outcomes of interest included participants' responses to the Health Education Impact Questionnaire, responses to the EuroQol Five Dimensions Questionnaire, and responses to four complementary items specific to the scope of this study. Both groups reported overall high ratings of perceived health throughout the study period. While there were no statistically significant differences between the two groups, the control group showed a substantial improvement in perceived understanding of their diagnosis over time. This randomized controlled trial did not find any significant health benefits as compared to standard treatment.Trial registration ClinicalTrials.gov (19/05/2025 NCT06986252). Retrospectively registered.
24 May 2026
Read appraisal →JMIR formative research
Use of Commercially Available Large Language Models to Generate Information Leaflets on Post-Intensive Care Syndrome: Clinical Utility Assessment
BACKGROUND: Patients and their families without medical knowledge may find professional health care information difficult to understand. The use of large language models (LLMs) to simplify and translate complex medical content holds promise for improving comprehension while reducing the burden on health care providers tasked with delivering explanations. OBJECTIVE: This study aims to evaluate the quality of information leaflets generated using commercially available LLMs. METHODS: Informational texts on post-intensive care syndrome were generated using 6 different LLMs and 4 prompt designs with varying levels of instructional guidance. Clinical practice guideline documents were uploaded and provided to the models as reference context, reflecting a pragmatic clinical scenario without model tuning or advanced retrieval pipelines. In total, 72 texts were generated (6 models × 4 prompts × 3 outputs). After excluding texts shorter than 500 characters (n=16) and those without explicit mention of post-intensive care syndrome (n=3), 53 texts remained. To enable balanced human evaluation across model-prompt combinations, the longest eligible response from each pair was selected (4 prompts × 4 models; n=16). Following independent expert review by 2 medical specialists, 7 texts were excluded, leaving 9 texts for final analysis. Ten individuals, including health care professionals and nonmedical personnel, assessed the texts using a 10-point Likert scale across multiple quality domains. An LLM-based parallel assessment was also conducted, and scores were compared across models and evaluator groups. RESULTS: In the human evaluation of the selected 9 texts, the generated texts achieved an average score of 6.8 or higher across all evaluation criteria, with no potentially harmful content identified. The text generated by LLaMA 3 70B, using a step-by-step approach combined with text-augmented prompting based on clinical guidelines, received the highest overall score, whereas the lowest-rated text was produced using a simple prompt without text augmentation. Although no consistent trends were observed across LLMs or prompt engineering strategies, text-augmented prompting was generally associated with higher evaluation scores. Ratings differed between professional and nonprofessional evaluators. Given the feasibility-driven screening process and the resulting limited sample size, the findings should be interpreted as exploratory and descriptive rather than definitive estimates of overall model performance. CONCLUSIONS: Among the selected texts included in the final human evaluation, informational materials generated using commercially available LLMs were generally rated as acceptable by human evaluators, and none contained harmful content. These findings suggest that LLMs may support the development of patient-facing informational materials under feasibility-constrained conditions, although further validation with larger and more diverse samples is warranted.
16 May 2026
Read appraisal →BMJ open
AI chatbots for health information seeking among Chinese patients with precancerous ENT lesions: a descriptive qualitative study
OBJECTIVE: To explore how Chinese patients with precancerous ear, nose and throat (ENT) lesions experience using artificial intelligence (AI)-driven chatbots for health-related information seeking, with particular attention to perceived benefits, challenges and influences on information-seeking practices. DESIGN: Descriptive qualitative study. SETTING: Department of Otolaryngology, West China Hospital, Sichuan University, China. PARTICIPANTS: 12 adult patients with clinically diagnosed precancerous ENT lesions who had used AI-driven chatbots at least three times in the previous 30 days to seek health-related information about their condition were purposively recruited. Face-to-face, semistructured interviews were conducted between 11 October 2024 and 10 November 2024. RESULTS: Interviews were analysed using Colaizzi's method. Four themes were identified. First, participants described AI chatbots as an immediate and accessible source of health information, particularly when questions arose outside clinical encounters or during periods of uncertainty. Second, many reported moving away from conventional search engines towards conversational information seeking, valuing direct and synthesised responses over link-based retrieval. Third, participants emphasised that obtaining useful answers depended on learning to ask clear and specific questions, suggesting that effective prompting was an important user skill. Fourth, some participants perceived chatbot interaction as emotionally safer than asking healthcare professionals certain questions, particularly when they felt embarrassed, worried about asking repetitive questions or feared being judged. CONCLUSIONS: Among this sample of Chinese patients with precancerous ENT lesions, AI-driven chatbots were perceived as a convenient and conversational supplementary source of health information. Participants valued their accessibility and interactional ease, but also indicated that their usefulness depended partly on users' ability to formulate effective prompts. Some participants additionally perceived chatbots as a more comfortable channel for asking sensitive or basic questions. The findings suggest that AI chatbots may have a complementary role in patient information support, but further research is needed to evaluate response accuracy, safety and appropriate integration into clinician-led care.
15 May 2026
Read appraisal →Journal of medical Internet research
Governing Patient-Facing AI-Generated Video in Digital Health: A Risk-and-Ethics Matrix for Deployment, Monitoring, and Change Control
In this Viewpoint, we argue that patient-facing high-fidelity artificial intelligence (AI)-generated video requires governance that is operational, life cycle based, and embedded in existing institutional review pathways rather than limited to predeployment checks alone. Patient-facing high-fidelity AI-generated video-synthetic or substantially AI-mediated video that presents realistic human likeness, voices, or clinical communication cues-is rapidly entering patient education and clinical communication. We propose a risk-and-ethics matrix that combines residual clinical risk (likelihood × severity after mitigations) with an ethical alignment score that operationalizes autonomy, beneficence, nonmaleficence, and justice to yield actionable dispositions (encourage, permit with oversight, restrict or redesign, or prohibit). The framework links each disposition to dossier-based review, minimum controls, and postdeployment monitoring triggers-focused on measurable outcomes (eg, comprehension, content-attributable follow-up burden, incidents and complaints, and equity gaps) as well as provenance and change control-to support auditable, revisitable decisions over the system life cycle.
13 May 2026
Read appraisal →Journal of medical Internet research
Automated Approaches of Text Simplification of Patient Education Materials: Scoping Review
BACKGROUND: Patient education materials (PEMs) often exceed the American Medical Association's (AMA) recommended sixth-grade reading grade level (RGL). While artificial intelligence (AI) offers potential for automated text simplification, concerns persist regarding linguistic quality, content fidelity, and the understandability of simplified PEMs by laypeople. OBJECTIVE: This scoping review maps existing evidence on automated language processing technologies for simplifying PEMs for laypeople. METHODS: Following the Joanna Briggs Institute (JBI) methodology and the PRISMA-ScR (Preferred Reporting Items for Systematic Reviews and Meta-Analyses extension for Scoping Reviews) guideline, 5 bibliographic databases (Ovid MEDLINE, Embase, CINAHL, PsycInfo, and IEEE Xplore) were systematically searched from 2019 to May 2025, supplemented by reference screening and gray literature searches. Eligible sources were peer-reviewed empirical studies published in English that examined large language models (LLMs), AI-supported writing assistants, AI-based conversational agents, or AI-supported tools designed for automatic text simplification of PEMs. Targeted outcomes included linguistic quality (ie, linguistic comprehensibility, linguistic correctness) and content fidelity (ie, factual accuracy, factual completeness) of simplified PEMs. Excluded sources comprised rule-based systems, manual text simplification, non-laypeople target groups, and technology-focused performance metrics. Results were synthesized via thematic analysis across the domains of targeted outcomes. In accordance with JBI methodology, a risk-of-bias assessment was not performed. RESULTS: A total of 31 eligible studies met the inclusion criteria, examining various LLMs, including OpenAI's GPT series, Gemini, Bard, Claude, Copilot, and Llama. Specifically, GPT-4.0 achieved the most consistent improvements in standardized readability metrics (eg, the Flesch-Kincaid Grade Level [FKGL]). However, achieving predefined target RGLs remained challenging across all LLMs, particularly at lower RGLs. Findings on content fidelity were inconsistent: despite high content similarity scores, content accuracy was often compromised. CONCLUSIONS: This is the first scoping review to comprehensively synthesize evidence on automated technologies for text simplification in PEMs. The review identified 2 critical validation gaps. First, no study examined the linguistic correctness (eg, grammar and typographical errors) of automatically simplified PEMs. Second, and most notably, the understandability of the simplified PEMs was assessed exclusively by experts, with no empirical evaluation involving laypeople. Although LLMs effectively reduce text complexity as measured by objective readability metrics, reliance on these formulas represents a critical limitation, as they serve merely as structural proxies. Improvements in readability do not guarantee the maintenance of content accuracy or laypeople's understandability. Current evidence is further limited by the lack of systematic prompt quality evaluation and the predominant focus on English-language PEMs in US contexts, restricting generalizability. This review provides a foundation for future research by highlighting the need for validated evaluation frameworks that encompass layperson testing and content verification. For clinical practice, LLMs should currently serve as assistive tools, with mandatory expert review remaining essential to verify content fidelity before disseminating LLM-simplified PEMs to laypeople.
12 May 2026
Read appraisal →JMIR medical informatics
Leveraging MedlinePlus to Improve Health Information Access Among Patients and Caregivers: Systematic Literature Review
BACKGROUND: MedlinePlus, developed by the National Library of Medicine (NLM) in the United States, is one of the most widely used, authoritative, consumer-grade health information resources on the web. Although extensively used and discussed in scholarly work for health literacy and patient education, it is unclear how MedlinePlus has been integrated into clinical care or embedded within health informatics applications. OBJECTIVE: This study aimed to understand how MedlinePlus has supported patients and caregivers by increasing access to health information for clinical care and illness management. The insights on this topic will inform the design and development of patient-facing digital health intervention tools for improved health communication, decision engagement, informed decision-making, and health outcomes. METHODS: We conducted a systematic literature review following the PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) guidelines. First, we developed a comprehensive literature search strategy, searched 9 citation databases, and aggregated and deduplicated search results before importing them into Covidence for manual screening using predefined inclusion and exclusion criteria. Second, reviewers independently assessed all studies at the title-abstract and full-text levels, resolving discrepancies through ongoing discussions. Third, we applied the PICO (problem/population, intervention, comparison, and outcome) and the Collaborative Chronic Care Model as guiding frameworks for data extraction and analysis. All included studies underwent quality assessment using the Mixed Methods Appraisal Tool. RESULTS: In total, 28 studies reported in 27 sources met our inclusion criteria. We categorized the extracted data into 4 areas. First, regarding bibliometrics, the studies were reported between 2004 and 2024, with 2010 having the highest number of studies. Of these studies, 25 were conducted in the United States, 2 were conducted in Iran, and 1 was conducted in Argentina. Health informatics journals and conference proceedings, as well as library science journals, were prominent publishing venues. The NLM funded half of the studies. Second, regarding participants, most studies focused on outpatients. Other participant roles included physicians, nurses, hospital staff, pharmacists, and librarians. Fewer than half of the studies addressed the social determinants of health. Third, regarding intervention, most studies implemented MedlinePlus information interventions within clinical settings. Other interventions occurred in community pharmacies, community organizations, libraries, online health platforms, or patient portals. Fourth, regarding outcome, only 4 studies assessed clinical outcomes, and the findings were mixed and inconsistent. However, 24 of 28 studies reported positive nonclinical outcomes, including improved attitudes toward and satisfaction with MedlinePlus and enhancements in patients' information-seeking behaviors, confidence, and willingness to engage in decision-making, physician-patient communication, self-management, and self-efficacy. CONCLUSIONS: This systematic literature review is the first comprehensive examination of how MedlinePlus has been integrated into clinical care, supporting patients and caregivers with enhanced access to health information. Our findings offer evidence and insights through the Collaborative Chronic Care Model lens and can guide the development of digital health interventions to improve patient health.
3 May 2026
Read appraisal →JMIR formative research
Retrieval-Augmented Generation Versus GPT-4o for Patient-Facing Gynecological Cancer Information: Quality Evaluation
Retrieval-augmented generation improved overall quality scores for patient-facing gynecological cancer information mainly through better source attribution.
28 Apr 2026
Read appraisal →Annals of medicine
The effect of nutrition education interventions on dialysis patients' outcomes: a systematic review and meta-analysis
BACKGROUND: Non-adherence to dietary and fluid restrictions among dialysis patients is associated with adverse clinical outcomes. Quantifying the effectiveness of nutrition education interventions can inform practice and policy. METHODS: We searched MEDLINE, EMBASE, CINAHL, CENTRAL, PsycINFO, Web of Science, and Scopus up to 15 July 2025, supplemented by trial registries and Google Scholar, with an updated search through 8 April 2026. Eligible studies included randomized and non-randomized trials evaluating nutritional education interventions in adult dialysis populations. Risk of bias was assessed using RoB-2 and ROBINS-I, certainty graded using GRADE, and random-effects meta-analyses conducted alongside subgroup, sensitivity, and meta-regression analyses. Publication bias was assessed with Egger and Begg tests and trim-and-fill where applicable. RESULTS: Forty-four studies comprising 4,106 participants were included. Nutrition education significantly improved knowledge (SMD = 1.09; 95% CI: 0.67-1.51) and health-related quality of life (SMD = 1.43; 95% CI: 0.86-2.00; I² = 0%), and reduced serum potassium (SMD = -0.52; 95% CI: -0.91 to -0.14; I² = 92%) and serum phosphate (SMD = -0.35; 95% CI: -0.56 to -0.15; I² = 81%). Results for albumin, creatinine, sodium, calcium, and BUN were inconsistent and non-significant. Most outcomes were rated low or very low certainty by GRADE, reflecting inconsistency, indirectness, and imprecision. Potential publication bias was identified for certain outcomes. CONCLUSIONS: Nutrition education consistently improves knowledge and quality of life and may modestly reduce serum phosphate and potassium in dialysis patients. High-quality registered trials with standardized outcomes and longer follow-up are needed to establish effectiveness and sustainability. PROSPERO REGISTRATION: CRD420251119567.
24 Apr 2026
Read appraisal →BMJ open
A double-blind, crossover, non-inferiority randomised controlled trial where primary care providers and patients compare human-generated and AI-generated digital health messages: the AI-CARE study protocol
INTRODUCTION: Primary care is facing multiple crises, including an increase in health misinformation. Digital health messaging by primary care providers has been shown to reach a diverse patient population. With the uptake of Generative Artificial Intelligence (GenAI) usage in healthcare, there is an important opportunity to rapidly create messages that are tailored to different populations and conditions. However, thoroughly assessing artificial intelligence (AI)-generated content is essential, as GenAI raises concerns regarding its accuracy, understandability, actionability and bias perpetuation. We aim to investigate whether digital health messages created by GenAI are evaluated as non-inferior compared with those created by human experts. METHODS AND ANALYSIS: The AI-CARE (AI to Create Accessible and Reliable patient Education materials) study is a double-blind, crossover, non-inferiority randomised controlled trial. Data collection began on 30 May 2025, and is expected to be completed at the end of May 2026. Over 12 months, 192 messages on 48 topics will be written: half by primary care and public health experts and half by a GenAI tool (OpenAI's ChatGPT). Review Panels composed of 24 primary care providers and 24 patients will evaluate these messages using an Evaluation Grid developed to assess the messages' quality of information, adaptation to the target audience, relevance and usefulness, and readiness to be shared with patients. Evaluations will be completed via online REDCap (Research Electronic Data Capture) surveys and the order in which the 192 messages appear will be randomised and will vary between individuals. Participants and analysts will be blinded to the generation source. The primary outcome will be the Clarity and Understandability score. ETHICS AND DISSEMINATION: The Research Ethics Boards of the Hôpital Montfort (24-25-11-038) and the University of Ottawa (S-12-24-11153) formally approved this study in December 2024. Reported data will be grouped and anonymised for dissemination in peer-reviewed scientific journals and conferences. TRIAL REGISTRATION NUMBER: NCT06997107.
9 Apr 2026
Read appraisal →Am J Obstet Gynecol
Can artificial intelligence improve the readability of patient education information in gynecology?
BACKGROUND: The American Medical Association recommends that patient information be written at a sixth-grade level to increase accessibility. However, most existing patient education materials exceed this threshold, posing challenges to patient comprehension. Artificial intelligence, particularly large language models, presents an opportunity to improve the readability of medical information. Despite the growing integration of artificial intelligence in healthcare, few studies have evaluated the effectiveness of large language models in generating or improving readability of existing patient education materials within gynecology. OBJECTIVE: To assess the readability and effectiveness of patient education materials generated by ChatGPT, Gemini, and CoPilot compared to American College of Obstetricians and Gynecologists and UpToDate.com. Additionally, to determine whether these large language models can successfully adjust the reading level to a sixth-grade standard. STUDY DESIGN: This cross-sectional study analyzed American College of Obstetricians and Gynecologists, UpToDate, and large language model-generated content, evaluating large language models for 2 tasks: 1) independent large language model-generated materials and 2) large language model-enhanced versions reducing existing patient information to sixth-grade level. All materials were assessed for basic textual analysis and readability using 8 readability formulas. Two board-certified obstetrician-gynecologists evaluated blinded patient education materials for accuracy, clarity, and comprehension. Analysis of variance was used to compare textual analysis and readability scores, with Tukey post-hoc tests identifying differences for both original and enhanced materials. An alpha threshold of P<.004 was used to account for multiple comparisons. RESULTS: Large language model-generated materials were significantly shorter (mean word count 407.9 vs 1132.0; P<.001) but had a higher proportion of difficult words (36.7% vs 27.4%; P<.001). American College of Obstetricians and Gynecologists and UpToDate materials averaged ninth-grade and 8.6-grade levels, respectively, while artificial intelligence-generated content reached a 10.6-grade level (P = .008). Although CoPilot and Gemini improved readability when prompted, no large language model successfully reached the sixth-grade benchmark, and ChatGPT increased reading difficulty. CONCLUSION: Large language models generated more concise patient education materials but often introduced more complex vocabulary, ultimately failing to meet recommended health literacy standards. Even when explicitly prompted, no large language model achieved the sixth-grade reading level required for optimal patient comprehension. Without proper oversight, artificial intelligence-generated patient education materials may create the illusion of simplicity while reducing true accessibility. Future efforts should focus on integrating health literacy safeguards into artificial intelligence models before clinical implementation.
1 Dec 2025
Read appraisal →