Research Appraisals
Evidence-based critical appraisals of the latest medical research, systematically evaluated using Oxford CEBM methodology.
Showing 4 appraisals
Journal of medical Internet research
Automated Approaches of Text Simplification of Patient Education Materials: Scoping Review
BACKGROUND: Patient education materials (PEMs) often exceed the American Medical Association's (AMA) recommended sixth-grade reading grade level (RGL). While artificial intelligence (AI) offers potential for automated text simplification, concerns persist regarding linguistic quality, content fidelity, and the understandability of simplified PEMs by laypeople. OBJECTIVE: This scoping review maps existing evidence on automated language processing technologies for simplifying PEMs for laypeople. METHODS: Following the Joanna Briggs Institute (JBI) methodology and the PRISMA-ScR (Preferred Reporting Items for Systematic Reviews and Meta-Analyses extension for Scoping Reviews) guideline, 5 bibliographic databases (Ovid MEDLINE, Embase, CINAHL, PsycInfo, and IEEE Xplore) were systematically searched from 2019 to May 2025, supplemented by reference screening and gray literature searches. Eligible sources were peer-reviewed empirical studies published in English that examined large language models (LLMs), AI-supported writing assistants, AI-based conversational agents, or AI-supported tools designed for automatic text simplification of PEMs. Targeted outcomes included linguistic quality (ie, linguistic comprehensibility, linguistic correctness) and content fidelity (ie, factual accuracy, factual completeness) of simplified PEMs. Excluded sources comprised rule-based systems, manual text simplification, non-laypeople target groups, and technology-focused performance metrics. Results were synthesized via thematic analysis across the domains of targeted outcomes. In accordance with JBI methodology, a risk-of-bias assessment was not performed. RESULTS: A total of 31 eligible studies met the inclusion criteria, examining various LLMs, including OpenAI's GPT series, Gemini, Bard, Claude, Copilot, and Llama. Specifically, GPT-4.0 achieved the most consistent improvements in standardized readability metrics (eg, the Flesch-Kincaid Grade Level [FKGL]). However, achieving predefined target RGLs remained challenging across all LLMs, particularly at lower RGLs. Findings on content fidelity were inconsistent: despite high content similarity scores, content accuracy was often compromised. CONCLUSIONS: This is the first scoping review to comprehensively synthesize evidence on automated technologies for text simplification in PEMs. The review identified 2 critical validation gaps. First, no study examined the linguistic correctness (eg, grammar and typographical errors) of automatically simplified PEMs. Second, and most notably, the understandability of the simplified PEMs was assessed exclusively by experts, with no empirical evaluation involving laypeople. Although LLMs effectively reduce text complexity as measured by objective readability metrics, reliance on these formulas represents a critical limitation, as they serve merely as structural proxies. Improvements in readability do not guarantee the maintenance of content accuracy or laypeople's understandability. Current evidence is further limited by the lack of systematic prompt quality evaluation and the predominant focus on English-language PEMs in US contexts, restricting generalizability. This review provides a foundation for future research by highlighting the need for validated evaluation frameworks that encompass layperson testing and content verification. For clinical practice, LLMs should currently serve as assistive tools, with mandatory expert review remaining essential to verify content fidelity before disseminating LLM-simplified PEMs to laypeople.
12 May 2026
Read appraisal →JMIR medical informatics
Leveraging MedlinePlus to Improve Health Information Access Among Patients and Caregivers: Systematic Literature Review
BACKGROUND: MedlinePlus, developed by the National Library of Medicine (NLM) in the United States, is one of the most widely used, authoritative, consumer-grade health information resources on the web. Although extensively used and discussed in scholarly work for health literacy and patient education, it is unclear how MedlinePlus has been integrated into clinical care or embedded within health informatics applications. OBJECTIVE: This study aimed to understand how MedlinePlus has supported patients and caregivers by increasing access to health information for clinical care and illness management. The insights on this topic will inform the design and development of patient-facing digital health intervention tools for improved health communication, decision engagement, informed decision-making, and health outcomes. METHODS: We conducted a systematic literature review following the PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) guidelines. First, we developed a comprehensive literature search strategy, searched 9 citation databases, and aggregated and deduplicated search results before importing them into Covidence for manual screening using predefined inclusion and exclusion criteria. Second, reviewers independently assessed all studies at the title-abstract and full-text levels, resolving discrepancies through ongoing discussions. Third, we applied the PICO (problem/population, intervention, comparison, and outcome) and the Collaborative Chronic Care Model as guiding frameworks for data extraction and analysis. All included studies underwent quality assessment using the Mixed Methods Appraisal Tool. RESULTS: In total, 28 studies reported in 27 sources met our inclusion criteria. We categorized the extracted data into 4 areas. First, regarding bibliometrics, the studies were reported between 2004 and 2024, with 2010 having the highest number of studies. Of these studies, 25 were conducted in the United States, 2 were conducted in Iran, and 1 was conducted in Argentina. Health informatics journals and conference proceedings, as well as library science journals, were prominent publishing venues. The NLM funded half of the studies. Second, regarding participants, most studies focused on outpatients. Other participant roles included physicians, nurses, hospital staff, pharmacists, and librarians. Fewer than half of the studies addressed the social determinants of health. Third, regarding intervention, most studies implemented MedlinePlus information interventions within clinical settings. Other interventions occurred in community pharmacies, community organizations, libraries, online health platforms, or patient portals. Fourth, regarding outcome, only 4 studies assessed clinical outcomes, and the findings were mixed and inconsistent. However, 24 of 28 studies reported positive nonclinical outcomes, including improved attitudes toward and satisfaction with MedlinePlus and enhancements in patients' information-seeking behaviors, confidence, and willingness to engage in decision-making, physician-patient communication, self-management, and self-efficacy. CONCLUSIONS: This systematic literature review is the first comprehensive examination of how MedlinePlus has been integrated into clinical care, supporting patients and caregivers with enhanced access to health information. Our findings offer evidence and insights through the Collaborative Chronic Care Model lens and can guide the development of digital health interventions to improve patient health.
3 May 2026
Read appraisal →BMJ open
Do studies of interventions to improve laypeople's critical thinking about health choices assess potential harms? A systematic review.
OBJECTIVES: To make informed health choices, and avoid waste and unnecessary suffering, people need critical thinking skills. However, like health interventions, educational interventions can have adverse effects. In this systematic review, the objective was to assess the extent to which researchers have included potential adverse effects in studies of interventions intended to improve the critical thinking of laypeople about health choices. DESIGN: This study was a systematic review, in which we updated the search for an earlier systematic review of intended effects of relevant interventions. The earlier review did not address potential adverse effects. We did not update the analysis of intended effects. DATA SOURCES: We searched Cochrane Central Register of Controlled Trials (CENTRAL), Cumulative Index to Nursing and Allied Health Literature (CINAHL), Embase, Epistemonikos, Medical Literature Analysis and Retrieval System Online (MEDLINE), Education Resources Information Center (ERIC) and Web of Science up to March 2025. In addition to studies from the original review and updated search, we included any additional studies included in a similar, even earlier review. Our unit of analysis was study report (eg, journal article). ELIGIBILITY CRITERIA: We included all studies from the original review. We applied the same inclusion criteria to the results of the updated search: the study included a comparison, the population was laypeople and the intervention was intended to improve understanding of ≥1 key concept for informed health choices. DATA EXTRACTION AND SYNTHESIS: We extracted data about study design (randomised trial or other), participants (children, adolescents or adults), study setting (countries), main intervention (resources delivered to participants) and comparator (usual/no intervention or other). For the analysis, we extracted verbatim text describing any assessment of a potential adverse effect of the intervention. We conducted a narrative synthesis of the extracted data. RESULTS: We included 35 reports of quantitative studies (including multi-method and mixed-methods studies). Most often, the study was a randomised trial, the setting was a high-income country, the population included adults (including university students) and the intervention was school-based (including university). In one of the 35 reports, authors described assessing a potential adverse effect. CONCLUSION: To our knowledge, this is the first systematic review assessing the extent to which researchers have assessed adverse effects of any category of educational interventions. Our review shows that researchers generally have not assessed potential adverse effects of interventions to improve critical thinking about health choices. Researchers should pay more attention to such effects, while policymakers and educators making decisions about implementing relevant interventions should consider the lack of evidence. The findings of this study suggest a need for research that facilitates assessing potential adverse effects of interventions to improve critical thinking about health choices.
27 Apr 2026
Read appraisal →Am J Obstet Gynecol
Can artificial intelligence improve the readability of patient education information in gynecology?
BACKGROUND: The American Medical Association recommends that patient information be written at a sixth-grade level to increase accessibility. However, most existing patient education materials exceed this threshold, posing challenges to patient comprehension. Artificial intelligence, particularly large language models, presents an opportunity to improve the readability of medical information. Despite the growing integration of artificial intelligence in healthcare, few studies have evaluated the effectiveness of large language models in generating or improving readability of existing patient education materials within gynecology. OBJECTIVE: To assess the readability and effectiveness of patient education materials generated by ChatGPT, Gemini, and CoPilot compared to American College of Obstetricians and Gynecologists and UpToDate.com. Additionally, to determine whether these large language models can successfully adjust the reading level to a sixth-grade standard. STUDY DESIGN: This cross-sectional study analyzed American College of Obstetricians and Gynecologists, UpToDate, and large language model-generated content, evaluating large language models for 2 tasks: 1) independent large language model-generated materials and 2) large language model-enhanced versions reducing existing patient information to sixth-grade level. All materials were assessed for basic textual analysis and readability using 8 readability formulas. Two board-certified obstetrician-gynecologists evaluated blinded patient education materials for accuracy, clarity, and comprehension. Analysis of variance was used to compare textual analysis and readability scores, with Tukey post-hoc tests identifying differences for both original and enhanced materials. An alpha threshold of P<.004 was used to account for multiple comparisons. RESULTS: Large language model-generated materials were significantly shorter (mean word count 407.9 vs 1132.0; P<.001) but had a higher proportion of difficult words (36.7% vs 27.4%; P<.001). American College of Obstetricians and Gynecologists and UpToDate materials averaged ninth-grade and 8.6-grade levels, respectively, while artificial intelligence-generated content reached a 10.6-grade level (P = .008). Although CoPilot and Gemini improved readability when prompted, no large language model successfully reached the sixth-grade benchmark, and ChatGPT increased reading difficulty. CONCLUSION: Large language models generated more concise patient education materials but often introduced more complex vocabulary, ultimately failing to meet recommended health literacy standards. Even when explicitly prompted, no large language model achieved the sixth-grade reading level required for optimal patient comprehension. Without proper oversight, artificial intelligence-generated patient education materials may create the illusion of simplicity while reducing true accessibility. Future efforts should focus on integrating health literacy safeguards into artificial intelligence models before clinical implementation.
1 Dec 2025
Read appraisal →