Research Appraisals
Evidence-based critical appraisals of the latest medical research, systematically evaluated using Oxford CEBM methodology.
Showing 4 appraisals
JMIR formative research
Automatic Speech Recognition and Large Language Models for Multilingual Pathology Report Generation: Proof-of-Concept Study
BACKGROUND: Accurate transcription of pathology gross examination dictation is important for clinical documentation, but multilingual dictation remains challenging in settings where clinicians mix Chinese and English while final pathology reports are written in English. OBJECTIVE: This study aimed to evaluate whether a Whisper-based automatic speech recognition (ASR) pipeline guided by contextual system messages and combined with open-source large language models (LLMs; Qwen2:72b, Llama3.1:70b, Gemma2:27b) could improve multilingual (Chinese-English) pathology dictation transcription accuracy and generate clinically appropriate English gross description reports. METHODS: We conducted a controlled proof-of-concept study using 125 simulated mixed Chinese-English pathology gross examination audio recordings created by physicians or pathologists. Audio recordings were transcribed using Whisper ASR with and without a contextual system message. The ASR transcripts were then converted into English gross description reports using 3 open-source LLMs: Qwen2:72b, Llama3.1:70b, and Gemma2:27b. Outcomes included character error rate, Bilingual Evaluation Understudy, Recall-Oriented Understudy for Gisting Evaluation (ROUGE)-1, ROUGE-2, ROUGE-L, Metric for Evaluation of Translation with Explicit Ordering, pathologist Win-Tie-Lose rankings, report-level error categories, inference time, and interrater agreement. RESULTS: The ASR contextual system message reduced the mean character error rate from 0.344 (SD 0.176; 95% CI 0.313-0.375) to 0.066 (SD 0.100; 95% CI 0.048-0.084; P<.001). Qwen2:72b achieved the highest automated metric scores, including a Bilingual Evaluation Understudy of 0.644 (SD 0.307), ROUGE-1 of 0.866 (SD 0.163), ROUGE-2 of 0.771 (SD 0.235), ROUGE-L of 0.842 (SD 0.178), and Metric for Evaluation of Translation with Explicit Ordering of 0.805 (SD 0.214). Pathologist-coded total error rates were 16.8% (21/125) for Qwen2:72b, 45.6% (57/125) for Llama3.1:70b, and 92.8% (116/125) for Gemma2:27b. The exact agreement between the 2 pathologists across full ranking categories was 76.8% (96/125; Cohen κ=0.668), and agreement on the top-ranked model or tied top group was 81.6% (102/125; Cohen κ=0.722). CONCLUSIONS: In this proof-of-concept evaluation, contextual prompting improved ASR transcription accuracy, and Qwen2:72b generated the most accurate English pathology reports among the evaluated LLMs. However, the study used simulated audio recordings, a local vocabulary prompt, and report-level rather than term-level clinical annotation. LLM-generated reports should therefore be considered draft documentation requiring pathologist verification, and prospective validation in real clinical workflows is needed before clinical deployment.
15 May 2026
Read appraisal →Annals of emergency medicine
Ambient Artificial Intelligence Scribe Adoption and Documentation Time in the Emergency Department
STUDY OBJECTIVES: To describe real-world adoption of an ambient artificial intelligence (AI) scribe in the emergency department (ED) and compare documentation time and note characteristics between ambient and standard encounters using electronic health record audit logs. METHODS: We performed a retrospective observational study of adult ED encounters at a tertiary academic medical center. Attending physicians could optionally use an ambient AI scribe to generate notes from patient-clinician conversations. We included single-attending encounters in core ED zones and excluded visits with human scribes. Electronic health record audit logs provided documentation of time during and after the shift, total electronic health record time, and note length. We summarized adoption by physician, zone, and acuity and compared medians between ambient and standard encounters. RESULTS: Among 8,740 eligible encounters, 976 (11.2%) used ambient AI. Thirty-five of 92 attendings (38%) used the tool, and a small group of high-frequency users accounted for most ambient encounters. Ambient use clustered in telemedicine and vertical-care zones (chair-based ambulatory care) and in lower-acuity patients, as well as those not requiring interpreters. Median on-shift documentation time was 2:45 min for ambient encounters versus 3:50 min for standard encounters (difference -1:05; -28%). Median total electronic health record time was 8:39 min versus 10:21 min (-16%), and ambient notes were shorter overall. CONCLUSION: Early ED implementation of ambient AI scribes showed low but highly skewed adoption, with physicians favoring lower acuity, noninterpreted encounters. When used, ambient AI was associated with shorter on-shift documentation time, total electronic health record time, and note length.
10 May 2026
Read appraisal →JMIR medical informatics
Quality of Clinical Notes Created by Ambient Listening Generative AI: Pragmatic Prospective Pilot Study
BACKGROUND: Physicians routinely document specifics of patient encounters in clinic visit notes, a critical but potentially time-consuming task. Ambient listening artificial intelligence (AI) technology is being integrated into clinical workflows to reduce documentation burden by creating draft visit notes. While this technology is promising, it is not perfect, and the potential for patient harm needs to be understood and mitigated. We developed and piloted an efficient, standardized approach to evaluating AI-generated notes for safety concerns in ambulatory care visits. OBJECTIVE: The objective of this quality improvement project was to develop and pilot an efficient, standardized, and scalable approach to evaluating AI-generated notes for safety concerns in ambulatory care visits. METHODS: During a 2-month pilot (July to August 2024), 31 physicians across multiple specialties used an ambient listening AI scribe to assist with the creation of 7545 clinic notes. A novel survey instrument was developed to assess note quality, focusing on 4 error types: accidental inclusions, accidental omissions, hallucinations, and bias. Physicians evaluated 356 (4.7%) AI-generated notes. Where an error was present, physicians rated its severity based on its potential to cause patient harm if it was not corrected, on a 0 to 5 scale. Additionally, a vendor-reported metric on the percentage of note content edited by physicians was analyzed. RESULTS: Of the 356 evaluated notes, accidental omissions were the most frequent error (n=64, 18%), followed by hallucinations (n=41, 11.5%), and accidental inclusions (n=33, 9.3%). Bias was rare (n=4, 1.1%). Most (119/142, 83.8%) errors were rated as mild to moderate (severity 1-3), with only 19 (5.3%) notes containing errors rated as posing serious or imminent risk (severity 4-5). Editing metrics across all AI-created notes showed a median of 9.0% (IQR 2.5%-21.9%) of AI-generated words were changed, with 14.9% (143/960) of notes left entirely unedited. Physician editing practices varied widely, with average percentages of AI-generated words changed ranging from 1.9% to 69.3% (median 9.0%, IQR 2.5%-21.9%). CONCLUSIONS: AI-generated clinical notes were generally of high quality, with 94.7% (337/356) free from significant errors. However, because a small number contained errors that carried the risk of serious harm if not corrected, careful clinician review of notes remains imperative. Prior to deploying an AI scribe, organizations should pilot the technology and include an efficient review process to understand the nature and type of errors common at their organization. This pilot provides a scalable model for other health systems seeking to implement AI scribe technology responsibly.
18 Apr 2026
Read appraisal →International journal of nursing studies
The use of large language models in clinical documentation: A scoping review
BACKGROUND: Clinical documentation is essential for safe, high-quality care but has become increasingly complex, contributing to clinician burnout. Large language models offer potential to ease documentation by generating summaries, structuring data, and ensuring compliance. However, concerns remain regarding accuracy, bias, privacy, and regulatory risks. OBJECTIVE: To map current literature on large language models applications in clinical documentation, evaluating their benefits, limitations, and ethical considerations. INFORMATION SOURCES: Five electronic databases (i.e., PubMed, Scopus, CINAHL, Cochrane Library, and IEEE Xplore) covering peer-reviewed literature published in English between January 2009 and August 2025. METHODS: This scoping review followed Arksey and OMalleys framework and was reported in accordance with PRISMA-ScR guidelines. Screening, data extraction, and quality appraisal were conducted independently by multiple reviewers using Joanna Briggs Institute tools. Findings were synthesized using descriptive and narrative approaches. RESULTS: Forty-one studies met inclusion criteria, most originating from the United States. Large language models were primarily applied to clinical note generation, discharge summaries, and provider-patient encounter documentation. Key evaluation metrics included content accuracy, linguistic quality, and summarization performance. Large language models demonstrated potential to improve documentation efficiency and readability, with some studies reporting up to 40 % time savings. However, concerns about factual inaccuracies, hallucinations, and reduced performance in complex cases were common. Clinician perceptions were mixed. Some found notes generated by large language models helpful and well-structured, while others raised concerns about reliability, liability, and loss of clinical nuance. Ethical challenges included data privacy, security, and algorithmic bias, with varying levels of compliance across settings. CONCLUSIONS: Large language models hold significant promise for enhancing clinical documentation by improving efficiency, standardization, and clarity. However, their safe and effective use requires rigorous attention to accuracy, ethical safeguards, and clinician trust. Integration must support, rather than supplant, clinical reasoning and patient-centered care. Co-design with clinicians, real-world evaluation, and artificial intelligence literacy are essential to ensure that these technologies augment, not erode, professional judgment and care quality. REGISTRATION: Open Science Framework Registries (https://osf.io/m4h3q).
2 Apr 2026
Read appraisal →