Advancing Knowledge in Evaluating the Clinical Impact of Large Language Models for Clinical Text Summarization: A Narrative Review
Clinical Snapshot
PICO Framework
| P — Population | Studies evaluating large language models for clinical text summarization from electronic health records |
| I — Intervention | Large language model-based clinical text summarization systems |
| C — Comparator | Not applicable (narrative review examining evaluation methods) |
| O — Outcomes | Clinical impact evaluation across four dimensions: utility, failure, patient safety risks, and bias |
Bottom Line
This narrative review identifies significant gaps in how we evaluate the clinical impact of large language models for summarizing electronic health records. While these systems show promise for reducing documentation burden, current evaluation methods are inconsistent and inadequately address patient safety risks. Only 3% of studies explicitly examined patient safety, despite the potential for generated summaries to propagate misinformation and harm patients. The review highlights the urgent need for standardized evaluation frameworks, consistent error taxonomies, and systematic bias detection methods before these technologies can be safely deployed in clinical practice. Healthcare organizations should proceed cautiously with LLM-based summarization systems until more robust evaluation standards are established.
Key Findings
Effect Size: Not applicable - descriptive analysis
Primary Outcome: Distribution of evaluation approaches: failure analysis (50%), utility analysis (25%), bias analysis (22%), patient safety risk analysis (3%)
Nnt Or Sensitivity: 32 studies included from 144 retrieved (22% inclusion rate)
Clinical Application
High relevance for informing evaluation frameworks, but insufficient evidence for immediate clinical deployment Relevant to Australian digital health initiatives and TGA considerations for AI-based medical devices, aligns with RACGP digital health guidelines emphasizing patient safety in technology adoption Healthcare organizations and researchers implementing or evaluating LLM-based clinical text summarization systems
Abstract
Large language models (LLMs) are increasingly explored for clinical text summarization from electronic health records (EHRs), where they could reduce documentation burden. While promising, errors in generated summaries may propagate misinformation, misrepresent clinical facts, and ultimately pose risks to patient safety. This review examined how the clinical impact of LLM-based summarization has been evaluated across four dimensions: utility, failure, patient safety risks, and bias. Literature was retrieved from PubMed between June 2024 and September 2025. Of the 144 retrieved studies, 32 studies were included in the analysis. Failure analysis was the most frequently reported (n=16, 50%), focusing on inaccuracies, omissions, and hallucinations, though definitions and methods varied widely. Utility analysis (n=8, 25%) examined workflow efficiency, readability, and understandability. Bias analysis (n=7, 22%) considered gender, race, and stigmatizing language. Only one study explicitly conducted patient safety risk analysis (n=1, 3%), using risk matrices. Findings indicate that evaluation frameworks are gradually moving beyond technical performance metrics; however, methodologies remain heterogeneous and lack standardization. Future research should establish consistent error taxonomies, structured risk assessment frameworks, and systematic bias detection methods to enable the safe clinical deployment of LLM-based summarization systems.
References
- 1.Bednarczyk, L., Bjelogrlic, M., Zaghir, J., Tcherepanova, M., Ehrsam, J., Bensahla, A., Lovis, C., & Gaudet-Blavignac, C. (2026). Advancing Knowledge in Evaluating the Clinical Impact of Large Language Models for Clinical Text Summarization: A Narrative Review. Studies in Health Technology and Informatics. https://doi.org/10.3233/SHTI260243
Related Research
Journal of medical Internet research
Advancing Human-Centered AI in Clinical Decision Support: Sociocognitive Human-in-the-Loop Study in HIV Care
2 Aug 2026
Neurological research
Federated deep learning model for epilepsy seizure detection using electroencephalogram (EEG) signal
2 Aug 2026
Neural networks : the official journal of the International Neural Network Society
MFBD: Model-free backdoor defense based on vision-language pre-trained models
2 Aug 2026
This content is for educational purposes for healthcare professionals only and does not constitute clinical advice. Clinical decisions should be based on individual patient assessment, current guidelines, and appropriate specialist consultation. Editorial Standards · Privacy Policy · Terms of Service