Research Appraisals
Evidence-based critical appraisals of the latest medical research, systematically evaluated using Oxford CEBM methodology.
Showing 6 appraisals
Studies in health technology and informatics
Towards Clinical NLP in a Low-Resource Language: Structuring Estonian Clinical Text
Healthcare sectors generate large volumes of unstructured text, yet mature clinical NLP is lacking for low-resource languages. This project targets Estonian, a morphologically rich, low-resource language, and outlines an end-to-end pipeline that converts free text into interoperable, structured data. A design-science approach guides development of multilingual speech-to-text for clinical dictation, clinical NLP for entity extraction and context handling, terminology binding to the Estonian SNOMED CT extension, and export as HL7 FHIR resources with mappings to EHR fields. Expected benefits include reduced documentation time through dictation with automatic structuring, improved data quality and consistency via controlled terminology, and faster retrieval of clinically relevant information to support decision-making. Pilot evaluations with healthcare partners will assess accuracy, usability, and workflow impact.
24 May 2026
Read appraisal →Studies in health technology and informatics
Advancing Knowledge in Evaluating the Clinical Impact of Large Language Models for Clinical Text Summarization: A Narrative Review
Large language models (LLMs) are increasingly explored for clinical text summarization from electronic health records (EHRs), where they could reduce documentation burden. While promising, errors in generated summaries may propagate misinformation, misrepresent clinical facts, and ultimately pose risks to patient safety. This review examined how the clinical impact of LLM-based summarization has been evaluated across four dimensions: utility, failure, patient safety risks, and bias. Literature was retrieved from PubMed between June 2024 and September 2025. Of the 144 retrieved studies, 32 studies were included in the analysis. Failure analysis was the most frequently reported (n=16, 50%), focusing on inaccuracies, omissions, and hallucinations, though definitions and methods varied widely. Utility analysis (n=8, 25%) examined workflow efficiency, readability, and understandability. Bias analysis (n=7, 22%) considered gender, race, and stigmatizing language. Only one study explicitly conducted patient safety risk analysis (n=1, 3%), using risk matrices. Findings indicate that evaluation frameworks are gradually moving beyond technical performance metrics; however, methodologies remain heterogeneous and lack standardization. Future research should establish consistent error taxonomies, structured risk assessment frameworks, and systematic bias detection methods to enable the safe clinical deployment of LLM-based summarization systems.
24 May 2026
Read appraisal →JMIR medical informatics
Expert Evaluation and Consensus on GPT-4o Summaries of Clinical Letters: Validation and Results of the Framework and Implementation of AI Tools Project
BACKGROUND: Large language models (LLMs) are increasingly used to summarize clinical documents; yet, automated metrics often inadequately capture clinical relevance and safety. In the initial phase of the "Framework and Implementation of AI Tools," an expert-driven, cocreated evaluation methodology was established to assess LLM-generated discharge letter summaries, combining prompt design considerations with intuitive expert appraisal. OBJECTIVE: This study aimed to quantify expert agreement and interrater reliability on LLM summaries of discharge letters, identify frequent and clinically relevant errors, and evaluate practical implications for integrating LLMs into documentation workflows. METHODS: Thirty expert-curated synthetic Dutch discharge letters were summarized. Thirty-one clinicians from Flemish care settings (1 university hospital, 2 private hospitals, and 2 general practice circles) evaluated the summaries. The evaluation framework consisted of 61 binary layout items assessing whether required sections and formatting were correctly present, 33 content items (correct or complete vs incorrect, subcategorizing missing, irrelevant, and hallucinated information), a 4-point global quality rating, and an open comment. Statistical analyses included descriptive statistics, mixed effects ordinal regression on the global score, consensus (agreement per question or letter) percentages, interrater reliability (Cohen κ, intraclass correlation coefficient [ICC], Fleiss κ, and prevalence index), and thematic synthesis of comments. RESULTS: Layout adherence was high (88%), especially in the conclusion section. The positive response rate for content was overall moderate (78%), with the best performance observed in the medical history section and the lowest performance observed in the medication section, which also showed the highest rate of hallucinations and the weakest interrater consensus. Across all sections, missing information was the most common error. Nearly 70% of global ratings were "good" or "very good." Higher positive response rates for content predicted better global scores (β=.079; P<.001), while layout and participant specialty were not relevant to global scoring. Consensus was high for the layout questions (median 96.8%, IQR 90.2%-100%) and somewhat lower for content (median 83.9%, IQR 67.7-96.8), with the lowest agreement in the medication section. Interrater agreement was moderate (median Cohen κ=0.36, IQR 0.29-0.43; range 0.07-0.56), but overall reliability was high (ICC 0.945, 95% CI 0.942-0.948), indicating strong consistency at the global level despite interrater variability. The prevalence index demonstrated that high ICC values were partly driven by the strong prevalence of affirmative responses in layout items, while content items showed more balanced distributions and lower agreement. CONCLUSIONS: Our framework offers a robust approach for evaluating LLM-generated discharge summaries, balancing usability and clinical relevance. Semantic integrity, especially regarding medication details, was identified as a key vulnerability. Perceived overall quality was driven by a positive response rate for content. High ICC values for global score, with lower item-level agreement lead toward the need for clearer, context-specific prompts and standardized evaluation criteria to reduce interrater variability. Human oversight and targeted automated checks for omissions and hallucinations are essential for safe clinical deployment.
14 May 2026
Read appraisal →Scientific reports
Automated phenotyping of ophthalmologic diseases from routine medical records using small language models and the human phenotype ontology (HPO)
Automated phenotyping in ophthalmology requires accurate standardization of clinical terms to facilitate interoperability and research. This study evaluates the suitability of the human phenotype ontology (HPO) for automated extraction of ophthalmic phenotypes from narrative documentation. We developed a locally operated AI pipeline combining text segmentation and negation detection based on a small language model (PHI-4) with a dense retrieval approach using an augmented multilingual HPO catalog. Synonyms were incorporated into the HPO during training on anonymized consecutive physician letters. To validate the pipeline, 175 anterior segment and fundus descriptions from randomly picked medical records were manually annotated with HPO terms as ground truth. Overall, 342 HPO terms were identified manually (on average 2.53 terms per document), with 341 retrieved by the pipeline (on average 2.52 terms per document). Performance metrics showed a median Jaccard similarity of 0.67, precision of 0.83, recall of 0.82, and F1 score of 0.80. These results demonstrate that our AI pipeline effectively extracts standardized HPO terms from free-text ophthalmic findings. Integration of this pipeline into clinical information systems may enhance data interoperability and reduce manual coding workload in ophthalmology practices in the future.
12 May 2026
Read appraisal →PloS one
Pretraining effective T5 generative models for clinical and biomedical applications
This paper presents a study of the impact of corpus selection and vocabulary design on the performance of T5-based language models in clinical and biomedical domains. We introduce five different T5-EHR models, each pretrained from scratch using different combinations of clinical and biomedical corpora alongside domain-specific vocabularies. We evaluated these models across a variety of clinical and biomedical tasks to quantify the impact of pretraining data and vocabulary tokenization choices on downstream performance. Our findings reveal the importance of aligning both pretraining corpus and vocabulary with the target domain. Models pretrained exclusively on clinical data achieve superior performance on clinical tasks, while adding biomedical data contributes only marginal gains in most cases, with a few exceptions. Similarly, the choice of vocabulary significantly influences model performance, with clinical-specific vocabularies outperforming general biomedical vocabularies in tasks requiring a deeper understanding of clinical language. Also, the T5 generative models perform competitively with state-of-the-art discriminative models on several biomedical benchmarks, demonstrating strong generalization to biomedical domain. Overall, these results emphasize that task-specific selection of corpus and vocabulary is essential for optimizing model performance in clinical and biomedical natural language processing (NLP).
18 Apr 2026
Read appraisal →Journal of biomedical informatics
Natural language generation in healthcare: A review of methods and applications
OBJECTIVE: This study presents a systematic review of natural language generation (NLG) methods and applications in the medical domain, providing quantitative and qualitative analyses to answer four key research questions regarding methods, evaluation, applications, and challenges of NLG in healthcare. METHODS: We searched PubMed, ACM Digital Library, Web of Science, Science Direct, Scopus, Embase, and ACL Anthology for NLG-related studies in healthcare from 2018 to 2024. Out of 3,988 research articles, 113 met the inclusion criteria and were analyzed across data modality, model architecture, evaluation metrics, and application domain. RESULTS AND CONCLUSION: NLG in healthcare has grown substantially, with annual publications increasing from 2 in 2018 to 40 in 2024. Of the 113 included studies, text-to-text generation was the most common data modality (65.5%), followed by image-to-text (19.5%) and multimodal-to-text (15.0%). Transformer-based architectures were dominant, especially encoder-decoder models (61.6%). Automatic evaluation metrics such as ROUGE (81.4%) and BLEU (57.5%) were widely used. Human evaluation metrics, such as Likert scales (31.9%), were increasingly adopted. The four most prevalent application domains include summarization (e.g., discharge summaries, radiology reports), clinical documentation, medical dialogue, and data augmentation. The transformer-based large language models (LLMs) and the accumulation of large-scale multimodal clinical datasets have remarkably advanced NLG in healthcare. However, challenges remain in factual consistency, explainability, evaluation robustness, and AI safety. Addressing these challenges is essential for the adoption of NLG in various healthcare applications.
4 Apr 2026
Read appraisal →