Research Appraisals
Evidence-based critical appraisals of the latest medical research, systematically evaluated using Oxford CEBM methodology.
Showing 7 appraisals
Journal of medical Internet research
Exploring Technological Solutions for Interoperability Between Patient Electronic Medical Records and Clinical Registries: Scoping Review
BACKGROUND: The use of electronic medical records (EMRs) and clinical registries has transformed health care delivery by improving data management, care coordination, and research capacity. However, the full potential of these technologies can only be realized through effective interoperability, thereby reducing the burden of manual data entry and enhancing the use of real-world clinical data. OBJECTIVE: This review examines technologies that enable automated data extraction and transfer, which promote interoperability between EMRs and clinical registries. METHODS: A search of PubMed, CINAHL, Embase, and Web of Science, including studies published between January 2013 and April 2025, was registered with Open Science Framework a priori and involved three key concepts: (1) "registry," (2) "electronic medical records," and (3) "interoperability." A 2-phase screen identified studies evaluating technologies that facilitate automated data extraction or interoperability. Automation was defined as fully automated, where data are extracted and transferred without human intervention, or semiautomated, where extraction or transfer is predominantly automated but may include manual validation. Only technologies supporting ongoing database integration were eligible for inclusion. Screening, data extraction, and synthesis were conducted by multiple independent reviewers. Technology experts provided extensive input and guidance throughout to ensure the accuracy and relevance of the extracted information. RESULTS: Overall, 36 studies met the inclusion criteria, representing 12 countries across 5 continents and addressing a wide range of acute and chronic health conditions. Epic was the most frequently reported EMR system, while the most common registry platforms were REDCap (Research Electronic Data Capture; Vanderbilt University), structured query language (SQL) server database, and EMR-embedded solutions. Most approaches centered around extracting data from structured formats (n=18), or a combination of both structured and unstructured formats (n=10), emphasizing the central role of structured EMR data in current automated extraction approaches. CONCLUSIONS: This review advances understanding of interoperability between EMRs and clinical registries by uniquely examining automated and sustainable solutions for data exchange, extending beyond prior work that has largely focused on technologies designed for isolated systems or study-specific data extraction. A novel contribution of this review is the synthesis of context-specific considerations derived from reported implementations, providing a comprehensive overview of how technology selection and implementation are shaped by the context in which they are deployed. While these advancements have reduced reliance on inefficient, error-prone, and resource-intensive manual processes, ongoing challenges in data standardization, seamless integration, and long-term sustainability are compounded by poor and inconsistent reporting across studies. Future efforts should follow comprehensive reporting guidelines, adhere to robust governance principles, and incorporate implementation science frameworks, to not only enable meaningful comparison and synthesis in future research, but also to ensure that technologies can be effectively, feasibly, and sustainably integrated within health care contexts, while upholding the ethical and equitable use of health care data.
26 May 2026
Read appraisal →Studies in health technology and informatics
De-Identification of Free-Text Medical Records Using Large Language Models
This study evaluates large language models (LLMs) for automated de-identification of free-text medical records. Using 150 synthetically generated, personal health information (PHI)-enriched doctor's letters (23.2 PHIs per letter), we compared the full LLaMA-3.1-8B model (BF16, GPU) with a quantized variant (Q8, CPU) under zero-shot and few-shot prompting. Few-shot prompting improved performance from a macro F1 of 0.989 to 0.996 (recall = 0.994); the quantized Q8 model achieved a macro F1 of 0.992 (recall = 0.987) while enabling fully local, GPU-free inference. Although LLMs perform at a level comparable to human annotation, false negatives remain a concern for highly sensitive PHI, such as names. While human annotators typically require 13.4 seconds per PHI, the LLMs de-identified complete letters within seconds, demonstrating an efficiency and scalability advantage alongside GDPR-compliant, high-precision de-identification.
24 May 2026
Read appraisal →Studies in health technology and informatics
Build and Query Indexes of Clinical Documents with Easy-to-Reuse Pipelines
Electronic Health Records are a central source of healthcare data, containing structured data alongside unstructured clinical texts. The latter capture detailed reasoning, observations, treatment plans and clinical evolutions, which are crucial for phenotyping, and real-world evidence generation. Natural language processing enables the extraction, thus the subsequent use, of these crucial elements; however, these extractions remain one-off, study-specific efforts. This is detrimental as the extracted elements could be valuable for future research. We present medkit Seshat, an open-source Python pipeline that: (1) ingests free text, (2) recognizes relevant entities, (3) normalizes them with OMOP vocabularies, (4) builds an index that can either be searched by concept or by document. In addition, we share a flexible web UI to illustrate the interest of built indexes in terms of search, text analysis and export. Seshat aims at facilitating the reuse and adaptation of this prototypical pipeline to various purposes, with the main objective of enabling the secondary use of results of phenotyping campaigns.
24 May 2026
Read appraisal →Studies in health technology and informatics
Advancing Knowledge in Evaluating the Clinical Impact of Large Language Models for Clinical Text Summarization: A Narrative Review
Large language models (LLMs) are increasingly explored for clinical text summarization from electronic health records (EHRs), where they could reduce documentation burden. While promising, errors in generated summaries may propagate misinformation, misrepresent clinical facts, and ultimately pose risks to patient safety. This review examined how the clinical impact of LLM-based summarization has been evaluated across four dimensions: utility, failure, patient safety risks, and bias. Literature was retrieved from PubMed between June 2024 and September 2025. Of the 144 retrieved studies, 32 studies were included in the analysis. Failure analysis was the most frequently reported (n=16, 50%), focusing on inaccuracies, omissions, and hallucinations, though definitions and methods varied widely. Utility analysis (n=8, 25%) examined workflow efficiency, readability, and understandability. Bias analysis (n=7, 22%) considered gender, race, and stigmatizing language. Only one study explicitly conducted patient safety risk analysis (n=1, 3%), using risk matrices. Findings indicate that evaluation frameworks are gradually moving beyond technical performance metrics; however, methodologies remain heterogeneous and lack standardization. Future research should establish consistent error taxonomies, structured risk assessment frameworks, and systematic bias detection methods to enable the safe clinical deployment of LLM-based summarization systems.
24 May 2026
Read appraisal →PloS one
Learning semantic similarity from sentence pairs using hybrid features centric approach and explainable siamese neural networks.
Semantic embeddings play an important role in modern natural language processing because they help models understand meaning beyond individual words. Accurate text similarity is essential for many applications such as search, automated scoring, summarization, and question answering. However, existing methods based on Term Frequency-Inverse Document Frequency (TF-IDF) or simple lexical overlap often fail when sentences differ in length, structure, or word choice. These methods are less in performance, especially when working with short or medium-length sentences where meaning is expressed in different ways. This study explores sentence-level similarity using a Siamese BiLSTM model that learns deep semantic patterns and relationships between two sentences. The model captures contextual meaning, word interactions, and paraphrastic variations more effectively than traditional approaches. Experimental results show that the proposed model achieves the highest performance among machine-learning regressors, with lower errors and improved stability. Compared to TF-IDF, cosine similarity, and feature-based regressors, the Siamese model provides more accurate judgments of semantic closeness with RMSE of 0.16 and MAE of 0.107. Feature-level analysis using TF-IDF, Jaccard similarity, and embedding distances further supports these findings. Explainable AI techniques (SHAP, LIME) confirm model transparency by highlighting meaningful semantic cues and distributing attention across important linguistic features.
23 May 2026
Read appraisal →Scientific reports
Automated phenotyping of ophthalmologic diseases from routine medical records using small language models and the human phenotype ontology (HPO)
Automated phenotyping in ophthalmology requires accurate standardization of clinical terms to facilitate interoperability and research. This study evaluates the suitability of the human phenotype ontology (HPO) for automated extraction of ophthalmic phenotypes from narrative documentation. We developed a locally operated AI pipeline combining text segmentation and negation detection based on a small language model (PHI-4) with a dense retrieval approach using an augmented multilingual HPO catalog. Synonyms were incorporated into the HPO during training on anonymized consecutive physician letters. To validate the pipeline, 175 anterior segment and fundus descriptions from randomly picked medical records were manually annotated with HPO terms as ground truth. Overall, 342 HPO terms were identified manually (on average 2.53 terms per document), with 341 retrieved by the pipeline (on average 2.52 terms per document). Performance metrics showed a median Jaccard similarity of 0.67, precision of 0.83, recall of 0.82, and F1 score of 0.80. These results demonstrate that our AI pipeline effectively extracts standardized HPO terms from free-text ophthalmic findings. Integration of this pipeline into clinical information systems may enhance data interoperability and reduce manual coding workload in ophthalmology practices in the future.
12 May 2026
Read appraisal →JMIR medical informatics
Automated ICD-10-Anchored Classification of Primary Care Text Data: Development and Evaluation of a Custom Multilabel Classifier
BACKGROUND: Electronic medical records are a vast and valuable source of information, useful for tasks such as estimating disease prevalence. However, in routine primary care, much of this information is in free-text format rather than in a structured form and, therefore, not readily amenable to analysis. Manual coding of this textual data is both time-consuming and resource-intensive, making it impractical for large datasets. Although powerful open-source language models offer new opportunities for automated coding, their use on short heterogeneous primary care notes, particularly in German-language settings, remains insufficiently studied. OBJECTIVE: By providing hands-on guidance for applied health researchers, this study aims to demonstrate the effective and accurate automatic classification of free-text notes using a language model fine-tuned for automated International Statistical Classification of Diseases, Tenth Revision (ICD-10) coding. METHODS: Building on the extensive Family Medicine Research Using Electronic Medical Records (FIRE) routine database from the Institute of Primary Care at the University Hospital Zurich and the University of Zurich, we trained a large language model-based multilabel classifier on a dataset of 38,728 free-text notes, which had been manually categorized into 47 classes using specific ICD-10 codes and code ranges or nondiagnostic/ad hoc labels (eg, "unclear diagnosis," "status post"). We stratified the labeled data into training (70%), validation (15%), and posttraining test (15%) sets, ensuring similar label distributions across these sets. Using the Transformers Python library, we trained the model over 10 epochs and evaluated it on the posttraining test dataset. RESULTS: Across 48 classes, the FIRE classifier achieved strong performance on the held-out posttraining set, with F1-scores of 0.85 (micro, overall across all predictions), 0.86 (macro, mean of per-class scores treating classes equally), and 0.84 (weighted, per-class scores weighted by class frequency). CONCLUSIONS: This study demonstrates steps for training open-source large language models and highlights the potential to streamline and scale the extraction of diagnostic information for practical applications. Our model can be robustly deployed, for example, for prescreening and labeling of free-text information, thus potentially reducing the burden of repetitive and error-prone manual handling.
7 Apr 2026
Read appraisal →