Research Appraisals
Evidence-based critical appraisals of the latest medical research, systematically evaluated using Oxford CEBM methodology.
Showing 6 appraisals
Studies in health technology and informatics
Leveraging Large Language Models with Retrieval-Augmented Generation for Semantic Mapping of Clinical Data Lakes to SNOMED CT
Mapping local clinical concepts to standardized terminologies such as SNOMED CT is essential for semantic interoperability and large-scale research, but manual mapping is labor-intensive and difficult to scale. We report a preliminary evaluation of a hybrid approach combining retrieval-augmented generation (RAG) and a large language model (LLM) to support the mapping of hospital datalake concepts to SNOMED CT. Using a dataset of 2,768 concepts annotated by the SIMED across five semantic categories (organisms, healthcare locations, laboratory samples, allergies, and clinical assessments), the pipeline consistently outperformed retrieval-only baselines. For organism concepts, strict accuracy reached 92.8% (weighted 98.0%, top-3 96.7%), while allergy and assessment concepts achieved 84.7% and 72.1% strict accuracy, respectively. Healthcare location and laboratory sample concepts remained more challenging, with strict accuracies of 57.5% and 57.1%, and weighted accuracy varied strongly depending on frequency distributions. Overall, strict accuracy was 72.8%, weighted accuracy 76.1%, and top-3 accuracy 83.1%. These findings suggest that combining RAG with LLM-based reasoning can reduce expert workload while maintaining accuracy, representing a step toward scalable, AI-assisted semantic interoperability in healthcare. The approach relies on a retrieval-augmented generation pipeline using embedding-based candidate retrieval followed by LLM-based disambiguation (GPT-4.1-mini), and is evaluated against a retrieval-only baseline. By improving semantic consistency in clinical data lakes, it indirectly supports interoperable access to patient-level health data.
24 May 2026
Read appraisal →Studies in health technology and informatics
A Federated Benchmark for Clinical Natural Language Processing (FedDRAGON)
We introduce the FedDRAGON challenge, a federated learning benchmark for clinical natural language processing. The challenge includes 12 information extraction tasks where data is extracted from clinical reports from 4 Dutch care centers. Baseline results show that the performance of the federated models surpass single-center performance and approaches that of centralized models. Benchmark, code, and pre-trained LLMs are publicly available.
24 May 2026
Read appraisal →Studies in health technology and informatics
Large Language Models for Automating Conformance to Health-Data Standards: The Interoperability Case of HL7 FHIR and OMOP
Interoperability standards enable different healthcare systems to exchange electronic health information seamlessly, and support large-scale analytics needed for current Secondary uses of healthcare data. Transforming diverse clinical inputs into standards-compliant, analyzable data remains labor-intensive and error-prone. Moreover, the learning curve for their proper exploitation can be steep due to extensive documentation, specific technical requirements, and the need for both semantic and technical integration, making them not fully accessible to practitioners in the healthcare domain unless they undergo extensive training. Our study investigated whether large language models can automate conformance while preserving syntactic validity, semantic correctness, and reproducibility, thereby simplifying the adoption of these standards. Using diabetes measurements as a probe domain, we validated extraction fidelity, coding accuracy against controlled vocabularies, and conformance to the domain's main standards. Our prototype emits FHIR Observations and OMOP MEASUREMENT rows.
24 May 2026
Read appraisal →Studies in health technology and informatics
Large Language Models for Health Knowledge Modelling in Data Interoperability: A Scoping Review of Methods, Standards, and Applications
Achieving interoperability in healthcare requires semantic alignment of data with standards such as SNOMED CT, LOINC, and FHIR. Manual ontology mapping is slow and inconsistent. Evolving Large Language Models (LLMs) can automate knowledge modelling and data harmonisation. This review explores how LLMs are applied to enhance or automate health knowledge modelling for data interoperability. Following PRISMA-ScR guidance, PubMed and Compendex were searched. Studies applying LLMs to ontology alignment, terminology mapping, schema integration, or knowledge-graph construction were included. From 166 records, 20 studies met the inclusion criteria. Most used GPT-4 or BERT-derived models with retrieval-augmented or embedding-based methods. Reported gains included automation, improved mapping precision, and scalability across standards such as FHIR, OMOP, and UMLS. LLMs show strong potential for automating semantic interoperability in health informatics. Standardised evaluation benchmarks and explainable frameworks are essential for trustworthy adoption.
24 May 2026
Read appraisal →JMIR medical informatics
A Practical Approach to Assessing the Completeness of Electronic Health Records for Medical Research: Data Quality Study
BACKGROUND: Data quality is the degree to which data are fit for their intended purpose and is described using quality dimensions. The increased use of medical data in clinical research and medical artificial intelligence development has rendered data quality assessment essential. Despite existing data quality definitions, frameworks, and tools, data quality assessment in real-world settings faces multiple challenges. This stems from a lack of understanding of how to assess real-world data quality and interpret the results. Therefore, practical approaches to data quality assessment are needed that are appropriate for diverse data environments, intended uses, quality dimensions, and requirements. OBJECTIVE: This study proposes a practical approach for assessing the completeness of electronic health records (EHRs) for medical research. This approach integrates structural completeness, rule-based assessment, and descriptive analyses of completeness and data diversity to clarify how data quality can be measured and meaningfully interpreted in practice. METHODS: The completeness of a large-scale EHR dataset from Gachon University Gil Medical Center was evaluated covering January 2005 to December 2023. Completeness was assessed using a three-part approach comprising (1) structural completeness assessment, (2) rule-based assessment, and (3) descriptive analyses of completeness and data diversity. Assessments were conducted using clinical data quality assessment tools. This practical approach was used to assess EHR completeness for medical research from 1,798,153 patient records. RESULTS: In the structural assessment, 12.8% (5/39) of the data tables were unavailable, indicating limited capturing of clinician free-text data. The rule-based assessment identified substantial missingness in vocabulary fields (38/124, 30.6%) and missing or special characteristic values in relation to observations (3,643,581/15,313,287, 23.8%), measurements (25,583,622/642,623,715, 4%), care sites (28/1715, 1.6%), and deaths (117/34,330, 0.3%). Descriptive analyses demonstrated a balanced gender distribution (886,489/1,798,153, 49.3% male and 911,664/1,798,153, 50.7% female) and a predominantly Korean racial distribution (1,739,628/1,798,153, 96.7%). Collectively, these findings illustrate the completeness quality of a multiperspective completeness assessment for medical research. CONCLUSIONS: This study demonstrates how data quality dimensions can be measured in practice through a real-world completeness assessment. This practical approach enables evaluation of EHR completeness and provides insights into data quality. Its findings have implications for researchers conducting data quality assessments and applying quality dimensions in medical research.
22 May 2026
Read appraisal →Journal of the American Medical Informatics Association : JAMIA
Evaluating the potential of fast healthcare interoperability resources for clinical registry data submission.
OBJECTIVE: We conducted the Clinical Registry Extraction and Data Submission (CREDS) project to evaluate the readiness of HL7 Fast Healthcare Interoperability Resources (FHIR) for provisioning data from health information systems for the American College of Cardiology Cardiac Catheterization Percutaneous Coronary Intervention (CathPCI) Registry. MATERIALS AND METHODS: The CREDS project had 3 workstreams: (1) evaluation of the readiness of clinical documentation for data transforms, (2) modeling of a FHIR-based clinical workflow for registry data submission, and (3) development and demonstration of a CREDS FHIR implementation for registry data submission. RESULTS: Of the 344 data concepts comprising the CathPCI Registry, only 111 (32%) were sufficiently discrete to be listed in the CathPCI Data Dictionary with a terminology mapping. Cardiologist informaticians identified an additional 42 concepts suitable for provisioning via a FHIR payload. The resulting notional workflow combined FHIR-based data assembly with manual chart abstraction of compound, summative, and complex clinical concepts. A CathPCI FHIR StructureDefinition artifact was authored, incorporated into a CREDS FHIR Implementation Guide, and balloted to Standard for Trial Use status. DISCUSSION: CREDS demonstrated both potential and limitations for using FHIR for registry data submission. The largest technical impediment was the volume of code (>11 000 lines) for the FHIR StructureDefinition. Lack of regularized clinical vocabularies, reliance of registries on complex clinical concepts, and absence of FHIR infrastructure must be overcome before CREDS can be used at scale. CONCLUSION: CREDS demonstrated proof-of-concept FHIR-based provisioning of clinical data for registry submission. All artifacts are open source to inform others with similar interests.
9 May 2026
Read appraisal →