Research Appraisals
Evidence-based critical appraisals of the latest medical research, systematically evaluated using Oxford CEBM methodology.
Showing 33 appraisals
International journal of medical informatics
From algorithmic innovation to clinical deployment: A systematic review of methodological gaps limiting federated learning in healthcare
OBJECTIVE: Federated learning (FL) is a distributed machine learning paradigm designed to enable model training across decentralized data sources without requiring data centralization. This review critically examines FL as a methodological approach in biomedical informatics, summarizing its conceptual foundations, methodological variants, successes, and limitations, and identifying directions for future research. METHODS: We conducted a systematic methodological review following PRISMA 2020 guidelines. Searches were performed in PubMed, Web of Science, Scopus, IEEE Xplore, and Google Scholar for studies published between January 2017 and March 2025. Articles were included if they proposed, extended, or critically analyzed FL methods for biomedical or life science data. Methods were categorized by federation topology, optimization strategy, data heterogeneity handling, privacy mechanisms, evaluation design, and translational readiness. RESULTS: From 8,412 records, 97 articles met inclusion criteria. Rapid methodological innovation exists in optimization and privacy mechanisms, while support for design-time evaluation and governance remains limited. CONCLUSION: Federated learning represents a significant methodological advance in biomedical informatics but current implementations address only a subset of translational challenges. Future work must integrate study design, evaluation, interpretability, and governance into FL methods.
16 June 2026
Read appraisal →JMIR research protocols
Effectiveness of Guideline-Based Clinical Decision Support Systems: Protocol for a Systematic Review
BACKGROUND: Clinical guidelines (CGs) standardize care through evidence-based recommendations, while clinical decision support systems (CDSS) can assist in applying these guidelines to individual patients. The scientific basis for the decisions offered by decision support systems is often not explicitly stated or not clearly specified in the literature on CDSS. Therefore, a systematic examination of the literature is needed to map the current state of CDSS, with a particular focus on the integration of CGs. OBJECTIVE: This study aims to systematically collect, describe, and synthesize evidence of randomized controlled studies of interventions using CDSS with a well-defined integration of evidence-based CGs and evaluating direct medical outcomes. METHODS: This systematic review adheres to the PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) and PRISMA-ScR (Preferred Reporting Items for Systematic Reviews and Meta-Analyses extension for Scoping Reviews) checklists. The eligibility criteria for this review are defined using the patient, intervention, control, outcome, and study design framework, including studies involving patients with any medical condition or disease. Study interventions need to include guideline-based CDSS, encompassing all types of interventions used for treatment. Each intervention must provide a sufficiently accessible technical description, including the types of data and algorithms used for decision-support procedures. The guidelines used within these CDSS interventions must be derived from a clearly defined, evidence-based guideline development process published by a discernible guideline-producing body. Studies must use a randomized controlled study design. Only studies evaluating the effectiveness of the interventions on direct medical outcomes are included. Web of Science, including MEDLINE, and Scopus will be searched with search expressions aligned with the eligibility criteria. RESULTS: On August 11, 2022, the initial search was conducted on Web of Science and Scopus. From a total of 6203 records, 1347 were removed prior to screening as duplicates, 2506 records were excluded during the first screening step, and 2291 were excluded during the second step. Next, 41 papers were excluded based on full-text review, and 18 papers were finally included in the review following this initial search. This review explores whether CDSS based on CGs can improve clinical outcomes, although their effectiveness may vary depending on various factors. Potential limitations, such as high study heterogeneity, have already been identified. An update of the review has been started in April 2025. CONCLUSIONS: To our knowledge, this is the first rigorous systematic review on the effectiveness of guideline-based decision support systems in which the technical integration and algorithmic embedding of CGs have been described or can be inferred from secondary literature. With this review, we aim to address this gap by providing a detailed analysis of existing research and identifying best practices, challenges, and areas for future investigation.
14 June 2026
Read appraisal →Med (New York, N.Y.)
Rethinking healthcare data interoperability in the age of large language models
Electronic health records contain extensive real-world clinical data, but their effective use is hindered by data heterogeneity and interoperability challenges. Traditional post hoc standardization is costly and labor-intensive and reduces data granularity. Large language models enable analysis of unstructured data without full harmonization but lack precision for some tasks. We propose a hybrid strategy that combines large-language-model-based analysis of legacy data with prospectively standardized data, offering a scalable alternative that challenges the need for retrospective data harmonization and improves interoperability.
13 June 2026
Read appraisal →JMIR medical informatics
Blockchain Smart Contracts for Automating Clinical Trials: Systematic Review and Proposed System Architecture
BACKGROUND: Blockchain technologies have revolutionized the financial sector through their ability to generate immutable, cryptographically secure records. Clinical trials and health care data possess several synergies with those of the financial sector, specifically pertaining to the importance of tamper-resistant recording of processes. The evolution of blockchain to autonomously execute tasks contingent upon predefined contractual terms via smart contracts (SCs) allows a dynamic chain of interlinked events to unfold independently and in sequence, with time-stamped records. In recent years, mistrust in clinical trial data has grown significantly. Recording the entire clinical trial lifecycle from application, registration, recruitment to conduct, finance management, statistical analysis, and reporting in an immutable, cryptographically secure ledger with SC execution of trial processes could limit the potential for human intervention and tampering. This would produce a time-stamped record of all events within the trial lifecycle. Leveraging the capabilities of SCs could alleviate recruitment challenges and address ongoing concerns regarding data transparency, ownership, and integrity that currently undermine clinical trial processes. OBJECTIVE: This study aimed to review the existing literature on SC applications in clinical trials and propose a system architecture for using SCs to automate key processes throughout the clinical trial lifecycle. METHODS: A systematic search was conducted in accordance with the PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) guidelines, identifying peer-reviewed studies and open-source repositories pertaining to the implementation of SCs in clinical trials. Data were extracted specific to the stage of the trial lifecycle described, SC architecture, and technical specifications for real-world implementation. Data were synthesized to propose an architecture for automating clinical trial processes within the lifecycle using SCs. RESULTS: A total of 144 records were screened; 10 studies met the inclusion criteria. Most implementations used private Ethereum-based networks (7/10, 70%). Reported applications included automated patient matching (5/10, 50%), consent management with dynamic permissioning (6/10, 60%), protocol enforcement and time-stamped audit logs (9/10, 90%), adverse event reporting (3/10, 30%), and financial or workflow automation (3/10, 30%). SC-based recruitment systems demonstrated rapid matching performance (eg, 6000 simulated patients matched in 2.13 s). However, all included systems were prototypes or simulations, and none were tested in real-world regulatory settings. Scalability, interoperability limitations, regulatory ambiguity (eg, General Data Protection Regulation right-to-erasure conflicts), and high infrastructural complexity were common gaps noted across studies. CONCLUSIONS: Current evidence suggests that SCs can enhance transparency, traceability, and automation throughout the clinical trial lifecycle. However, the literature remains dominated by simulation-based prototypes, primarily Ethereum-dependent architectures, and lacks analyses of cost-effectiveness, governance, and integration with institutional workflows. Future research should evaluate hybrid architectures, develop interoperability standards, and assess regulatory and ethical implications in real deployments.
30 May 2026
Read appraisal →Science robotics
Embodied or virtually represented: Navigating the embodiment debate in human-robot interaction.
The validity of virtually represented robots in HRI experiments depends on when, where, and for whom it matters.
30 May 2026
Read appraisal →European radiology experimental
Automated identification of MRI series using a hierarchical modular machine-learning pipeline
OBJECTIVE: The volume and diversity of large MR imaging datasets require efficient automated labelling tools for cataloguing MR series, as manual annotation is impractical and costly. However, relying on DICOM header fields alone is unreliable because sequence descriptors are heterogeneous and locally defined, frequently missing or incorrect, and may be altered or removed during anonymisation. MATERIALS AND METHODS: We developed an AI-based modular model to classify MR series. The pipeline comprises five sequential classifiers (Family, Weighting, Fat Suppression, Contrast, and Others) and was trained and tested on 18,181 MRI series from the multicentre PRIMAGE repository. The dataset was split by patient into 80% training/validation and 20% testing; within the training/validation subset, five-fold cross-validation was used. With the exception of contrast classification, all modules used DICOM tag-based machine learning models (CatBoost/Random Forest), while the Contrast classifier incorporated image analysis using a pretrained single-slice ResNet-50. Ethical approval for the study was obtained. RESULTS: Accuracy was 0.994 for Weighting, 0.984 for Family, 0.959 for Fat suppression and 0.958 for Others; the Contrast classifier recorded 0.841. Overall, the end-to-end classification yielded a weighted F1 of 0.849 (CI: 0.837-0.861) and an accuracy of 0.853 (CI: 0.841-0.865). CONCLUSION: The proposed approach provides a reliable and scalable solution for labelling large, heterogeneous MRI datasets across multiple anatomical regions. The pipeline achieved excellent performance for Weighting and Family classification, solid performance for Fat Suppression and 'Others'. However, Contrast classification remains the main limitation and warrants further refinement and/or additional modules. RELEVANCE STATEMENT: Reliable, automated MRI sequence labelling enables faster, reproducible cohort selection and protocol harmonisation in large archives, supporting downstream clinical research and AI tools while reducing manual curation effort and error. KEY POINTS: Multicentre PRIMAGE cohort (18,181 MR series) enabling evaluation in a heterogeneous setting. Machine learning model combining DICOM metadata and image features. High accuracy in series classification: up to 0.994 on key tasks. A scalable pipeline reduces manual annotation workload for radiologists.
30 May 2026
Read appraisal →PloS one
A hybrid machine learning and explainable AI framework for optimizing risk-based authentication
As online platforms continue to grow, the need for strong authentication mechanisms becomes increasingly important to protect sensitive information and networks. Risk-Based Authentication (RBA) is an adaptive approach that dynamically adjusts authentication decisions based on user behavior and contextual information, thereby improving both security and user experience. This study proposes a hybrid RBA framework that integrates machine learning ensemble techniques, fuzzy logic, clustering, and optimization to enhance account takeover detection and dynamic risk assessment. The ensemble classifier, combining Gradient Boosting, SVM, and XGBoost, predicts the probability of account compromise based on login behavior, device attributes, and network information. K-Means clustering is used to generate initial risk thresholds (low, medium, and high), which are further refined using a fuzzy logic system to map probabilities to risk levels. The L-BFGS-B optimization algorithm is employed to fine-tune fuzzy membership boundaries and improve threshold consistency. Experimental results demonstrate strong performance, achieving 97.77% accuracy, 99.41% precision, 98.04% recall, 98.72% F1-score, and an EER of 0.0303. On large-scale datasets ranging from 2M to 30M records, the proposed framework demonstrates consistent improvement in authentication decisions. For the 2M dataset, Allow Login actions increase from 349,432-349,923, while Deny Login actions decrease from 1,462-1,228, along with a slight reduction in additional authentication prompts. Furthermore, the use of Explainable AI techniques, particularly SHAP, enhances the transparency and interpretability of the model, supporting more informed decision-making. Overall, the proposed framework is accurate, adaptive, and suitable for real-world risk-based authentication applications.
27 May 2026
Read appraisal →Studies in health technology and informatics
A Federated Benchmark for Clinical Natural Language Processing (FedDRAGON)
We introduce the FedDRAGON challenge, a federated learning benchmark for clinical natural language processing. The challenge includes 12 information extraction tasks where data is extracted from clinical reports from 4 Dutch care centers. Baseline results show that the performance of the federated models surpass single-center performance and approaches that of centralized models. Benchmark, code, and pre-trained LLMs are publicly available.
24 May 2026
Read appraisal →Studies in health technology and informatics
Large Language Models for Automating Conformance to Health-Data Standards: The Interoperability Case of HL7 FHIR and OMOP
Interoperability standards enable different healthcare systems to exchange electronic health information seamlessly, and support large-scale analytics needed for current Secondary uses of healthcare data. Transforming diverse clinical inputs into standards-compliant, analyzable data remains labor-intensive and error-prone. Moreover, the learning curve for their proper exploitation can be steep due to extensive documentation, specific technical requirements, and the need for both semantic and technical integration, making them not fully accessible to practitioners in the healthcare domain unless they undergo extensive training. Our study investigated whether large language models can automate conformance while preserving syntactic validity, semantic correctness, and reproducibility, thereby simplifying the adoption of these standards. Using diabetes measurements as a probe domain, we validated extraction fidelity, coding accuracy against controlled vocabularies, and conformance to the domain's main standards. Our prototype emits FHIR Observations and OMOP MEASUREMENT rows.
24 May 2026
Read appraisal →Studies in health technology and informatics
MAP-CARE: Enhancing Cross-Lingual Medical Intervention Terms
Traditional keyword-based or single-language systems are not able to align data from surgical and interventional procedures, especially from non-English healthcare systems. This study aims to develop a pipeline for cross-lingual retrieval and integration of medical procedures data. Methods: MAP-CARE is a novel framework that leverages Large Language Models (LLMs) for translating and transforming medical procedures into a unified multilingual embedding space. Semantic embeddings are used to enhance retrieval accuracy and interoperability across languages and healthcare systems. Results: MAP-CARE demonstrated high accuracy in the translation and mapping of clinical terms. Its cross-language translation performance proved robust, achieving up to 90% accuracy in translating procedure classification codes across English, German, French, and Italian-when considering the correct term among the top five retrieved. Conclusion / Discussion: MAP-CARE offers a flexible, scalable, and robust solution for the multilingual and cross-system integration of medical procedural data.
24 May 2026
Read appraisal →Studies in health technology and informatics
Large Language Models for Health Knowledge Modelling in Data Interoperability: A Scoping Review of Methods, Standards, and Applications
Achieving interoperability in healthcare requires semantic alignment of data with standards such as SNOMED CT, LOINC, and FHIR. Manual ontology mapping is slow and inconsistent. Evolving Large Language Models (LLMs) can automate knowledge modelling and data harmonisation. This review explores how LLMs are applied to enhance or automate health knowledge modelling for data interoperability. Following PRISMA-ScR guidance, PubMed and Compendex were searched. Studies applying LLMs to ontology alignment, terminology mapping, schema integration, or knowledge-graph construction were included. From 166 records, 20 studies met the inclusion criteria. Most used GPT-4 or BERT-derived models with retrieval-augmented or embedding-based methods. Reported gains included automation, improved mapping precision, and scalability across standards such as FHIR, OMOP, and UMLS. LLMs show strong potential for automating semantic interoperability in health informatics. Standardised evaluation benchmarks and explainable frameworks are essential for trustworthy adoption.
24 May 2026
Read appraisal →PloS one
Learning semantic similarity from sentence pairs using hybrid features centric approach and explainable siamese neural networks.
Semantic embeddings play an important role in modern natural language processing because they help models understand meaning beyond individual words. Accurate text similarity is essential for many applications such as search, automated scoring, summarization, and question answering. However, existing methods based on Term Frequency-Inverse Document Frequency (TF-IDF) or simple lexical overlap often fail when sentences differ in length, structure, or word choice. These methods are less in performance, especially when working with short or medium-length sentences where meaning is expressed in different ways. This study explores sentence-level similarity using a Siamese BiLSTM model that learns deep semantic patterns and relationships between two sentences. The model captures contextual meaning, word interactions, and paraphrastic variations more effectively than traditional approaches. Experimental results show that the proposed model achieves the highest performance among machine-learning regressors, with lower errors and improved stability. Compared to TF-IDF, cosine similarity, and feature-based regressors, the Siamese model provides more accurate judgments of semantic closeness with RMSE of 0.16 and MAE of 0.107. Feature-level analysis using TF-IDF, Jaccard similarity, and embedding distances further supports these findings. Explainable AI techniques (SHAP, LIME) confirm model transparency by highlighting meaningful semantic cues and distributing attention across important linguistic features.
23 May 2026
Read appraisal →JMIR mental health
Automated Safety Testing and Reporting Application for Conversational Safety Monitoring of Generative AI Tools for Mental Health: Development and Validation Study
BACKGROUND: Artificial intelligence (AI)-based conversational tools are rapidly expanding within mental health care as a means of increasing access and scalability. At the same time, these systems introduce distinct safety risks arising from both user disclosures (eg, self-harm ideation) and inappropriate or inadequate AI responses. OBJECTIVE: This study aimed to develop and evaluate the Automated Safety Testing and Reporting Application (ASTRA), an external system intended to identify clinically relevant risk behaviors across entire AI-mediated mental health conversations. METHODS: ASTRA was tested on a dataset of 100 synthetic therapeutic conversations written by licensed clinicians to reflect risk behaviors and harmful responses between users and AI tools. Conversations varied in length and included both subtle and overt risk behavior examples across 8 predefined categories. Human coder consensus ratings served as the reference standard. ASTRA's classifications were evaluated across 2 prompt iterations using standard diagnostic performance metrics and agreement statistics. RESULTS: ASTRA demonstrated consistently high concordance with expert human ratings across all categories. Accuracy exceeded 0.90 for all risk behavior categories examined, with specificity uniformly high and sensitivity varying by category (range 0.55-1.00). Agreement beyond chance was substantial to almost perfect between ASTRA and human raters (κ=0.65-1.00). Detection of user self-harm indicators was particularly accurate, even in conversations where risk was expressed subtly. CONCLUSIONS: In this initial validation study, ASTRA reliably identified multiple forms of mental health-related risk behaviors at the conversation level. These findings support the feasibility of independent safety monitoring systems as a complement to AI tools used in mental health contexts and underscore the need for further evaluation using larger and real-world datasets.
21 May 2026
Read appraisal →JMIR medical education
A Competency Framework for Medical AI Education: Mixed Methods Study
BACKGROUND: Although artificial intelligence (AI) is increasingly adopted in health care, clinicians face barriers, including insufficient understanding, limited trust, and challenges in interpreting AI outputs. Existing frameworks, such as the United Nations Educational, Scientific and Cultural Organization (UNESCO) AI competency framework, lack clinical specificity. Additionally, there remains limited evidence on framework-based training programs for medical professionals. OBJECTIVE: This study aimed to (1) develop a medical AI competency framework and (2) apply the framework to design an AI training program and pilot 1 module. METHODS: We conducted a mixed methods study comprising 2 phases. In Study 1, we developed a medical AI competency framework by integrating the UNESCO AI framework with the Miller pyramid model. The framework was refined through expert input from 24 stakeholders (6 hospital administrators, 8 medical professionals, and 10 university instructors). Expert responses were analyzed using deductive content analysis based on predefined codebooks. In study 2, we designed an AI training program based on the framework, and evaluated it using a 2-round Delphi process. Nine educators with expertise in instructional design, medical education, and AI participated in the Delphi study. Consensus in round 1 was defined as an IQR≤1, agreement score proportion >75%, and full-score frequency >49.23%; in round 2, consensus was defined as agreement score proportion >80%. A pilot workshop with 28 participants and 4 instructors assessed the feasibility of 1 module using self-reported measures of satisfaction, engagement, and confidence. RESULTS: A 6D 4-level medical AI competency framework was developed. Among the 24 experts, 19 (79.17%) mentioned competencies related to AI foundations, and 23 (95.83%) mentioned competencies related to application skills. The framework was translated into a 5-module training program, covering patient-centered and ethical AI, privacy and security, medical AI applications, bias and health equity, and generative AI in health care. Each module included 5 elements: content, learning goals, teaching activities, learning resources, and assessment. The Delphi process achieved complete consensus across all 25 elements of the training program. The pilot workshop indicated high participant satisfaction (mean 4.00, SD 0.52), good engagement (mean 3.80, SD 0.71-mean 4.05, SD 0.51), and moderate self-reported confidence (mean 3.63, SD 0.53). These findings suggest that the module was feasible, although outcomes should be interpreted cautiously given the self-reported and short-term nature of the evaluation. CONCLUSIONS: The framework provides a structured reference for AI training program design in medical education. The workshop findings provide preliminary support for the feasibility of the program's technical module, while also highlighting the need for broader and longer-term evaluations. Future work should expand the framework and training program to new regions and delivery formats (eg, semester-long courses and continuing medical education) and evaluate their long-term impact.
21 May 2026
Read appraisal →PloS one
Design and evaluation of a resilient IBN architecture: Integrating post-quantum cryptography with adaptive threat detection using machine learning
As the domain of network security keeps on evolving rapidly, especially in sensitive areas such as healthcare systems, the demand for reliable device verification, controlling access, and spotting threats is growing sharply. This paper presents the design, implementation, and systematic evaluation of an improved Intent-Based Networking (IBN) system that integrates post-quantum cryptography, certificate-based identity management, and machine learning-based anomaly detection within a unified framework. The system incorporates SPHINCS+ post-quantum digital signatures for quantum-resilient authentication, X.509 certificate lifecycle management for establishing device trust, and hardware-aware cryptographic operations to maintain efficiency. It further enforces fine-grained access policies using Role-Based Access Control (RBAC) augmented with Multi-Factor Authentication (MFA), ensuring strong access governance across network segments. For early threat detection, machine learning models such as Isolation Forest and MiniBatch KMeans are employed to learn communication patterns and detect anomalous device behavior. Additionally, event logs are maintained using asynchronous, hash-chained logging mechanisms inspired by blockchain principles, ensuring auditability and data integrity. To address evaluation transparency and rigor, the framework is assessed using a controlled prototype testbed with explicitly defined traffic features and reproducible experimental settings. The evaluation considers cryptographic correctness, access control performance, anomaly detection capability, and scalability under increasing workloads. Experimental results demonstrate 100% success in post-quantum signature generation and verification, effective anomaly detection with no observed false negatives in the evaluated scenarios, and stable log-processing throughput as the number of events grows. Importantly, this work does not claim novelty in individual components, but contributes through the system-level integration and empirical evaluation of a quantum-safe, ML-assisted IBN security architecture. The findings highlight key trade-offs between security enforcement and usability, while also identifying limitations such as certificate expiry handling gaps, conservative policy behavior, and lack of large-scale statistical validation. These observations establish a reproducible baseline and motivate future work toward statistically rigorous validation, real-world deployment, and adaptive policy optimization.
17 May 2026
Read appraisal →JMIR medical informatics
A Multiassessment and Multiprofessional Agents Approach for Medical Chatbot Risk Estimation: Development and Evaluation Study
BACKGROUND: Assessing chatbot responses across 3 domains-medical, ethical, and legal-is essential to ensuring the safe use of artificial intelligence in health care. Although advancements in the use of large language models (LLMs) show significant improvements in evaluating question-answer datasets, such as multiple-choice medical exams, existing systems use general LLMs without incorporating specialized domain knowledge. They rely on standardized instructions without integrating real-world information, and ensemble methods such as majority voting fail to resolve disagreements among agents, resulting in misclassification and challenges in risk assessment. OBJECTIVE: This study aims to design, develop, and evaluate a synergistic approach for assessing risks associated with chatbot responses using multiassessment (MA) and multiprofessional agents (MPAs). METHODS: We designed and developed an approach consisting of MA and MPA, specifically initial assessment (MA1), which internalizes 3 roles and provides an initial risk estimation, and final assessment (MA3), which aims to reach a final consensus based on the previous assessments (MA1 and MA2), with each using 1 LLM. The verification assessment (MA2) incorporates an MPA or role-based LLM specialized agents for each risk domain (medical, ethical, and legal). We evaluated the proposed approach using the MedNLP-CHAT (Medical Natural Language Processing for AI Chat) corpus (N=226; 100 train, 126 test), covering baseline, enhanced prompt, embedding-based search, and retrieval-augmented generation (RAG). Primary metrics included macro F1-score and joint accuracy to evaluate system performance, along with CI and paired macro F1-score difference (Δ) as supporting metrics to assess the approach's effectiveness. RESULTS: The MA-MPA framework integrated with RAG achieved the highest average macro F1-score of 0.800 across risk domains and a joint accuracy of 76 (60.3%) correct predictions across all risk domains out of 126 question-answer pairs, with notable improvements over the best reported Eighteenth NII Testbeds and Community for Information Access Research Project (NTCIR-18) MedNLP-CHAT systems in the ethical (+0.252) and legal (+0.096) risk domains, while the medical domain showed a modest increase of +0.070. The MA approach contributed the largest gains, particularly from MA1 to MA2, with paired macro F1-score gains ranging from +0.176 to +0.214 across systems. The MPA approach performed better when integrated with MA and external knowledge, with paired bootstrap estimates showing a gain of +0.037 (95% CI 0.003-0.074) over baseline; however, joint accuracy gains were not evident (95% CI -2.9% to 7.7%), and gains relative to the enhanced prompt were small. Notably, MA alone achieved higher joint accuracy than RAG (62.7% vs 60.3%), indicating a metric-specific trade-off rather than consistent superiority across all metrics. CONCLUSIONS: The MA-MPA approach shows potential for improving risk estimation in chatbot responses. The results suggest that the framework is particularly useful for enhancing balanced overall performance, especially when combined with external knowledge, although the medical risk domain remains challenging. Furthermore, more specialized LLMs may further improve contextually grounded risk estimation.
17 May 2026
Read appraisal →Philosophical transactions. Series A, Mathematical, physical, and engineering sciences
Levels of analysis for large language models
Modern artificial intelligence systems, such as large language models, are increasingly powerful but also increasingly hard to understand. Recognizing this problem as analogous to the historical difficulties in understanding the human mind, we argue that methods developed in cognitive science can be useful for understanding large language models. We propose a framework for applying these methods based on the levels of analysis that David Marr proposed for studying information processing systems. By revisiting established cognitive science techniques relevant to each level and illustrating their potential to yield insights into the behaviour and internal organization of large language models, we aim to provide a toolkit for making sense of these new kinds of minds. This article is part of the theme issue 'World models in natural and artificial intelligence'.
15 May 2026
Read appraisal →JMIR formative research
Automatic Speech Recognition and Large Language Models for Multilingual Pathology Report Generation: Proof-of-Concept Study
BACKGROUND: Accurate transcription of pathology gross examination dictation is important for clinical documentation, but multilingual dictation remains challenging in settings where clinicians mix Chinese and English while final pathology reports are written in English. OBJECTIVE: This study aimed to evaluate whether a Whisper-based automatic speech recognition (ASR) pipeline guided by contextual system messages and combined with open-source large language models (LLMs; Qwen2:72b, Llama3.1:70b, Gemma2:27b) could improve multilingual (Chinese-English) pathology dictation transcription accuracy and generate clinically appropriate English gross description reports. METHODS: We conducted a controlled proof-of-concept study using 125 simulated mixed Chinese-English pathology gross examination audio recordings created by physicians or pathologists. Audio recordings were transcribed using Whisper ASR with and without a contextual system message. The ASR transcripts were then converted into English gross description reports using 3 open-source LLMs: Qwen2:72b, Llama3.1:70b, and Gemma2:27b. Outcomes included character error rate, Bilingual Evaluation Understudy, Recall-Oriented Understudy for Gisting Evaluation (ROUGE)-1, ROUGE-2, ROUGE-L, Metric for Evaluation of Translation with Explicit Ordering, pathologist Win-Tie-Lose rankings, report-level error categories, inference time, and interrater agreement. RESULTS: The ASR contextual system message reduced the mean character error rate from 0.344 (SD 0.176; 95% CI 0.313-0.375) to 0.066 (SD 0.100; 95% CI 0.048-0.084; P<.001). Qwen2:72b achieved the highest automated metric scores, including a Bilingual Evaluation Understudy of 0.644 (SD 0.307), ROUGE-1 of 0.866 (SD 0.163), ROUGE-2 of 0.771 (SD 0.235), ROUGE-L of 0.842 (SD 0.178), and Metric for Evaluation of Translation with Explicit Ordering of 0.805 (SD 0.214). Pathologist-coded total error rates were 16.8% (21/125) for Qwen2:72b, 45.6% (57/125) for Llama3.1:70b, and 92.8% (116/125) for Gemma2:27b. The exact agreement between the 2 pathologists across full ranking categories was 76.8% (96/125; Cohen κ=0.668), and agreement on the top-ranked model or tied top group was 81.6% (102/125; Cohen κ=0.722). CONCLUSIONS: In this proof-of-concept evaluation, contextual prompting improved ASR transcription accuracy, and Qwen2:72b generated the most accurate English pathology reports among the evaluated LLMs. However, the study used simulated audio recordings, a local vocabulary prompt, and report-level rather than term-level clinical annotation. LLM-generated reports should therefore be considered draft documentation requiring pathologist verification, and prospective validation in real clinical workflows is needed before clinical deployment.
15 May 2026
Read appraisal →PloS one
Extracting user profile via large language models and ontologies
Extracting user-specific profiles that include general personal information, such as hobbies, occupation, or age, is a valuable asset for systems like recommendation engines or personalized chatbots. Currently, most approaches moved toward exploiting the capabilities of large language models (LLMs) on rich languages while less-resourced languages still present significant opportunities for exploration and improvement. In this research, we present a multi-step approach for profile extraction from dialogue systems in Persian as a use case. Through an extensive set of experiments and analyses on various models, we show that LLMs struggle with these languages due to limited language-specific resources and complex linguistic structures. To address this, we propose a hybrid method that combines techniques such as slot filling, in-context learning, and ontology-based inference. Our final results demonstrate a significant improvement over current state-of-the-art models, including LLMs with few-shot examples and even their fine-tuned version. Our method achieves an F-score of 90.46, outperforming GPT-4o and Llama-3-70B by an absolute difference of 17.08 and 24.51 respectively. Our system can also detect inconsistencies in presented information in which our performance substantially exceeds the best performance of GPT-4o and Llama-3-70B with an accuracy of 92%. It is 21% absolutely better than GPT-4o and 47% better than Llama.
13 May 2026
Read appraisal →Scientific reports
Automated phenotyping of ophthalmologic diseases from routine medical records using small language models and the human phenotype ontology (HPO)
Automated phenotyping in ophthalmology requires accurate standardization of clinical terms to facilitate interoperability and research. This study evaluates the suitability of the human phenotype ontology (HPO) for automated extraction of ophthalmic phenotypes from narrative documentation. We developed a locally operated AI pipeline combining text segmentation and negation detection based on a small language model (PHI-4) with a dense retrieval approach using an augmented multilingual HPO catalog. Synonyms were incorporated into the HPO during training on anonymized consecutive physician letters. To validate the pipeline, 175 anterior segment and fundus descriptions from randomly picked medical records were manually annotated with HPO terms as ground truth. Overall, 342 HPO terms were identified manually (on average 2.53 terms per document), with 341 retrieved by the pipeline (on average 2.52 terms per document). Performance metrics showed a median Jaccard similarity of 0.67, precision of 0.83, recall of 0.82, and F1 score of 0.80. These results demonstrate that our AI pipeline effectively extracts standardized HPO terms from free-text ophthalmic findings. Integration of this pipeline into clinical information systems may enhance data interoperability and reduce manual coding workload in ophthalmology practices in the future.
12 May 2026
Read appraisal →Artificial intelligence in medicine
Learning with less: A survey of deep learning in medical imaging under varying supervision levels
The rapid evolution of deep learning has significantly advanced the field of medical image analysis. However, despite these achievements, further progress is hindered by the scarcity of large, well-annotated datasets. To overcome this limitation, recent years have seen a growing focus on developing methods that leverage varying levels of supervision as alternatives to fully supervised learning. Instead of relying solely on complete annotations, these approaches employ partial annotations or other supervision strategies to train deep learning models. This paper presents a systematic review of deep learning methods for medical image analysis under different levels of supervision. To this end, we categorize these methods based on the level of supervision they rely on, encompassing categories such as no supervision, inexact supervision, incomplete supervision, inaccurate supervision, and only limited supervision. We further divide these categories into finer subcategories. For example, we categorize inexact supervision into multiple instance learning and learning with weak annotations. Similarly, we categorize incomplete supervision into semi-supervised learning, active learning, and domain-adaptive learning and so on. Furthermore, we systematically summarize commonly used datasets for different types of supervision in deep learning for medical image analysis and explore future research directions to conclude this survey.
9 May 2026
Read appraisal →Medical image analysis
Robust fine-grained echocardiographic view classification with supervised contrastive learning
Accurate classification of echocardiographic views is fundamental for automated cardiac analysis. However, clinical practice relies on a large, heterogeneous set of fine-grained acquisitions that introduce substantial inter-observer variability. Existing studies have primarily focused on limited view sets, often collapsing specialised views into broad categories, which limits their clinical relevance. We address this limitation by introducing TTE47, the first publicly available benchmark comprising 47 clinically meaningful views annotated independently by three experts. This dataset enables the rigorous quantification of inter-observer agreement and establishes a foundation for reproducible, clinically relevant evaluation. To tackle the dual challenges of subtle inter-class distinctions and structured label variability, we propose a novel supervised contrastive learning framework incorporating a tailored loss function. Our method outperforms cross-entropy and standard supervised contrastive baselines, achieving leading performance among evaluated methods on TTE47 and surpassing prior work on TMED-2 without dataset-specific pretraining, using a model pretrained on TTE47. Beyond accuracy, we introduce clustering-based metrics, Detection Rate and Label Recovery Precision, that measure semantic coherence and the model's ability to resist annotation variability. Results show that the learned feature space aligns more strongly with underlying anatomical structure than with any single annotator's style, enabling resilience to label shifts and maintaining robustness comparable to human-level disagreement. By integrating multi-expert evaluation, robust representation learning, and interpretable feature-space analysis, this work establishes a scalable and clinically relevant framework for fine-grained echo view classification. Our findings highlight the potential of contrastive pretraining to standardise interpretation, mitigate subjectivity, and enhance the reliability of AI-assisted echocardiography in diverse clinical settings.
6 May 2026
Read appraisal →Science (New York, N.Y.)
Performance of a large language model on the reasoning tasks of a physician.
More than 65 years ago, complex clinical diagnostic reasoning cases were introduced as the gold standard for the evaluation of expert medical computing systems, a standard that has held ever since. In this study, we report the results of a physician evaluation of a large language model (LLM) on challenging clinical cases across five experiments with a baseline of hundreds of physicians. We then report a real-world study comparing human expert and artificial intelligence (AI) second opinions in randomly selected patients in the emergency room of a major tertiary academic medical center. In all experiments, the LLM outperformed physician baselines and displayed continued improvement from prior generations of AI clinical decision support. Our study suggests that LLMs have eclipsed most benchmarks of clinical reasoning, motivating the urgent need for prospective trials.
5 May 2026
Read appraisal →PloS one
Pretraining effective T5 generative models for clinical and biomedical applications
This paper presents a study of the impact of corpus selection and vocabulary design on the performance of T5-based language models in clinical and biomedical domains. We introduce five different T5-EHR models, each pretrained from scratch using different combinations of clinical and biomedical corpora alongside domain-specific vocabularies. We evaluated these models across a variety of clinical and biomedical tasks to quantify the impact of pretraining data and vocabulary tokenization choices on downstream performance. Our findings reveal the importance of aligning both pretraining corpus and vocabulary with the target domain. Models pretrained exclusively on clinical data achieve superior performance on clinical tasks, while adding biomedical data contributes only marginal gains in most cases, with a few exceptions. Similarly, the choice of vocabulary significantly influences model performance, with clinical-specific vocabularies outperforming general biomedical vocabularies in tasks requiring a deeper understanding of clinical language. Also, the T5 generative models perform competitively with state-of-the-art discriminative models on several biomedical benchmarks, demonstrating strong generalization to biomedical domain. Overall, these results emphasize that task-specific selection of corpus and vocabulary is essential for optimizing model performance in clinical and biomedical natural language processing (NLP).
18 Apr 2026
Read appraisal →JMIR nursing
Application of the Technology Acceptance Model to Predict Nursing Students' Intention to Use Informatics: Cross-Sectional Study
BACKGROUND: Nursing informatics is essential for digital health transformation; however, the technology acceptance of undergraduate nursing students in Saudi Arabia remains underexplored. OBJECTIVE: This study examined factors influencing nursing students' intention to use informatics technologies using the technology acceptance model. METHODS: A cross-sectional survey was conducted with 132 undergraduate nursing students. Data were analyzed using descriptive, correlational, and hierarchical regression analyses. RESULTS: Perceived usefulness (mean 3.68, SD 1.22) and perceived ease of use (mean 3.64, SD 1.32) were the strongest predictors of acceptance, together explaining 87% of the variance (R²=0.87; β=0.323 for usefulness, P<.001; β=0.195 for ease of use, P=.032). Only 25.8% (n=34) of the students often used electronic health records, while 31.8% (n=42) had no electronic health record experience, indicating a clear gap in practical informatics exposure. CONCLUSIONS: Nursing students' acceptance of informatics is primarily driven by its perceived usefulness and perceived ease of use. These findings highlight the urgent need to integrate practical, user-centered informatics training and clinical simulation into undergraduate nursing curricula to better prepare students for technology-based practice.
16 Apr 2026
Read appraisal →JMIR nursing
Nurses' Evaluation of a Service Robot for Inpatient Care: Technology Acceptance Study
BACKGROUND: The integration of robotic systems into nursing practice is increasingly discussed as a potential strategy to alleviate workload and support care processes in response to demographic changes and staffing shortages. However, the acceptance of nursing staff as primary end users remains a critical determinant for successful implementation. Despite technological advances, the practical requirements and perspectives of nursing staff have not been adequately considered in research and development efforts to date. OBJECTIVE: Building on the user-centered development approach applied, this study aimed to examine nursing staff's evaluation of a service robot designed to assist with routine tasks in inpatient care, as well as their intention to use it, while accounting for technology-specific and psychological determinants of acceptance. METHODS: A total of 30 nurses tested the robot across 3 application scenarios (information service, item delivery, and beverage delivery) in a simulated care setting, alternating between the roles of nurse and care recipient. Acceptance-related constructs, including intention to use, were measured using the Technology Usage Inventory. General attitudes toward robots were assessed via the General Attitudes Towards Robots Scale. Participants' prior experience with robotics was also documented. Spearman rank correlations and Mann-Whitney U tests were used for analysis. RESULTS: The robot was rated positively across all dimensions. Usability was high (median 20, IQR 18-21; scale range 3-21), as was perceived usefulness (median 21, IQR 16-24; range 4-28). Skepticism was low (median 10.5, IQR 7-12; scale range 4-28), and accessibility was moderate (median 10, IQR 8-13; scale range: 3-21). Intention to use was strong (median 224.5, IQR 157-248; scale range 0-300) and correlated positively with usability (rs(28)=0.505; P=.004), perceived usefulness (rs(28)=0.74; P<.001), and accessibility (rs(28)=0.628; P<.001), and negatively with skepticism (rs(28)=-0.516; P=.004). More positive personal attitudes toward robots were also associated with higher perceived usefulness (rs(28)=0.549; P=.002) and greater intention to use (rs(28)=0.483; P=.007). No significant differences in intention to use were found between participants with and without prior robotics experience (U=83.5; P=.62). CONCLUSIONS: The findings indicate a high level of acceptance among the participating nursing staff for the developed service robot within the tested scenarios. Considering the chosen user-centered development approach, they further underscore the need for strategies that combine participatory design, transparent communication of system capabilities and limitations, and structured opportunities for hands-on experience. Such measures, together with proactive knowledge transfer and skills development, are essential to sustainably leverage the practical potential of service robotics in nursing practice.
15 Apr 2026
Read appraisal →PloS one
Privacy-preserving multimodal federated learning pipeline for cyber-resilient healthcare systems
The integration of Internet of Things (IoT) devices and electronic medical records (EMRs) has transformed healthcare delivery but has also created new vulnerabilities to cyberattacks that threaten both data confidentiality and patient safety. Conventional centralized machine learning approaches for intrusion detection are impractical in this domain due to strict privacy regulations, heterogeneous data sources, and the risk of single points of failure. To address these challenges, we propose a secure distributed machine learning pipeline for cyber-resilient healthcare systems. The framework combines federated optimization with split learning for sensitive EMR data, robust aggregation to mitigate poisoned updates, and differential privacy with secure aggregation to protect against inference attacks. Multimodal fusion is enabled through temporal consistency regularization for IoT traffic and cross-layer contrastive alignment to link EMR representations, ensuring improved anomaly detection across diverse healthcare environments. Experiments conducted on representative IoT and EMR datasets demonstrate that the proposed pipeline achieves accuracy of 0.942 on IoT data, 0.931 on EMR data, and 0.953 in the combined setting, with corresponding F1-scores of 0.921, 0.908, and 0.932. Ranking metrics further confirm superiority with AUROC up to 0.961 and AUPRC up to 0.947, outperforming deep baselines by margins of +0.025 to +0.033. Robustness analysis shows graceful degradation under client poisoning ([Formula: see text] at 30% malicious clients) and resilience under severe communication constraints (accuracy [Formula: see text] at 90% update sparsification). Detection latency is reduced to an average of 5.9 time steps, compared to 7.8 for the strongest deep baseline. These results highlight that secure distributed pipelines can deliver both strong detection capabilities and regulatory compliance, providing a practical path toward safeguarding next-generation healthcare infrastructures against evolving cyber threats.
12 Apr 2026
Read appraisal →PDA journal of pharmaceutical science and technology
The European Qualified Person: Legal Accountability, Quality Leadership, and AI Integration in Modern Pharmaceutical Manufacturing: Navigating Complex Therapies and AI-Supported Oversight to Safeguard Patient Safety
The European Qualified Person (QP) holds unique legal responsibility for certifying batch release, ensuring patient safety, and maintaining regulatory compliance. Modern pharmaceutical manufacturing-especially in biologics, advanced therapy medicinal products (ATMPs), and personalized medicines-generates complex, multi-site, data-rich environments that challenge traditional oversight. Artificial Intelligence (AI) offers predictive analytics, anomaly detection, and trend recognition to support decision-making but cannot currently replace the QP's scientific judgment or legal accountability. The forthcoming Annex 22 introduces Human-in-the-Loop (HITL) frameworks that embed human oversight within AI-supported processes, aligning technological advancement with regulatory control. This review explores evolving QP responsibilities, core and emerging competencies, AI integration within Quality Risk Management (QRM) and Good Manufacturing Practice (GMP) frameworks, and ethical considerations, illustrated with practical case studies. By developing AI literacy and applying HITL oversight, QPs can more effectively translate technological potential into transparent, science-based, and patient-centered batch-release decisions.
12 Apr 2026
Read appraisal →Journal of medical Internet research
Context-Aware Sentence Classification of Radiology Reports Using Synthetic Data: Development and Validation Study
BACKGROUND: Automated structuring of radiology reports is essential for data utilization and the development of medical artificial intelligence models. However, manual annotation by experts is labor-intensive, and processing real clinical data through commercial large language models (LLMs) presents significant privacy risks. These challenges are particularly pronounced for non-English languages like Japanese, where specialized medical corpora are scarce. While synthetic data generation offers a potential privacy-preserving alternative, its effectiveness in capturing complex clinical nuances-such as negation and contextual dependencies-to train robust classification models without any real-world training data has not been fully established. OBJECTIVE: This study aimed to develop a context-aware sentence classification model for Japanese radiology reports using an entirely synthetic training pipeline, thereby eliminating reliance on real-world clinical data during the development phase. Furthermore, we sought to evaluate the generalizability of this approach by validating the model's performance on diverse, multi-institutional, real-world reports. METHODS: Japanese radiology reports (n=3104) were generated using GPT-4.1 and automatically annotated at the sentence level into 4 categories (background, positive finding, negative finding, and continuation) using GPT-4.1-mini. The synthetic data were partitioned into training (n=2670), validation (n=334), and test (n=100) sets. We fine-tuned several models, including lightweight local LLMs (Qwen3 and Llama 3.2 series) using low-rank adaptation and Japanese text classification models (Bidirectional Encoder Representations from Transformers [BERT]-base Japanese v3, Japanese Medical Robustly Optimized BERT Pretraining Approach [JMedRoBERTa]-base, and ModernBERT-Ja-130M). External validation was performed using 280 real-world reports (3477 sentences) from 7 institutions in the Japan Medical Image Database, with ground-truth labels established by board-certified radiologists. Evaluation metrics included accuracy, macro-averaged F1 (macro F1) score, and positive predictive value for positive findings (PPV_1). RESULTS: All models achieved high performance on the synthetic test set (accuracy: 0.938-0.951; macro F1-score: 0.924-0.940). Overall performance declined on the external validation dataset (accuracy: 0.783-0.813; macro F1-score: 0.761-0.790), reflecting distributional differences between synthetic and real-world reports; however, PPV_1 remained stable and high across datasets (eg, 0.957 on the synthetic test set vs 0.952 on the external validation dataset for Qwen3 [4B]). Parsing errors occurred in LLM-based approaches (19-260 sentences, 0.55%-7.48% in the external dataset). CONCLUSIONS: This study demonstrates the feasibility of developing context-aware sentence classification models for Japanese radiology reports using a training pipeline based entirely on synthetic data. The stability of PPV_1 indicates that the models successfully captured the essential clinical terminology and linguistic patterns required to identify positive findings in real-world reports, despite the observed performance degradation during external validation. This approach substantially reduces manual annotation requirements and privacy risks, providing a scalable foundation for constructing structured radiology datasets to support the development of clinically relevant medical artificial intelligence models.
11 Apr 2026
Read appraisal →Journal of medical Internet research
Initial Insights Into an Institutional Secure Large Language Model for Magnetic Resonance Imaging Examination Requests: Retrospective Study
BACKGROUND: Incomplete clinical details on magnetic resonance imaging (MRI) examination requests (MERs) can lead to suboptimal protocol selection. An institutional secure large language model (sLLM) with access to manually retrieved salient data from the electronic medical record (EMR) may improve request completeness and protocol accuracy across multiple MRI subspecialties. OBJECTIVE: The objective of this study was to compare clinician MERs with sLLM-augmented MERs for information quality and to evaluate the protocoling accuracy of the sLLM versus board-certified radiologists across body, musculoskeletal, and neuroradiology MRI. METHODS: This retrospective study included 608 random outpatient MRI examinations performed between September 2023 and July 2024 (body 206, musculoskeletal 203, neuroradiology 199). The cohort comprised 528 patients (mean 51.2 years, SD 19.2; range 4-93; n=279, 52.8% women, n=249, 47.2% men). MERs without EMR access were excluded. A privately hosted Anthropic Claude 3.5 model (temperature 0) augmented each MER with manually retrieved salient EMR data and, via rule-based parsing, mapped the extracted elements onto predefined institutional criteria to recommend region or coverage and contrast use. Two experienced radiologists established a consensus reference standard. Two board-certified general radiologists (Rad 3 and Rad 4) and the sLLM were compared with this standard. Clinical information quality was graded using the Reason-for-Exam Imaging Reporting and Data System (RI-RADS). Interrater reliability was quantified with Gwet AC1. Paired accuracies were compared with the McNemar test to determine whether there was a statistically significant difference. RESULTS: Interreader agreement for RI-RADS was almost perfect for sLLM-augmented MERs (AC1 0.97, 95% CI 0.94-0.99) and moderate for clinician MERs (AC1 0.43, 95% CI 0.34-0.52). Limited or deficient clinical information (RI-RADS C/D) fell to 0% to 0.7% (0/608 to 4/608) with sLLM augmentation vs 4.1% to 20.4% (25/608 to 124/608) for clinician MERs. Overall protocol accuracy was 93.1% (566/608; 95% CI 89.6-96.6) for the sLLM, 91.4% (556/608; 95% CI 87.6-95.3) for Rad 3, and 92.1% (560/608; 95% CI 88.4-95.8) for Rad 4 (sLLM vs Rad 3 P=.23 vs Rad 4 P=.40). Region or coverage accuracy was similar (sLLM: 579/608, 95.2%; Rad 3: 585/608, 96.2%; Rad 4: 573/608, 94.2%; P=.46 and P=.36). Contrast decisions were more accurate using the sLLM at 94.4% (574/608; 95% CI 91.3-97.5) vs Rad 3 at 92.1% (560/608; 95% CI 88.4-95.8; P=.027) and were not significantly different to Rad 4 at 92.9% (565/608; 95% CI 89.4-96.4; P=.16). Subspecialty analyses showed similar patterns, with the sLLM outperforming Rad 4 for musculoskeletal MRI contrast decisions (96.6% vs 91.1%; P=.006) and matching readers elsewhere. Manual review indicated that sLLM improvements arose from EMR details not listed on the MER (infection/inflammation, tumor history, prior surgery). No clinically significant hallucinations were identified in a manual review of discordant cases. CONCLUSIONS: Across body, musculoskeletal, and neuroradiology MRI, sLLM-augmented examination requests improved clinical context and enhanced contrast selection while demonstrating accuracy comparable to general radiologists for region or coverage. Integrating sLLMs into routine vetting workflows may reduce manual workload in protocol selection for more efficient, standardized protocoling.
8 Apr 2026
Read appraisal →JMIR formative research
Performance of DeepSeek V3, DeepSeek R1, ChatGPT 4o, and ChatGPT o1 on the National Health Professional and Technical Qualification Examination (Intermediate Level) in China: Comparative Analysis
BACKGROUND: In recent years, large language models (LLMs) have undergone swift cycles of refinement and iteration. However, in the realm of clinical medicine, different LLMs' capability of logical reasoning and disease diagnosis needs further investigation. OBJECTIVE: The aim of our study was to evaluate the performance of 4 different LLMs in the National Health Professional and Technical Qualification Examination in China. METHODS: A total of 398 multiple-choice questions of 5 different question types were integrated within the examination with respect to the diagnosis or care of cases. These questions were categorized into different cardiology subspecialties and different clinical disciplines. DeepSeek V3 and R1 were accessed through an application programming interface, while ChatGPT 4o and o1 were queried via its public chat-based interface. We offered the same prompts instructing LLMs to assume the role of a physician and provide answers with explanations at the beginning of each conversation. We assessed different LLMs' performance by the accuracy in the responses to the multiple-choice questions. For the first 3 examination sections, McNemar test was used to compare the accuracy among the models, with post hoc pairwise comparisons performed using partitions of chi-square method and Bonferroni correction (significance set at P<.008). For the fourth section involving partially credit scoring, one-way ANOVA was performed to compare the mean scores among the models, with statistical significance set at P<.05. RESULTS: Both DeepSeek V3 and R1 showed superior performance in the first 3 sections of the Chinese National Health Professional and Technical Qualification Examination, achieving an overall performance of 93% and 93.6%, respectively. ChatGPT 4o and o1 achieved accuracies of 73.3% and 69%, respectively (all P<.001 compared with DeepSeek V3). For the fourth section, the performance of all 4 LLMs markedly declined compared to their results in the preceding sections. Particularly, in the section of gastroenterology and hematology, DeepSeek V3 achieved the highest accuracy, while R1 ranked first in cardiology and neurology. ChatGPT o1 achieved the highest accuracy in the topic of coronary artery disease, with no statistical significance. CONCLUSIONS: DeepSeek V3 and R1 showed remarkable potential in facilitating clinical decision-making in the Chinese professional examination, with both outperforming ChatGPT 4o and o1. Nonetheless, future research should continue evaluating their economic efficiency and susceptibility to hallucination.
7 Apr 2026
Read appraisal →European radiology
Deep learning models for radiography body-part classification and chest radiograph projection/orientation classification: a multi-institutional study.
OBJECTIVES: Large-scale radiographic datasets often include errors in labels such as body parts or projection, which can undermine automated image analysis. Therefore, we aimed to develop and externally validate two deep-learning models-one for categorising radiographs by body part, and another for identifying projection and rotation of chest radiographs-using large, diverse datasets. MATERIALS AND METHODS: We retrospectively collected radiographs from multiple institutions and public repositories. For the first model (Xp-Bodypart-Checker), we included seven categories (Head, Neck, Chest, Incomplete Chest, Abdomen, Pelvis, Extremities). For the second model (CXp-Projection-Rotation-Checker), we classified chest radiographs by projection (anterior-posterior, posterior-anterior, lateral) and rotation (upright, inverted, left rotation, right rotation). Both models were trained, tuned, and internally tested on separate data, then externally tested on radiographs from different institutions. Model performance was assessed using overall accuracy (micro, macro, and weighted) as well as one-vs.-all area under the receiver operating characteristic curve (AUC). RESULTS: In the Xp-Bodypart-Checker development phase, we included 429,341 radiographs obtained from Institutions A, B, and MURA. In the CXp-Projection-Rotation-Checker development phase, we included 463,728 chest radiographs from CheXpert, PadChest, and Institution A. The Xp-Bodypart-Checker achieved AUC values of 1.00 (99% CI: 1.00-1.00) for all classes other than Incomplete Chest, which had an AUC value of 0.99 (99% CI: 0.98-1.00). The CXp-Projection-Rotation-Checker demonstrated AUC values of 1.00 (99% CI: 1.00-1.00) across all projection and rotation classifications. CONCLUSION: These models help automatically verify image labels in large radiographic databases, improving quality control across multiple institutions. KEY POINTS: Question This study examines how deep learning can accurately classify radiograph body parts and detect chest projection/orientation in large, multi-institutional datasets, enhancing metadata consistency for clinical and research workflows. Findings Xp-Bodypart-Checker classified radiographs into seven categories with AUC values of over 0.99 for all classes, while CXp-Projection-Rotation-Checker achieved AUC values of 1.00 across all projection and rotation classifications. Clinical relevance Trained on over 860,000 multi-institutional radiographs, our two deep-learning models classify radiograph body-part and chest radiograph projection/rotation, identifying mislabeled data and enhancing data integrity, thereby improving reliability for both clinical use and deep-learning research.
4 Apr 2026
Read appraisal →Aesthetic Plast Surg
The Rise of Intelligent Plastic Surgery: A 10-Year Bibliometric Journey Through AI Applications, Challenges, and Transformative Potential
BACKGROUND: Driven by advancements in deep learning, surgical robots, and predictive modeling technologies, the integration of artificial intelligence (AI) and plastic surgery has expanded rapidly. Although AI shows the potential to enhance precision and efficiency, its clinical integration faces challenges, including ethical concerns and interdisciplinary complexity, which require a systematic analysis of research trends. METHODS: The CiteSpace and VOSviewer software were used to conduct a quantitative analysis of 235 documents in the core collection of Web of Science from 2016 to 2024. Co-citation networks, keyword co-occurrence, burst detection, and cluster analysis were employed to map the research trajectories. The inclusion criteria gave priority to studies that explicitly incorporated artificial intelligence into surgical designs or outcomes. The contributions of countries, institutions, and authors were evaluated through centrality indicators. RESULT: Publications related to artificial intelligence have grown exponentially, with the USA, Germany, and Canada leading research output. Harvard and Stanford Universities dominate in terms of institutional contributions, but cross-institutional collaboration remains limited. The keyword cluster highlights the innovations of artificial intelligence in breast reconstruction, facial analysis, and automated grading systems. Burst terms such as "deep learning," "risk assessment," and "attractiveness" underscore AI's role in optimizing surgical outcomes, but they also expose biases against Western-centric beauty standards. Ethical concerns, dataset diversity gaps, and overreliance on AI-driven decisions have become key obstacles. CONCLUSION: The integration of artificial intelligence in plastic surgery goes beyond the utility based on tools and into data-informed surgical engineering. The persistent gap in collaboration and dataset diversity highlights the need for global, interdisciplinary efforts to address technical and ethical challenges while advancing AI's clinical utility. Future research must prioritize transparency, inclusivity, and collaborative innovation to realize AI's transformative potential while mitigating risks. LEVEL OF EVIDENCE IV: This journal requires that authors assign a level of evidence to each article. For a full description of these Evidence-Based Medicine ratings, please refer to the Table of Contents or the online Instructions to Authors www.springer.com/00266 .
1 Mar 2026
Read appraisal →