Research Appraisals
Evidence-based critical appraisals of the latest medical research, systematically evaluated using Oxford CEBM methodology.
Showing 3 appraisals
The Journal of prosthetic dentistry
Performance of large language models conducting systematic review tasks in prosthodontics.
STATEMENT OF PROBLEM: Systematic reviews (SRs) are time-consuming and resource-intensive processes. Whether large language models (LLMs) can improve the process is unclear. PURPOSE: The purpose of this study was to evaluate the accuracy and reliability of 4 LLMs (GPT-4, Gemini, Claude, and Elicit) in performing SR tasks (full-text screening, data extraction, and risk of bias assessment) at 3 different sequential periods of time (0, 15, and 30 days). MATERIAL AND METHODS: A comprehensive systematic search was conducted across 5 databases in December 2024, with 59 articles evaluated for screening (2 used for pilot) and 31 for data extraction (2 used for pilot). A 3-pronged prompting strategy was used, including persona-based initialization, few-shot learning, and structured population, intervention, control, outcome (PICO) criteria. Performance was assessed through 3 repeated evaluations at 2-week intervals by measuring accuracy and reliability using standard metrics (accuracy, precision, F1-score, sensitivity, and specificity) against expert assessments, data extraction quality on a 0 to 5 scale, and risk of bias agreement via the Cohen kappa, with statistical analysis using Kruskal-Wallis and Dunn post hoc tests (α=.05). RESULTS: In full-text screening, Claude achieved the highest sensitivity at 97%, while Claude and Elicit both showed strong overall performance with 86% accuracy and 87% F1-scores. All models maintained sensitivity above 90%. For data extraction, GPT-4 consistently performed best with median scores of 5.0, while Claude and Gemini showed similar capabilities. Significant differences only appeared in labeling and modeling tasks during Week 1 (P=.04). Risk of bias assessment agreement with experts varied from 55% to 90% across different criteria. CONCLUSIONS: LLMs show potential for SR efficiency (especially for data extraction) but require human oversight because of variable performance across models and tasks.
2 July 2026
Read appraisal →PloS one
Development and validation of search hedges for Transgender and Gender Diverse (TGD) populations in Ovid MEDLINE and Ovid APA PsycInfo
INTRODUCTION: This paper describes the development and validation of highly sensitive search hedges for Ovid MEDLINE and Ovid APA PsycInfo that effectively identify literature on transgender and gender diverse (TGD) populations. METHODS: Two librarians developed the search hedges using relevant keywords and controlled vocabulary terms, building on previous work on identifying transgender populations in evidence synthesis. The hedges were tested and refined to capture diverse and expansive gender identities across cultures and disciplines. The hedges were validated for sensitivity using a gold standard set of 144 articles from the Knowsy portal of evidence syntheses tagged as Two-Spirit, transgender, or gender non-binary. To assess precision an international research team of subject experts independently screened a randomized sample of search results in a two-stage screening process with an additional screener resolving disputes. RESULTS: The final search hedges demonstrated 100% sensitivity in both MEDLINE and APA PsycInfo, identifying all 144 relevant articles from the Knowsy gold standard set. The MEDLINE search hedge achieved a 71% precision, and the APA PsycInfo hedge achieved a 67% precision. These results balance comprehensive retrieval while minimizing non-relevant articles for an efficient screening process. CONCLUSIONS: These search hedges in MEDLINE and APA PsycInfo are valuable tools for researchers and librarians to more effectively identify literature on TGD populations. These tools will be crucial for ongoing work in addressing gaps in research and health disparities faced by TGD populations and will be particularly valuable for researchers conducting evidence synthesis projects related to this population.
14 June 2026
Read appraisal →Systematic reviews
Using full agreement across multiple large language models for title-and-abstract screening in systematic reviews: a proof-of-concept
BACKGROUND: The exponential growth of scientific literature poses significant challenges for conducting systematic reviews, particularly in the labor-intensive title-and-abstract screening phase. This study examines the feasibility of using multiple large language models (LLMs) for title-and-abstract screening in systematic reviews. METHODS: We propose and evaluate a full-agreement approach using three commercially available LLMs from different model families (ChatGPT, Gemini, and Claude), in which automated classification decisions are only accepted when all three models assign the same label. This approach was examined across six datasets to assess its effectiveness. A structured workflow was developed to support implementation without requiring specialized technical expertise. Additionally, a stop criterion was introduced to ensure that LLM-based classification is only applied when predefined performance thresholds are met. RESULTS: Across the six datasets, approximately four in five abstracts received full agreement across models and could therefore be classified automatically. For this subset of abstracts, classification performance was consistently higher than that of previous automated approaches using LLMs, with statistically significant improvements in all prespecified performance metrics. CONCLUSIONS: A full agreement approach across three LLMs may offer a promising and conservative strategy for title-and-abstract screening in systematic reviews by automating concordant decisions while reserving human review for discordant cases. As a proof-of-concept, the present findings support this approach as a possible workflow for reducing screening burden in the face of continued growth in the scientific literature, although its broader generalizability remains to be established.
14 June 2026
Read appraisal →