Performance of large language models conducting systematic review tasks in prosthodontics.

Clinical Snapshot

Key Findings

• P Value: P=.04 for labelling and modelling tasks at Week 1 (Kruskal-Wallis with Dunn post hoc); no other significant differences reported across time points

• Effect Size: Inter-model performance differences were statistically significant only for labelling and modelling sub-tasks during Week 1 (P=.04); no significant differences were detected at Weeks 2 or 4, suggesting relative temporal stability for most tasks

• Primary Outcome: Full-text screening performance: Claude achieved the highest sensitivity (97%) with 86% accuracy and 87% F1-score; all four models maintained sensitivity above 90%. Data extraction: GPT-4 consistently achieved median scores of 5.0/5.0; Claude and Gemini showed similar capabilities. Risk of bias assessment: agreement with expert judgements ranged from 55% to 90% across different criteria.

• Nnt Or Sensitivity: Sensitivity (full-text screening): Claude 97%, all models >90%. Specificity: not explicitly reported in abstract. Cohen kappa (risk of bias): range 55%–90% across criteria — indicating moderate to substantial agreement at best, with some criteria showing only poor-to-moderate agreement

• Confidence Interval: Not reported

Clinical Application

GPT-4, Gemini, and Claude are commercially accessible via subscription or API in most high-income countries. Elicit is a dedicated research tool with free and paid tiers. Implementation requires structured prompting expertise and mandatory human oversight at all SR stages. The temporal reliability data (three time points) suggest that outputs should be re-verified if significant time elapses between LLM query and manuscript submission, given potential model updates. All four LLMs evaluated are accessible to Australian dental researchers and clinicians. The Australian Dental Association (ADA) and the Dental Board of Australia have not yet issued formal guidance on LLM use in research workflows. The NHMRC's 2022 guidelines on systematic reviews do not address LLM-assisted evidence synthesis. Australian dental schools and the Cochrane Oral Health Group would benefit from local validation studies before adopting LLM-assisted SR workflows. No PBS or TGA implications apply directly, though TGA-regulated dental devices and materials are common subjects of prosthodontic systematic reviews, making accurate data extraction and risk of bias assessment clinically consequential. Dental and prosthodontic researchers conducting systematic reviews who are considering LLM-assisted workflows for full-text screening, data extraction, or risk of bias assessment. Applicable to academic dental institutions, postgraduate research programmes, and evidence synthesis teams in oral health.

Abstract

STATEMENT OF PROBLEM: Systematic reviews (SRs) are time-consuming and resource-intensive processes. Whether large language models (LLMs) can improve the process is unclear. PURPOSE: The purpose of this study was to evaluate the accuracy and reliability of 4 LLMs (GPT-4, Gemini, Claude, and Elicit) in performing SR tasks (full-text screening, data extraction, and risk of bias assessment) at 3 different sequential periods of time (0, 15, and 30 days). MATERIAL AND METHODS: A comprehensive systematic search was conducted across 5 databases in December 2024, with 59 articles evaluated for screening (2 used for pilot) and 31 for data extraction (2 used for pilot). A 3-pronged prompting strategy was used, including persona-based initialization, few-shot learning, and structured population, intervention, control, outcome (PICO) criteria. Performance was assessed through 3 repeated evaluations at 2-week intervals by measuring accuracy and reliability using standard metrics (accuracy, precision, F1-score, sensitivity, and specificity) against expert assessments, data extraction quality on a 0 to 5 scale, and risk of bias agreement via the Cohen kappa, with statistical analysis using Kruskal-Wallis and Dunn post hoc tests (α=.05). RESULTS: In full-text screening, Claude achieved the highest sensitivity at 97%, while Claude and Elicit both showed strong overall performance with 86% accuracy and 87% F1-scores. All models maintained sensitivity above 90%. For data extraction, GPT-4 consistently performed best with median scores of 5.0, while Claude and Gemini showed similar capabilities. Significant differences only appeared in labeling and modeling tasks during Week 1 (P=.04). Risk of bias assessment agreement with experts varied from 55% to 90% across different criteria. CONCLUSIONS: LLMs show potential for SR efficiency (especially for data extraction) but require human oversight because of variable performance across models and tasks.

References

1. 1.Rokhshad, R., Motie, P., Shirani, M., Sameie, A., & Revilla-León, M. (2026). Performance of large language models conducting systematic review tasks in prosthodontics. The Journal of Prosthetic Dentistry. Advance online publication. https://doi.org/10.1016/j.prosdent.2026.02.009