Using full agreement across multiple large language models for title-and-abstract screening in systematic reviews: a proof-of-concept

Clinical Snapshot

Key Findings

• P Value: Statistically significant improvements reported

• Effect Size: Approximately 80% of abstracts achieved full agreement

• Primary Outcome: Proportion of abstracts achieving full agreement across three LLMs

• Nnt Or Sensitivity: Superior classification performance compared to single-LLM approaches

• Confidence Interval: Not reported in abstract

Clinical Application

Requires access to multiple commercial LLM platforms and technical workflow implementation Could support NHMRC systematic review guidelines and Cochrane Australia activities, though cost-effectiveness and TGA regulatory considerations for clinical decision support tools need evaluation Systematic review teams conducting title-and-abstract screening across various clinical domains

Abstract

BACKGROUND: The exponential growth of scientific literature poses significant challenges for conducting systematic reviews, particularly in the labor-intensive title-and-abstract screening phase. This study examines the feasibility of using multiple large language models (LLMs) for title-and-abstract screening in systematic reviews. METHODS: We propose and evaluate a full-agreement approach using three commercially available LLMs from different model families (ChatGPT, Gemini, and Claude), in which automated classification decisions are only accepted when all three models assign the same label. This approach was examined across six datasets to assess its effectiveness. A structured workflow was developed to support implementation without requiring specialized technical expertise. Additionally, a stop criterion was introduced to ensure that LLM-based classification is only applied when predefined performance thresholds are met. RESULTS: Across the six datasets, approximately four in five abstracts received full agreement across models and could therefore be classified automatically. For this subset of abstracts, classification performance was consistently higher than that of previous automated approaches using LLMs, with statistically significant improvements in all prespecified performance metrics. CONCLUSIONS: A full agreement approach across three LLMs may offer a promising and conservative strategy for title-and-abstract screening in systematic reviews by automating concordant decisions while reserving human review for discordant cases. As a proof-of-concept, the present findings support this approach as a possible workflow for reducing screening burden in the face of continued growth in the scientific literature, although its broader generalizability remains to be established.

References

1. 1.Hilkenmeier, F., Stoltenberg, M., & Stierle, C. (2026). Using full agreement across multiple large language models for title-and-abstract screening in systematic reviews: a proof-of-concept. Systematic Reviews. https://doi.org/10.14324/LRE.19.1.32