Classification of Cochrane Plain Language Summaries by Conclusiveness Using Transformer-Based Models and ChatGPT: Retrospective Observational Study

Clinical Snapshot

Key Findings

• P Value: Not reported

• Effect Size: Longformer: 60.9% balanced accuracy; SciBERT: 56.6% balanced accuracy; ChatGPT: 74.2% accuracy

• Primary Outcome: Classification accuracy for three conclusiveness levels (conclusive, inconclusive, unclear)

• Nnt Or Sensitivity: AUCROC values: Longformer 0.86 (conclusive), 0.67 (inconclusive), 0.72 (unclear); Cohen's κ 0.57 for ChatGPT

• Confidence Interval: Not reported

Clinical Application

Limited by poor generalization of fine-tuned models; ChatGPT shows promise but requires validation in clinical settings Relevant for NHMRC evidence synthesis, RACGP guideline development, and clinical decision-making tools used in Australian healthcare Evidence-based medicine practitioners, systematic review authors, healthcare professionals using Cochrane reviews

Abstract

BACKGROUND: Cochrane plain language summaries (PLSs) aim to make systematic review findings more accessible to the general public. However, inconsistencies in how conclusions are presented may impact comprehension and decision-making. Classifying PLSs based on conclusiveness can improve clarity and facilitate informed health decisions. OBJECTIVE: This study aimed to develop and evaluate deep learning language models for the classification of PLSs according to 3 levels of conclusiveness (conclusive, inconclusive, and unclear) and to compare their performance with a general-purpose large language model (GPT-4o). METHODS: We used a publicly available dataset containing 4405 Cochrane PLSs of systematic reviews published until 2019, already classified by humans according to 9 categories of conclusiveness regarding the intervention's effectiveness or safety. We merged these categories into 3 classes based on the strength of conclusiveness: conclusive, inconclusive, and unclear. For the fine-tuning, we used Scientific Bidirectional Encoder Representations from Transformers (SciBERT), a pretrained language model trained on 1.14 million papers primarily from the health sciences, and Longformer, a transformer model designed specifically to process long documents. The script was developed using the Python programming language and the PyTorch framework. We computed evaluation metrics using the scikit-learn machine learning library and determined the area under the curve of the receiver operating characteristic (AUCROC) to measure the model performance in balancing sensitivity and specificity. We also analyzed a separate set of 213 PLSs and compared the predictions of our pretrained models with both manual verification and outputs generated by ChatGPT. RESULTS: The model based on SciBERT achieved a balanced accuracy of 56.6%. The AUCROC was 0.91 for "conclusive," 0.67 for "inconclusive," and 0.75 for "unclear" conclusiveness classes. The Longformer-based model had a balanced accuracy of 60.9%, with AUCROCs of 0.86 for "conclusive," 0.67 for "inconclusive," and 0.72 for "unclear" conclusiveness classes. Both models underperformed compared with ChatGPT, which demonstrated higher accuracy (74.2%), better precision and recall, and a higher Cohen κ (0.57). CONCLUSIONS: Fine-tuning 2 transformer-based language models showed mixed results in classifying Cochrane PLSs by conclusiveness, likely due to semantic overlap and subtle linguistic differences. Despite satisfactory internal test metrics, the fine-tuned models failed to generalize to newly published PLSs, where performance dropped to near-chance levels. These findings suggest that general-purpose large language models like GPT-4o may currently offer more reliable results for practical classification tasks in biomedical applications.

References

1. 1.Mijatović, A., Ursić, L., Bralić, N., Bandić, R., Ćaćić, B., Buljan, I., & Marušić, A. (2026). Classification of Cochrane Plain Language Summaries by Conclusiveness Using Transformer-Based Models and ChatGPT: Retrospective Observational Study. JMIR Medical Informatics, 14, e72657. https://doi.org/10.2196/72657