Research Appraisals
Evidence-based critical appraisals of the latest medical research, systematically evaluated using Oxford CEBM methodology.
Showing 4 appraisals
Behavior research methods
Automating data extraction in meta-research: A multi-model benchmark in network psychometrics papers
Manual data extraction in meta-research is often tedious, time-consuming, and error-prone. In this paper, we investigate whether the current generation of large language models (LLMs) can be used to extract accurate information from scientific papers. Across the meta-research literature, these tasks usually range from extracting verbatim information (e.g., the number of participants in a study, effect sizes, or whether a study is preregistered) to making subjective inferences. Using a publicly available dataset containing a wide range of metascientific variables from 43 network psychometrics papers, we tested five LLMs (Claude 4.6 Opus, Claude 4.5 Sonnet, Claude 4.5 Haiku, GPT-5.2, and GPT-5 mini). We used an automated API-based pipeline to extract variables from the documents. This approach allows batch processing of research papers. As such, it represents a more efficient and scalable way to extract metascientific data than the default chat interface. The extraction accuracy ranged from 79.6% to 91.3% across the models. The extraction performance was generally higher for more explicit, verbatim information and worse for variables that required more complicated inference. Furthermore, most models were able to convey uncertainty in more contentious cases. We provide a comparison of the accuracy and cost-effectiveness of the individual models and discuss the characteristics of variables that are and are not suitable for automatic coding. Furthermore, we describe some of the common pitfalls and best practices of automated LLM data extraction. The proposed procedure can substantially reduce the time and costs associated with conducting meta-research.
25 May 2026
Read appraisal →Research synthesis methods
Transforming evidence synthesis: A systematic review of the evolution of automated meta-analysis in the age of AI
Exponential growth in scientific literature has heightened the demand for efficient evidence-based synthesis, driving the rise of the field of automated meta-analysis (AMA) powered by natural language processing and machine learning. This PRISMA systematic review introduces a structured framework for assessing the current state of AMA, based on screening 13,216 papers (2006-2024) and analyzing 61 studies across diverse domains. Findings reveal a predominant focus on automating data processing (52.5%), such as extraction and statistical modeling, while only 16.4% address advanced synthesis stages. Just one study (approximately 2%) explored preliminary full-process automation, highlighting a critical gap that limits AMA's capacity for comprehensive synthesis. Despite recent breakthroughs in large language models and advanced AI, their integration into statistical modeling and higher-order synthesis, such as heterogeneity assessment and bias evaluation, remains underdeveloped. This has constrained AMA's potential for fully autonomous meta-analysis (MA). From our dataset spanning medical (67.2%) and non-medical (32.8%) applications, we found that AMA has exhibited distinct implementation patterns and varying degrees of effectiveness in actually improving efficiency, scalability, and reproducibility. While automation has enhanced specific meta-analytic tasks, achieving seamless, end-to-end automation remains an open challenge. As AI systems advance in reasoning and contextual understanding, addressing these gaps is now imperative. Future efforts must focus on bridging automation across all MA stages, refining interpretability, and ensuring methodological robustness to fully realize AMA's potential for scalable, domain-agnostic synthesis.
6 May 2026
Read appraisal →JMIR medical informatics
Classification of Cochrane Plain Language Summaries by Conclusiveness Using Transformer-Based Models and ChatGPT: Retrospective Observational Study
BACKGROUND: Cochrane plain language summaries (PLSs) aim to make systematic review findings more accessible to the general public. However, inconsistencies in how conclusions are presented may impact comprehension and decision-making. Classifying PLSs based on conclusiveness can improve clarity and facilitate informed health decisions. OBJECTIVE: This study aimed to develop and evaluate deep learning language models for the classification of PLSs according to 3 levels of conclusiveness (conclusive, inconclusive, and unclear) and to compare their performance with a general-purpose large language model (GPT-4o). METHODS: We used a publicly available dataset containing 4405 Cochrane PLSs of systematic reviews published until 2019, already classified by humans according to 9 categories of conclusiveness regarding the intervention's effectiveness or safety. We merged these categories into 3 classes based on the strength of conclusiveness: conclusive, inconclusive, and unclear. For the fine-tuning, we used Scientific Bidirectional Encoder Representations from Transformers (SciBERT), a pretrained language model trained on 1.14 million papers primarily from the health sciences, and Longformer, a transformer model designed specifically to process long documents. The script was developed using the Python programming language and the PyTorch framework. We computed evaluation metrics using the scikit-learn machine learning library and determined the area under the curve of the receiver operating characteristic (AUCROC) to measure the model performance in balancing sensitivity and specificity. We also analyzed a separate set of 213 PLSs and compared the predictions of our pretrained models with both manual verification and outputs generated by ChatGPT. RESULTS: The model based on SciBERT achieved a balanced accuracy of 56.6%. The AUCROC was 0.91 for "conclusive," 0.67 for "inconclusive," and 0.75 for "unclear" conclusiveness classes. The Longformer-based model had a balanced accuracy of 60.9%, with AUCROCs of 0.86 for "conclusive," 0.67 for "inconclusive," and 0.72 for "unclear" conclusiveness classes. Both models underperformed compared with ChatGPT, which demonstrated higher accuracy (74.2%), better precision and recall, and a higher Cohen κ (0.57). CONCLUSIONS: Fine-tuning 2 transformer-based language models showed mixed results in classifying Cochrane PLSs by conclusiveness, likely due to semantic overlap and subtle linguistic differences. Despite satisfactory internal test metrics, the fine-tuned models failed to generalize to newly published PLSs, where performance dropped to near-chance levels. These findings suggest that general-purpose large language models like GPT-4o may currently offer more reliable results for practical classification tasks in biomedical applications.
16 Apr 2026
Read appraisal →PloS one
The impact of physical adjunctive interventions on outcomes of clear aligner treatment: A systematic review of randomized controlled trials
INTRODUCTION: Physical adjunctive interventions (PAIs), including vibration devices and low-level laser therapy, are promoted to accelerate tooth movement, improve aligner tracking, and reduce discomfort in clear aligner treatment (CAT), but randomized evidence remains inconsistent. This systematic review aimed to assess whether PAIs improve CAT outcomes in terms of objective tooth-movement metrics, aligner tracking, overall treatment efficiency, and patient-centered outcomes, and to appraise the risk of bias and the certainty of the evidence at the outcome level. METHODS: Comprehensive electronic searches of PubMed, Embase, Scopus, Web of Science, and CENTRAL were conducted from database inception (earliest available indexing date in each database) to 30 June 2025 (coverage cut-off). Grey literature searching and hand-searching were also performed, with no restrictions on language or publication status. Eligible studies were human randomized controlled trials (RCTs) comparing CAT combined with PAIs versus CAT alone or sham interventions. Two independent reviewers performed study selection and data extraction, with risk of bias assessed using the Cochrane RoB 2 tool and certainty of evidence appraised with GRADE. Random-effects meta-analyses were conducted when feasible; otherwise, results were narratively summarized. The protocol was prospectively registered (PROSPERO CRD420251132229). RESULTS: Seven RCTs involving 266 participants were included. No significant improvement in Little's Irregularity Index (maxilla MD = 0.08, p = 0.869; mandible MD = 0.44, p = 0.487). Vibration improved aligner tracking under a 7-day change schedule (p = 0.003) but not case completion (p = 0.999). Overall compliance was approximately 77%, and no between-group difference was observed (p = 0.390). Pain was slightly lower on days 1-3 only (p < 0.05); no quality-of-life benefits were observed; periodontal indices remained unchanged; biomarkers showed inconsistent results. The certainty of evidence was low to very low, suggesting that further well-designed RCTs are likely to change the effect estimates and may alter the conclusions. CONCLUSIONS: Across objective tooth-movement metrics, aligner tracking, treatment efficiency, and patient-centered outcomes, current randomized evidence does not demonstrate a consistent benefit of physical adjunctive interventions in clear aligner treatment. Interpretation is limited by outcome-level risk of bias concerns and low to very low certainty of evidence.
9 Apr 2026
Read appraisal →