Automating data extraction in meta-research: A multi-model benchmark in network psychometrics papers

Clinical Snapshot

Key Findings

• P Value: Not reported

• Effect Size: 12-point accuracy range between best and worst performing models

• Primary Outcome: LLM extraction accuracy ranged from 79.6% to 91.3% across five models

• Nnt Or Sensitivity: Higher accuracy for explicit verbatim information versus subjective inferences

• Confidence Interval: Not reported

Clinical Application

High feasibility with API access; requires technical expertise for implementation Relevant for Australian research institutions conducting meta-analyses; could support NHMRC systematic review guidelines and reduce research costs Meta-researchers and systematic reviewers in psychology and related fields

Abstract

Manual data extraction in meta-research is often tedious, time-consuming, and error-prone. In this paper, we investigate whether the current generation of large language models (LLMs) can be used to extract accurate information from scientific papers. Across the meta-research literature, these tasks usually range from extracting verbatim information (e.g., the number of participants in a study, effect sizes, or whether a study is preregistered) to making subjective inferences. Using a publicly available dataset containing a wide range of metascientific variables from 43 network psychometrics papers, we tested five LLMs (Claude 4.6 Opus, Claude 4.5 Sonnet, Claude 4.5 Haiku, GPT-5.2, and GPT-5 mini). We used an automated API-based pipeline to extract variables from the documents. This approach allows batch processing of research papers. As such, it represents a more efficient and scalable way to extract metascientific data than the default chat interface. The extraction accuracy ranged from 79.6% to 91.3% across the models. The extraction performance was generally higher for more explicit, verbatim information and worse for variables that required more complicated inference. Furthermore, most models were able to convey uncertainty in more contentious cases. We provide a comparison of the accuracy and cost-effectiveness of the individual models and discuss the characteristics of variables that are and are not suitable for automatic coding. Furthermore, we describe some of the common pitfalls and best practices of automated LLM data extraction. The proposed procedure can substantially reduce the time and costs associated with conducting meta-research.

References

1. 1.Simsa, B., Buts, A., Ropovik, I., & Adamkovič, M. (2026). Automating data extraction in meta-research: A multi-model benchmark in network psychometrics papers. Behavior Research Methods. https://doi.org/10.1136/bmj-2021-069155