Transforming evidence synthesis: A systematic review of the evolution of automated meta-analysis in the age of AI

Clinical Snapshot

Key Findings

• P Value: Not applicable for descriptive synthesis

• Effect Size: Only 16.4% of studies address advanced synthesis stages

• Primary Outcome: Current AMA predominantly focuses on data processing automation (52.5%) rather than comprehensive synthesis

• Nnt Or Sensitivity: Approximately 2% of studies achieved preliminary full-process automation

• Confidence Interval: Not reported

Clinical Application

Current AMA tools may assist with data extraction but insufficient for replacing manual meta-analysis processes Relevant to NHMRC evidence synthesis guidelines, Cochrane Australia activities, and TGA regulatory evidence requirements Healthcare researchers, systematic reviewers, evidence synthesis teams, clinical guideline developers

Abstract

Exponential growth in scientific literature has heightened the demand for efficient evidence-based synthesis, driving the rise of the field of automated meta-analysis (AMA) powered by natural language processing and machine learning. This PRISMA systematic review introduces a structured framework for assessing the current state of AMA, based on screening 13,216 papers (2006-2024) and analyzing 61 studies across diverse domains. Findings reveal a predominant focus on automating data processing (52.5%), such as extraction and statistical modeling, while only 16.4% address advanced synthesis stages. Just one study (approximately 2%) explored preliminary full-process automation, highlighting a critical gap that limits AMA's capacity for comprehensive synthesis. Despite recent breakthroughs in large language models and advanced AI, their integration into statistical modeling and higher-order synthesis, such as heterogeneity assessment and bias evaluation, remains underdeveloped. This has constrained AMA's potential for fully autonomous meta-analysis (MA). From our dataset spanning medical (67.2%) and non-medical (32.8%) applications, we found that AMA has exhibited distinct implementation patterns and varying degrees of effectiveness in actually improving efficiency, scalability, and reproducibility. While automation has enhanced specific meta-analytic tasks, achieving seamless, end-to-end automation remains an open challenge. As AI systems advance in reasoning and contextual understanding, addressing these gaps is now imperative. Future efforts must focus on bridging automation across all MA stages, refining interpretability, and ensuring methodological robustness to fully realize AMA's potential for scalable, domain-agnostic synthesis.

References

1. 1.Li, L., Mathrani, A., & Susnjak, T. (2026). Transforming evidence synthesis: A systematic review of the evolution of automated meta-analysis in the age of AI. Research Synthesis Methods. https://doi.org/10.1109/BSB.2016.7552152