Crowdsourced versus large language models forecasting: evidence for the accuracy-correlation effect

Clinical Snapshot

Key Findings

• P Value: Not reported in abstract

• Effect Size: Robust positive association substantially exceeding independent-errors predictions

• Primary Outcome: Positive association between LLM forecasting accuracy and human-AI correlation

• Nnt Or Sensitivity: Correlations lower for superforecasters than general public; weaker for market questions than database questions

• Confidence Interval: Not reported in abstract

Clinical Application

High feasibility for research applications but limited immediate clinical utility Minimal direct relevance to Australian clinical practice, PBS, or TGA processes; may inform health technology assessment methodologies Limited direct clinical application; primarily relevant to healthcare researchers and policy makers using forecasting methods

Abstract

Over the past quarter century, crowdsourced forecasting has largely outperformed individual forecasters. Today, large language models (LLMs), aggregating human knowledge at scale, constitute a new form of collective intelligence (CI). A central question is how LLM predictive accuracy associates with human-AI correlation, and whether this relationship exceeds what would be expected if both merely track the same underlying truth. We investigate this through the accuracy-correlation effect (ACE), which posits that as algorithmic systems improve, they increasingly correlate with human predictions, potentially diminishing human value in hybrid ensembles. Using 76 model × prompt forecast sets from 16 LLMs on 580 resolved ForecastBench questions, we computed LLM accuracy and correlations with two human aggregates (superforecasters, general public), separately for databases (n = 526) and prediction markets (n = 54) questions. Linear mixed-effects models show a robust positive association between LLM accuracy and human-AI correlation that substantially exceeds independent-errors predictions. Correlations were lower for superforecasters than for the general public, and weaker for markets than for data questions. These results support ACE while indicating that increasing correlation reflects more than improved signal tracking alone, suggesting that simultaneous increases in accuracy and correlation may reduce optimal human weights in data-rich settings, while human judgement retains critical value in contextual reasoning scenarios. This article is part of the theme issue 'The evolution of collective intelligence'.

References

1. 1.Jeddi, Y., Segovia-Martin, J., & Servan-Schreiber, E. (2026). Crowdsourced versus large language models forecasting: evidence for the accuracy-correlation effect. Philosophical Transactions of the Royal Society B, 381(1897). https://doi.org/10.1098/rstb.2024.0456