Man vs. Machine: can ML algorithms diagnose headaches as accurately as clinicians? A systematic review.

Clinical Snapshot

Key Findings

• P Value: Not reported for pooled analysis

• Effect Size: Sensitivity 47.5-100%, Specificity 50.4-100%

• Primary Outcome: Diagnostic accuracy of AI/ML models for headache classification

• Nnt Or Sensitivity: Realistic models: 74-86% accuracy; Neuroimaging models: near-perfect but likely overfit

• Confidence Interval: AUC-ROC 0.658-1.000 (wide variability across studies)

Clinical Application

Low feasibility for immediate implementation; requires prospective real-world validation Relevant to Australian neurology and general practice; aligns with TGA requirements for medical device validation before clinical use Currently not applicable to clinical practice due to methodological limitations

Abstract

BACKGROUND: Headache disorders are frequently misdiagnosed. We aimed to systematically evaluate the diagnostic accuracy, methodological quality, and clinical applicability of artificial intelligence (AI) and machine learning (ML) models for classifying adult headache disorders against clinician diagnoses using the International Classification of Headache Disorders (ICHD) criteria. METHODS: In this systematic review, we searched PubMed, Embase, and the Cochrane Library (January 2015-December 2025) for AI/ML diagnostic headache studies. Two reviewers extracted data and assessed risk of bias using the QUADAS-2 tool with the QUADAS-AI extension. Main outcomes were sensitivity, specificity, area under the receiver operating characteristic curve (AUC-ROC), and risk of bias. RESULTS: We included 74 studies encompassing 154,856 participants. Models utilized traditional ML (n = 47), deep learning (n = 18), and hybrid or rule-based approaches (n = 9). Data inputs included neuroimaging (n = 27), multimodal datasets (n = 20), neurophysiological signals (n = 17), and clinical questionnaires (n = 10). Only 4 studies performed independent external validation. Overall sensitivity ranged from 47.5% to 100.0%, specificity from 50.4% to 100.0%, and AUC-ROC from 0.658 to 1.000. Models using structured questionnaires reported realistic accuracies (74-86%), whereas neuroimaging models frequently produced near-perfect, likely overfit estimates. Most studies (65 of 74) exhibited a high risk of bias driven by artificial case-control designs, data leakage, and absent external validation. CONCLUSIONS: AI-based headache diagnostic systems demonstrate promising but highly variable accuracy. Pervasive methodological flaws-specifically severe spectrum bias, data leakage, and a lack of independent external validation-currently preclude clinical implementation. Future studies require prospective, real-world validation to safely integrate these tools into practice.

References

1. 1.Manickam, Appukutty, Valliappan, Abirami. (2026). Man vs. Machine: can ML algorithms diagnose headaches as accurately as clinicians? A systematic review. Journal of Neurology. https://doi.org/10.1089/neu.2018.5916