Comparing clinical decision-making between colposcopists and large language models in cervical dysplasia management: a pilot prospective multicenter study

Clinical Snapshot

Key Findings

• P Value: Not reported

• Effect Size: Overall concordance: clinicians 69.6%, ChatGPT-4o 69.6%, ChatGPT-5 65.2%. Subgroup: ChatGPT-5 outperformed clinicians in precancerous lesions (81.8% vs. 66.4%); clinicians outperformed LLMs in unspecific histopathology cases (86% vs. 60%)

• Primary Outcome: Concordance rate with guideline-defined gold standard treatment decisions across 23 cervical dysplasia cases

• Nnt Or Sensitivity: Not applicable to this comparative accuracy design; no sensitivity, specificity, or diagnostic accuracy metrics reported

• Confidence Interval: Not reported

Clinical Application

LLMs are commercially available and accessible, but integration into clinical workflows requires validation against local guidelines, regulatory approval, and robust governance frameworks. Multiple-choice prompting as tested here does not reflect real-world clinical complexity. Australian cervical screening follows the National Cervical Screening Program (NCSP) renewed in 2017, with management guided by NHMRC-endorsed guidelines that differ from German protocols used in this study. The Therapeutic Goods Administration (TGA) has not approved any LLM as a clinical decision support device in this context. The RACGP and RANZCOG would require prospective validation against Australian guidelines before any clinical implementation. The finding that clinicians overtreated CIN I is relevant to Australian practice, where conservative surveillance of low-grade lesions is guideline-recommended. PBS does not fund LLM-based decision support tools. Board-certified colposcopists managing patients with screen-detected cervical abnormalities, particularly in settings where decision support tools for low-grade lesion management are being considered

Abstract

PURPOSE: This prospective multicenter study aimed to compare the decision-making abilities of board-certified colposcopists and two commercially available large language models (LLM), ChatGPT-4o and ChatGPT-5, in cervical dysplasia management. METHODS: Twenty-three anonymized real-life patient cases with multiple-choice (MC) questions regarding treatment decisions were used to assess answer quality. Ten board-certified colposcopists and the two LLMs addressed the MC questions. The gold standard was defined by two guideline authors. LLMs were prompted to justify their responses. Concordance rates were calculated and compared across all questions and histopathological subgroups, including cervical intraepithelial neoplasia (CIN), unspecific histopathological results, and cervical cancer cases. RESULTS: Clinicians and LLMs achieved similar overall concordance rates compared to the gold standard (69.6% for clinicians, 69.6% for ChatGPT-4o, and 65.2% for ChatGPT-5). ChatGPT-5 outperformed clinicians in precancerous lesions (81.8% vs. 66.4%), while clinicians excelled in complex cases with unspecific histopathology (86% vs. 60%). Clinicians showed a tendency to overtreat low-grade lesions (CIN I), opting for more intensive surveillance. ChatGPT-4o performed better than ChatGPT-5 in cervical cancer cases, though both models struggled with these scenarios. CONCLUSION: This study highlights the potential of LLMs as decision support tools in cervical dysplasia management, particularly for straightforward cases like precancerous lesions. However, clinicians remain superior in handling complex or ambiguous cases. The tendency of clinicians to overtreat low-grade lesions may offer the potential to test the implementation of a decision support tool for those cases. While LLMs show promise, exploring open-ended clinical scenarios and integrating retrieval-augmented generation could enhance their practical application.

References

1. 1.Stalp, J. L., Schneider, J. A., Steinkasserer, L., Hachenberg, J., Jentschke, M., Hillemanns, P., Wolff, D., & Denecke, A. (2026). Comparing clinical decision-making between colposcopists and large language models in cervical dysplasia management: a pilot prospective multicenter study. Archives of Gynecology and Obstetrics. https://doi.org/10.1007/s00404-025-08123-4