Promise and pitfalls of AI chatbots in complex decision-making for thyroid nodules and papillary thyroid cancer

Clinical Snapshot

Key Findings

• P Value: Not reported

• Effect Size: Guideline concordance varied from 33-83% depending on model and timepoint

• Primary Outcome: AI chatbot agreement with endocrinologist responses reached maximum 25% across scenarios

• Nnt Or Sensitivity: Inter-model agreement ranged 33-67%, indicating substantial variability

• Confidence Interval: Not reported

Clinical Application

Current AI chatbots show insufficient reliability for independent clinical decision-making in complex thyroid scenarios Relevant to Australian endocrinologists following ATA/ETA guidelines; caution needed before PBS or TGA consideration of AI-assisted thyroid management tools Endocrinologists managing indeterminate thyroid nodules and low-risk papillary thyroid carcinoma

Abstract

INTRODUCTION: Artificial intelligence (AI) chatbots are increasingly used in medicine, but their reliability in scenarios with multiple management options is unclear. Indeterminate thyroid nodules and low- and low-to-intermediate-risk papillary thyroid carcinoma (PTC) represent such cases. METHODS: In a nationwide web-based survey, 201 members of the Hellenic Endocrine Society evaluated 12 clinical vignettes on indeterminate thyroid nodules and low- and low-to-intermediate-risk PTC. Their responses were compared with those generated by four conversational AI models (ChatGPT, Gemini, Copilot, and DeepSeek) at two time points, 11 months apart. DeepSeek was assessed only at the second time point. Chatbot outputs were assessed for agreement with endocrinologists' predominant answers, concordance with the most guideline-consistent options (American and European Thyroid Association recommendations), temporal stability, and inter-model agreement. RESULTS: Alignment between chatbots and endocrinologists' predominant responses was limited, reaching at most 25% across scenarios. In contrast, concordance with the most guideline-consistent options was higher, up to 83% (10/12 scenarios), depending on the model and time point. Across 12 scenarios, ChatGPT, Gemini, and Copilot changed their responses in 4, 7, and 5 scenarios, respectively, with some updates moving closer to, and others further from, guideline-based answers. Inter-model agreement ranged from 33 to 67%, indicating substantial variability among chatbots. CONCLUSION: AI chatbots show evolving but inconsistent performance in complex thyroid management scenarios. While guideline concordance can be relatively high, substantial variability across models, limited temporal reproducibility, and poor alignment with clinical practice highlight the need for ongoing longitudinal evaluation before safe integration into clinical decision-making.

References

1. 1.Effraimidis, G., Kasotas, A., Varsami, S., Sazakli, E., Karapanou, O., Saltiki, K., & Michalaki, M. (2026). Promise and pitfalls of AI chatbots in complex decision-making for thyroid nodules and papillary thyroid cancer. European Thyroid Journal. https://doi.org/10.1530/ETJ-25-0385