Accuracy and completeness of large language models in Epidemic keratoconjunctivitis Queries: A Comparative study

Clinical Snapshot

Key Findings

• P Value: Not reported

• Effect Size: ChatGPT: 4.87 accuracy, 4.90 completeness; Meta AI: 4.87 accuracy, 4.93 completeness

• Primary Outcome: Mean accuracy and completeness scores on 5-point Likert scale

• Nnt Or Sensitivity: Inter-rater agreement κ = 0.78-0.88 (substantial agreement)

• Confidence Interval: Not reported

Clinical Application

High feasibility for educational purposes, limited for direct clinical decision-making Relevant for Australian ophthalmologists and GPs managing EKC, though TGA medication recommendations may differ from LLM training data Healthcare professionals seeking information about epidemic keratoconjunctivitis

Abstract

BACKGROUND: Large language models (LLMs) are increasingly applied in clinical contexts, yet their reliability in disease-specific ophthalmic domains remains insufficiently characterized. Epidemic keratoconjunctivitis (EKC), a highly contagious adenoviral ocular infection, represents a critical area where accurate diagnostic and therapeutic information is essential. This study aimed to assess and compare the accuracy and completeness of responses generated by two widely used LLMs, namely ChatGPT and Meta AI, to standardized ophthalmology questions related to EKC. METHODS: This cross-sectional expert evaluation study was conducted between January and March 2025. Thirty-four structured questions covering five EKC domains (etiology, clinical features, diagnosis, treatment, and prognosis) were developed using authoritative ophthalmology sources. Responses generated by ChatGPT (Plus subscription, web interface) and Meta AI (web interface) were independently scored by two fellowship-trained cornea specialists using a five-point Likert scale for accuracy and completeness. Inter-rater agreement was assessed using Cohen's κ and mean absolute error (MAE). RESULTS: Both models demonstrated strong overall performance, with mean accuracy and completeness scores of 4.87 and 4.90 for ChatGPT and 4.87 and 4.93 for Meta AI, respectively. Agreement between expert raters was substantial (κ range 0.78-0.88). Domain-specific analysis revealed consistently high scores across all categories, with Meta AI showing slightly higher completeness, particularly in diagnostic and symptom-related responses. CONCLUSION: LLMs exhibit consistently strong performance and domain consistency when addressing ophthalmic queries related to EKC. Their performance supports their potential use as adjunct informational and educational tools for ophthalmic conditions such as EKC, particularly for structured knowledge retrieval, treatment explanation, and patient counseling, provided that use remains under appropriate professional oversight.