Comparing the performance of four mainstream large language models on medical literature review generation: a human expert evaluation in SMILE surgery

Clinical Snapshot

Key Findings

• P Value: p < 0.001 for model comparisons and reference authenticity differences

• Effect Size: Statistically significant differences between models (p < 0.001)

• Primary Outcome: Significant performance variations across all four LLMs and evaluation dimensions

• Nnt Or Sensitivity: Grok-3 demonstrated highest reference authenticity; GPT-4 showed lowest reference authenticity

• Confidence Interval: Not reported in abstract

Clinical Application

Highly feasible given widespread LLM availability, though requires expert oversight for reference verification Relevant to RANZCO members and Australian ophthalmology research community; aligns with emerging TGA guidance on AI medical applications Australian ophthalmologists and researchers considering LLM tools for literature review assistance in refractive surgery

Abstract

PURPOSE: To systematically evaluate and compare the performance of four leading large language models (LLMs) in generating medical literature reviews across topics of varying research maturity, thereby providing insights for their effective and responsible application in academic writing. METHODS: In this comparative study, using standardized prompts, we instructed four leading LLMs (GPT-4, Gemini 2.5 Pro, Grok-3, and DeepSeek R1) to generate literature reviews on nine topics related to small incision lenticule extraction (SMILE) surgery. These topics were categorized into three groups by research maturity: well-researched, controversial, and open. Seven ophthalmology experts evaluated the generated content across four dimensions: quality, accuracy, bias, and relevance, while all references were verified for authenticity. Performance differences among models were evaluated using group comparison tests followed by post-hoc analysis. RESULTS: Significant performance variations were identified across all four models and dimensions (p < 0.001). Specifically, Gemini ranked highest in content quality, accuracy, and bias control. In contrast, DeepSeek, despite its high-quality score, received the lowest relevance score. Grok-3 demonstrated the highest reference authenticity (p < 0.001), whereas GPT-4's was the lowest (p < 0.001). All models showed diminished performance on open topics and exhibited severe reference fabrication ("hallucinations"). CONCLUSION: Rather than excelling universally, LLMs exhibit distinct and task-specific strengths that mandate a task-driven, hybrid strategy in tool selection. Reference fabrication was found to be a pervasive issue across all models, regardless of the task topic, elevating human verification from a best practice to an essential safeguard for academic integrity.

References

1. 1.Zhou, M., Gui, F., Ai, C., Ling, L., Zhang, X., Lu, Y., Han, D., Zhao, B., Zhong, F., Liu, J., Zhu, Z., Li, J., Huang, F., Lin, C., Liu, W., & Xiong, J. (2026). Comparing the performance of four mainstream large language models on medical literature review generation: a human expert evaluation in SMILE surgery. Graefe's Archive for Clinical and Experimental Ophthalmology. https://doi.org/10.1001/jamaophthalmol.2025.3153