Comparative study on the efficacy of large language models (LLMs) in health management for patients with urological and andrological diseases: a multidimensional quality evaluation and applicability analysis

Clinical Snapshot

Key Findings

• P Value: Significant differences found in only 3 of 30 comparisons (P < 0.001 to P = 0.024)

• Effect Size: Not reported - only p-values provided

• Primary Outcome: LLM response quality measured by PEMAT-AI and DISCERN-AI scores

• Nnt Or Sensitivity: DeepSeek superior in 2/30 quality assessments and 1/30 expert assessments; Kimi superior in readability

• Confidence Interval: Not reported

Clinical Application

Low feasibility for Australian practice due to language and cultural barriers Limited relevance to Australian practice. TGA has not approved LLMs for clinical decision support. RACGP guidelines emphasise evidence-based medicine over automated tools. Results not applicable to English-language LLMs commonly used in Australia Limited applicability - study focused on Chinese LLMs and Chinese healthcare context

Abstract

PURPOSE: This study aims to evaluate the efficacy of large language models (LLMs) in health management for urological and andrological conditions by comparing two Chinese LLMs: DeepSeek and Kimi. METHODS: The responses of DeepSeek and Kimi to 30 questions on six urological and andrological diseases were quantitatively assessed using quality assessment tools, expert assessments, and readability metrics. RESULTS: In quality assessments, DeepSeek outperformed Kimi in only two instances: responses to "What are the surgical options for kidney stones?" (PEMAT-AI: P = 0.012; DISCERN-AI: P = 0.024) and "What are the treatment modalities for bladder cancer?" (PEMAT-AI and DISCERN-AI: P < 0.001). Expert assessments revealed that DeepSeek demonstrated superior accuracy and safety compared to Kimi only in addressing "Dietary considerations for kidney stone patients" (accuracy: P < 0.001; safety: P = 0.02). No statistically significant differences were found in quality assessments or expert assessments between the models for disease-specific categories. However, Kimi outperformed DeepSeek in readability evaluations. CONCLUSION: Both DeepSeek and Kimi exhibit substantial efficacy and clinical applicability in managing urological and andrological diseases. To enhance their practical use, Kimi requires improvements in accuracy and safety, while DeepSeek should focus on improving readability.

References

1. 1.Chen, G., Bi, K., Cai, Q., Wang, Z., An, Z., & Fu, W. (2026). Comparative study on the efficacy of large language models (LLMs) in health management for patients with urological and andrological diseases: a multidimensional quality evaluation and applicability analysis. World Journal of Urology. https://doi.org/10.1002/9781394219230.ch2