Research Appraisals
Evidence-based critical appraisals of the latest medical research, systematically evaluated using Oxford CEBM methodology.
Showing 2 appraisals
World journal of urology
Assessing multiple chatGPT versions on novel content in the Taiwan urology board examination: accuracy, speed, and domain-specific performance
PURPOSE: Large language models demonstrate increasing utility in healthcare; however, their capacity to handle newly developed medical content beyond training data boundaries remains unclear. This study evaluated the performance of four ChatGPT models on 120 newly developed questions from the 2024 Taiwan Urology Board Examination, all created after the models' October 2023 training cutoff, thereby reducing the risk of prior model exposure. METHODS: Model performance was assessed using accuracy and processing time. Of 150 examination items, the first 120 newly developed questions constituted the primary analysis set; the remaining 30 previously used archival questions were excluded to reduce potential bias from prior exposure. The primary paired accuracy comparison evaluated o1-preview versus GPT-4o across all 120 questions. Additional analyses across 12 urological subspecialties and question-complexity strata were treated as exploratory. RESULTS: Across all 120 questions, o1-preview achieved 66.7% accuracy and significantly outperformed GPT-4o (55.8%; p = 0.012), although with longer processing time (19.20 vs. 14.94 s). Performance varied significantly across 12 urological subspecialties (p < 0.001). Unlike the other tested models, o1-preview showed slightly higher accuracy on high-complexity than on low-complexity questions (68.3% vs. 65.0%). On high-complexity surgical anatomy items, o1-preview achieved 80.0% accuracy, whereas GPT-4o and GPT-4o mini both scored 0.0% (p = 0.024). CONCLUSION: On this newly developed post-cutoff, text-based urology board question set, o1-preview achieved higher overall accuracy than GPT-4o, particularly on high-complexity surgical anatomy items, at the cost of longer response times. These findings suggest that reasoning-oriented models may offer advantages for selected specialty-level assessment tasks, although generalizability to other languages, modalities, and clinical settings requires further validation.
30 May 2026
Read appraisal →JMIR formative research
Performance of DeepSeek V3, DeepSeek R1, ChatGPT 4o, and ChatGPT o1 on the National Health Professional and Technical Qualification Examination (Intermediate Level) in China: Comparative Analysis
BACKGROUND: In recent years, large language models (LLMs) have undergone swift cycles of refinement and iteration. However, in the realm of clinical medicine, different LLMs' capability of logical reasoning and disease diagnosis needs further investigation. OBJECTIVE: The aim of our study was to evaluate the performance of 4 different LLMs in the National Health Professional and Technical Qualification Examination in China. METHODS: A total of 398 multiple-choice questions of 5 different question types were integrated within the examination with respect to the diagnosis or care of cases. These questions were categorized into different cardiology subspecialties and different clinical disciplines. DeepSeek V3 and R1 were accessed through an application programming interface, while ChatGPT 4o and o1 were queried via its public chat-based interface. We offered the same prompts instructing LLMs to assume the role of a physician and provide answers with explanations at the beginning of each conversation. We assessed different LLMs' performance by the accuracy in the responses to the multiple-choice questions. For the first 3 examination sections, McNemar test was used to compare the accuracy among the models, with post hoc pairwise comparisons performed using partitions of chi-square method and Bonferroni correction (significance set at P<.008). For the fourth section involving partially credit scoring, one-way ANOVA was performed to compare the mean scores among the models, with statistical significance set at P<.05. RESULTS: Both DeepSeek V3 and R1 showed superior performance in the first 3 sections of the Chinese National Health Professional and Technical Qualification Examination, achieving an overall performance of 93% and 93.6%, respectively. ChatGPT 4o and o1 achieved accuracies of 73.3% and 69%, respectively (all P<.001 compared with DeepSeek V3). For the fourth section, the performance of all 4 LLMs markedly declined compared to their results in the preceding sections. Particularly, in the section of gastroenterology and hematology, DeepSeek V3 achieved the highest accuracy, while R1 ranked first in cardiology and neurology. ChatGPT o1 achieved the highest accuracy in the topic of coronary artery disease, with no statistical significance. CONCLUSIONS: DeepSeek V3 and R1 showed remarkable potential in facilitating clinical decision-making in the Chinese professional examination, with both outperforming ChatGPT 4o and o1. Nonetheless, future research should continue evaluating their economic efficiency and susceptibility to hallucination.
7 Apr 2026
Read appraisal →