Potential of large language models for rapid clinical information support: evidence from acute kidney injury knowledge testing.
Clinical Snapshot
PICO Framework
| P — Population | 123 volunteer medical professionals at the 131st Annual Congress of the German Society of Internal Medicine and 13 publicly available large language models |
| I — Intervention | Large language model performance assessment |
| C — Comparator | Human physician performance on standardized clinical knowledge assessment |
| O — Outcomes | AKI knowledge assessment scores (out of 15), completion time, and accuracy rates |
Bottom Line
This cross-sectional study demonstrates that large language models significantly outperformed medical professionals on standardized acute kidney injury knowledge assessments (90% vs 48.7% accuracy) with substantially faster completion times. While these findings suggest potential for LLMs as rapid clinical knowledge support tools, several methodological limitations warrant caution. The study used volunteer participants from a German medical congress, potentially introducing selection bias, and lacked adjustment for important confounders such as clinical specialty and experience level. No statistical testing or confidence intervals were reported. The practical implications for real-world clinical care remain unclear, as knowledge testing may not translate to appropriate clinical decision-making in complex patient scenarios. Australian clinicians should view these results as preliminary evidence supporting further research into LLM applications, while maintaining that human clinical judgment remains essential for safe, contextualised patient care.
Key Findings
P Value: Not reported
Effect Size: LLMs: mean 13.5/15 (90%) vs Humans: mean 7.3/15 (48.7%)
Primary Outcome: AKI knowledge assessment score out of 15 points
Nnt Or Sensitivity: Only 16.3% of human participants scored ≥11 points vs several LLMs achieving perfect scores
Confidence Interval: Not reported
Clinical Application
High feasibility given rapid completion time (0.5 vs 7.3 minutes) and accessibility of LLMs Relevant to Australian nephrology and emergency medicine practice; could support clinical decision-making but requires validation with ANZDATA registry outcomes and local clinical guidelines Medical professionals requiring rapid access to AKI clinical knowledge
Abstract
Large language models (LLMs) are increasingly applied in clinical contexts; however, to our knowledge, their performance in acute kidney injury (AKI) has not been directly compared with physician performance on standardized clinical knowledge assessments. We conducted a cross-sectional study to evaluate the performance of 13 publicly available LLMs and 123 volunteer participants at the 131st Annual Congress of the German Society of Internal Medicine in Wiesbaden, Germany. Both groups completed an identical AKI knowledge assessment consisting of two case vignettes and 15 single-best-answer multiple-choice questions. LLMs achieved a mean score of 13.5 out of 15 (90%), with several models reaching a perfect score, while human participants averaged 7.3 out of 15 (48.7%). Only 16.3% of participants scored 11 points or higher. As an illustrative example, ChatGPT-4o completed the test in approximately 0.5 minutes, whereas humans required a mean of 7.3 minutes. These findings demonstrate that LLMs substantially outperformed a heterogeneous group of medical professionals in AKI knowledge assessments and did so with markedly greater efficiency. While this highlights the potential of LLMs as rapid, cost-effective tools for clinical knowledge support, their role in real-world patient care remains undetermined, and human clinical judgment remains essential to ensure safe, context-sensitive, and patient-centered care.
References
- 1.Russ, P., Bedenbender, S., Einloft, J., Meyer, H. L., Wenzel, L. T., Ganser, A., Hirsch, M. C., & Grgic, I. (2026). Potential of large language models for rapid clinical information support: evidence from acute kidney injury knowledge testing. Scientific Reports. https://doi.org/10.1186/s12911-020-01332-6
Related Research
International urology and nephrology
Acute kidney injury prediction and prognostication using machine learning
1 Aug 2026
Journal of robotic surgery
Robot-assisted versus open kidney transplantation: an umbrella review of systematic reviews and meta-analyses
27 July 2026
Journal of robotic surgery
Transperitoneal versus retroperitoneal single-port robot-assisted partial nephrectomy: systematic review and meta-analysis of perioperative and functional outcomes
21 July 2026
This content is for educational purposes for healthcare professionals only and does not constitute clinical advice. Clinical decisions should be based on individual patient assessment, current guidelines, and appropriate specialist consultation. Editorial Standards · Privacy Policy · Terms of Service