Performance of a large language model on the reasoning tasks of a physician.

Clinical Snapshot

Key Findings

• P Value: Not reported

• Effect Size: Not quantified in abstract - described as LLM 'outperformed physician baselines'

• Primary Outcome: Clinical reasoning performance comparison between LLM and physicians

• Nnt Or Sensitivity: Not reported - diagnostic accuracy metrics needed

• Confidence Interval: Not reported

Clinical Application

Implementation would require significant infrastructure, training, and integration with existing clinical workflows Relevance to Australian practice unclear - would need TGA approval for clinical decision support tools, consideration of medico-legal implications, and integration with existing electronic health records Emergency department physicians and potentially other specialists involved in complex diagnostic reasoning

Abstract

More than 65 years ago, complex clinical diagnostic reasoning cases were introduced as the gold standard for the evaluation of expert medical computing systems, a standard that has held ever since. In this study, we report the results of a physician evaluation of a large language model (LLM) on challenging clinical cases across five experiments with a baseline of hundreds of physicians. We then report a real-world study comparing human expert and artificial intelligence (AI) second opinions in randomly selected patients in the emergency room of a major tertiary academic medical center. In all experiments, the LLM outperformed physician baselines and displayed continued improvement from prior generations of AI clinical decision support. Our study suggests that LLMs have eclipsed most benchmarks of clinical reasoning, motivating the urgent need for prospective trials.

References

1. 1.Brodeur, P. G., Buckley, T. A., Kanjee, Z., Goh, E., Ling, E. B., Jain, P., Cabral, S., Abdulnour, R. E., Haimovich, A. D., Freed, J. A., Olson, A., Morgan, D. J., Hom, J., Gallo, R., McCoy, L. G., Mombini, H., Lucas, C., Fotoohi, M., Gwiazdon, M., Restifo, D., Restrepo, D., Horvitz, E., Chen, J., Manrai, A. K., & Rodman, A. (2026). Performance of a large language model on the reasoning tasks of a physician. Science, 384(6695). https://doi.org/10.1126/science.adz4433