Error Detection in Emergency Radiology Reports Using a Large Language Model: Multistage Evaluation Study

Clinical Snapshot

Key Findings

• P Value: P=0.003 for few-shot vs 0-shot comparison

• Effect Size: 84.4% detection rate for DeepSeek-R1 in few-shot setting vs 60.9% in 0-shot setting

• Primary Outcome: Error detection rate in emergency radiology reports

• Nnt Or Sensitivity: 95% detection rate in independent validation dataset, 56.5% positive predictive value in real-world validation

• Confidence Interval: Not reported

Clinical Application

Requires integration with existing radiology information systems and training for few-shot prompting approaches Potential application in Australian emergency departments, though validation needed for English-language reports and local clinical workflows. Would need TGA consideration for clinical decision support software classification Emergency departments using radiology reporting systems, particularly in high-volume settings with time pressures

Abstract

BACKGROUND: Emergency radiology requires highly accurate reporting under time constraints; yet, increasing workloads raise the risk of errors. While large language models (LLMs) show potential for proofreading in general radiology, their performance in emergency settings and non-English contexts remains unclear. OBJECTIVE: We aim to evaluate the performance of a domain-optimized LLM, DeepSeek-R1, for identifying errors in Chinese emergency radiology reports, with comparison against assessments by board-certified radiologists. METHODS: We compiled 7435 emergency reports (dataset 1; radiography, computed tomography, and magnetic resonance imaging) collected from November 2024 to April 2025. In stage 1, a total of 5 LLMs were evaluated using 200 reports. The best model, DeepSeek-R1, proceeded to stages 2 and 3, where 0-shot and few-shot learning were tested on a separate set (n=100). Model performance was compared against 12 radiologists. Stage 4 validated real-world utility on 800 verified reports. RESULTS: In subdataset 1, under stress-testing conditions, DeepSeek-R1 achieved a higher error detection rate in the few-shot setting than in the 0-shot setting (84.4% vs 60.9%, P=.003). Its performance exceeded that of radiology residents (84.4% vs 51.6% and 53.1%, respectively; both P<.05) and showed no statistically significant difference compared with senior radiologists and attending radiologists (84.4% vs 68.8%-93.8%, P=.26 to ≥.99). Compared with residents, DeepSeek-R1 detected more critical omissions (100% vs 25% and 50%; both P<.05) and other errors (92% vs 33% and 33%; both P=.02). In dataset 2, collected from independent institutions, DeepSeek-R1 achieved a detection rate of 95% under the few-shot setting. Reading time was shorter than that of human readers (92 vs 109 s). In real-world validation, DeepSeek-R1 identified 117 true reporting errors, yielding a positive predictive value of 56.5%. CONCLUSIONS: DeepSeek-R1 holds promise for improving quality control in emergency radiology reports. Its performance and efficiency support its use as an assistive proofreading tool in real-world radiology workflows.

References

1. 1.Shen, H., Wu, T., Wang, F., Fang, J., Li, Y., Wu, X., Liu, S., Chen, L., Ren, Q., Meng, X., Xu, J., Sun, J., Zhao, Y., Liu, X., Wang, L., Mai, G., You, J., Jin, Z., Wu, X., He, W., Han, X., Zhang, S., Zeng, D., & Zhang, B. (2026). Error detection in emergency radiology reports using a large language model: Multistage evaluation study. Journal of Medical Internet Research. https://doi.org/10.2196/86841