Who labels best? Radiologists, rules, or large language models for CT reports on pulmonary embolism.

Clinical Snapshot

Key Findings

• P Value: p < 0.001 for hybrid vs RBE and hybrid vs radiologists

• Effect Size: Hybrid RBE-LLM workflow F1 0.99 vs RBE alone F1 0.81

• Primary Outcome: F1 score for pulmonary embolism labelling accuracy

• Nnt Or Sensitivity: 99.8% accuracy for hybrid workflow; 85.6% reduction in LLM calls compared to LLM-only approach

• Confidence Interval: F1 0.99 (95% CI: 0.98-0.99) for hybrid workflow

Clinical Application

Requires structured reporting systems and technical infrastructure for LLM implementation; may need adaptation for different reporting formats Relevant for Australian radiology departments with structured reporting; could support TGA medical device software regulations and RACGP quality improvement initiatives Radiology departments using structured CT reporting for pulmonary embolism, particularly those seeking automated quality assurance

Abstract

OBJECTIVE: To compare open-weight and proprietary large language models (LLMs), a rule-based extractor (RBE) and radiologists for labelling pulmonary embolism CT reports, and to test whether a hybrid RBE-LLM workflow improves labelling performance. MATERIALS AND METHODS: This single-centre retrospective study included structured CT reports from October 2021 to March 2025. Three labelling pipelines were evaluated: an RBE; a model-agnostic LLM extractor (18 open-weight, four GPT-4 variants); and a hybrid pipeline routing only RBE failures to an LLM. Ground truth was defined at the report-text level by deterministic schema matching for initially RBE-valid fields and blinded adjudication of RBE-invalid fields by two attending radiologists. Eight radiologists provided a human baseline. Outcomes included F1 scores, accuracy, LLM-based salvage of RBE failures, and labelling time. RESULTS: In total, 2,923 reports from 2,923 patients (mean age 66 ± 17 years; 1,465 women) were included. Falcon3-10b and GPT-4.1-mini achieved similar item-level performance (F1 0.98 [95% CI, 0.97-0.98] for both; p = 0.70) and both exceeded the RBE (F1 0.81 [95% CI, 0.80-0.82]; p < 0.001). Salvage of RBE failures was comparable between open-weight and proprietary models (88.1% vs 91.9%; p = 0.12). The hybrid RBE-LLM workflow achieved 99.8% accuracy and F1 0.99 (0.98-0.99), exceeding both the RBE and pooled radiologists (F1 0.92 [95% CI, 0.90-0.93]; all p < 0.001). CONCLUSION: Schema-constrained open-weight and proprietary LLMs exceeded rule-based extraction and, at the upper end of performance, matched a pooled radiologist label-transfer baseline. A rules-first, targeted LLM workflow enabled near-perfect extraction from finalised structured pulmonary embolism CT reports. RELEVANCE STATEMENT: A rules-first LLM workflow can automate high-fidelity extraction of structured CT findings from finalised radiology reports, enabling scalable, auditable, and more consistent cohort curation for clinical research, registries, and quality improvement. KEY POINTS: A hybrid rules-first workflow combining a rule-based extractor (RBE) with targeted large language model (LLM) salvage achieved the highest overall performance for labelling of pulmonary embolism CT reports (F1, 0.99; accuracy, 99.8%). The top standalone open-weight and proprietary LLMs (Falcon3-10b and GPT-4.1-mini) both exceeded the RBE and, at the upper end of performance, matched a pooled radiologist label-transfer baseline. The hybrid workflow reduced cohort-curation time from 32.2 h for radiologists to 1.0 h while reducing LLM calls by 85.6%, because the LLM was only triggered for rule-failed fields.

References

1. 1.Fink, M. A., Bischoff, A., Atsiatorme, E., Kremer, A., Kroschke, J., Moll, M., Stein, P., Riebl, V., Leichenich, T., Kauczor, H.-U., & Schlamp, K. (2026). Who labels best? Radiologists, rules, or large language models for CT reports on pulmonary embolism. European Radiology Experimental. https://doi.org/10.1148/ryai.230364