Initial Insights Into an Institutional Secure Large Language Model for Magnetic Resonance Imaging Examination Requests: Retrospective Study

Clinical Snapshot

Key Findings

• P Value: P=0.027 for contrast decisions vs Rad 3, P=0.006 for musculoskeletal contrast vs Rad 4

• Effect Size: Contrast decision accuracy improved by 2.3 percentage points (94.4% vs 92.1%)

• Primary Outcome: Overall protocol accuracy: sLLM 93.1% vs radiologists 91.4-92.1%

• Nnt Or Sensitivity: RI-RADS C/D reduced from 4.1-20.4% to 0-0.7% with sLLM augmentation

• Confidence Interval: sLLM overall accuracy 95% CI 89.6-96.6%

Clinical Application

Requires significant IT infrastructure, secure hosting, and integration with existing EMR systems Would require TGA approval for clinical use, compliance with Australian privacy legislation, and integration with existing radiology information systems. Cost-effectiveness analysis needed for PBS consideration Outpatient MRI services across body, musculoskeletal, and neuroradiology subspecialties

Abstract

BACKGROUND: Incomplete clinical details on magnetic resonance imaging (MRI) examination requests (MERs) can lead to suboptimal protocol selection. An institutional secure large language model (sLLM) with access to manually retrieved salient data from the electronic medical record (EMR) may improve request completeness and protocol accuracy across multiple MRI subspecialties. OBJECTIVE: The objective of this study was to compare clinician MERs with sLLM-augmented MERs for information quality and to evaluate the protocoling accuracy of the sLLM versus board-certified radiologists across body, musculoskeletal, and neuroradiology MRI. METHODS: This retrospective study included 608 random outpatient MRI examinations performed between September 2023 and July 2024 (body 206, musculoskeletal 203, neuroradiology 199). The cohort comprised 528 patients (mean 51.2 years, SD 19.2; range 4-93; n=279, 52.8% women, n=249, 47.2% men). MERs without EMR access were excluded. A privately hosted Anthropic Claude 3.5 model (temperature 0) augmented each MER with manually retrieved salient EMR data and, via rule-based parsing, mapped the extracted elements onto predefined institutional criteria to recommend region or coverage and contrast use. Two experienced radiologists established a consensus reference standard. Two board-certified general radiologists (Rad 3 and Rad 4) and the sLLM were compared with this standard. Clinical information quality was graded using the Reason-for-Exam Imaging Reporting and Data System (RI-RADS). Interrater reliability was quantified with Gwet AC1. Paired accuracies were compared with the McNemar test to determine whether there was a statistically significant difference. RESULTS: Interreader agreement for RI-RADS was almost perfect for sLLM-augmented MERs (AC1 0.97, 95% CI 0.94-0.99) and moderate for clinician MERs (AC1 0.43, 95% CI 0.34-0.52). Limited or deficient clinical information (RI-RADS C/D) fell to 0% to 0.7% (0/608 to 4/608) with sLLM augmentation vs 4.1% to 20.4% (25/608 to 124/608) for clinician MERs. Overall protocol accuracy was 93.1% (566/608; 95% CI 89.6-96.6) for the sLLM, 91.4% (556/608; 95% CI 87.6-95.3) for Rad 3, and 92.1% (560/608; 95% CI 88.4-95.8) for Rad 4 (sLLM vs Rad 3 P=.23 vs Rad 4 P=.40). Region or coverage accuracy was similar (sLLM: 579/608, 95.2%; Rad 3: 585/608, 96.2%; Rad 4: 573/608, 94.2%; P=.46 and P=.36). Contrast decisions were more accurate using the sLLM at 94.4% (574/608; 95% CI 91.3-97.5) vs Rad 3 at 92.1% (560/608; 95% CI 88.4-95.8; P=.027) and were not significantly different to Rad 4 at 92.9% (565/608; 95% CI 89.4-96.4; P=.16). Subspecialty analyses showed similar patterns, with the sLLM outperforming Rad 4 for musculoskeletal MRI contrast decisions (96.6% vs 91.1%; P=.006) and matching readers elsewhere. Manual review indicated that sLLM improvements arose from EMR details not listed on the MER (infection/inflammation, tumor history, prior surgery). No clinically significant hallucinations were identified in a manual review of discordant cases. CONCLUSIONS: Across body, musculoskeletal, and neuroradiology MRI, sLLM-augmented examination requests improved clinical context and enhanced contrast selection while demonstrating accuracy comparable to general radiologists for region or coverage. Integrating sLLMs into routine vetting workflows may reduce manual workload in protocol selection for more efficient, standardized protocoling.

References

1. 1.Hallinan, J. T., Leow, N. W., Low, Y. X., Lee, A., Ong, W., Chan, M. D., Devi, G. K., He, S. S., Loh, D. D., Lim, D. S., Low, X. Z., Lim, M. C., Yong, C., Sng, W. J., Teo, E. C., Tan, J. H., Kumar, N., Makmur, A., & Ting, Y. (2026). Initial insights into an institutional secure large language model for magnetic resonance imaging examination requests: Retrospective study. Journal of Medical Internet Research. https://doi.org/10.2196/82579