Retrospective validation study of a large language model approach to screening for intraabdominal surgical site infections for quality and safety reporting

Clinical Snapshot

Key Findings

• P Value: Not reported

• Effect Size: LLM identified 104 high-risk cases vs 288 cases by EHR workflow

• Primary Outcome: Diagnostic accuracy of LLM screening for intra-abdominal surgical site infections

• Nnt Or Sensitivity: PPV 53.8% (LLM) vs 19.4% (EHR); NPV 100% for both methods

• Confidence Interval: Not reported

Clinical Application

Requires local LLM hosting capability and integration with existing surveillance workflows. Cost approximately $0.05 per case appears feasible. Relevant to Australian hospitals implementing National Safety and Quality Health Service Standards for infection prevention. Could support TGA post-market surveillance requirements and align with RACGP quality improvement initiatives. Patients undergoing abdominal surgery in hospitals with comprehensive EHR systems and infection prevention programmes

Abstract

Surgical site infections (SSIs), particularly intra-abdominal (IAB) infections, are challenging to identify and remain a resource-intensive focus of infection prevention programmes. Current automated screening measures rely on discrete data from the electronic health record (EHR), such as microbiology results, diagnosis codes and/or return to the operating room. This approach has poor specificity, and therefore surveillance methods depend heavily on additional manual chart review by trained infection preventionists. Large language models (LLMs) offer an opportunity to improve surveillance by synthesising complex clinical documentation alongside structured data elements.We evaluated the performance of a locally hosted LLM (gpt-35-turbo-16k) to improve IAB SSI screening using perioperative clinical notes and microbiology results. The model analysed documentation across the perioperative period (3 days before through 30 days after surgery) to generate case-level SSI summaries and likelihood assessments. We compared the performance of this tool against the current EHR-based screening workflow.Among 1977 abdominal surgical cases, including 56 with confirmed IAB SSIs, the LLM screened 104 cases as high risk, identifying all infections (negative predictive value (NPV) 100%) and achieving a positive predictive value (PPV) of 53.8%. In contrast, the EHR-based workflow identified 288 cases for further review, with a PPV of only 19.4% and the same NPV of 100%. Analysis of 57 224 notes required ~107 million tokens, translating to approximately USD 0.05 per case.An LLM-based approach to SSI surveillance has the potential to substantially improve efficiency while remaining highly accurate and cost-effective. By reducing reliance on manual chart review, this strategy could allow infection preventionists to shift their attention from surveillance toward quality improvement and patient safety initiatives.

References

1. 1.Pierce, L., Pak, C., Stanley, K., Wang, L., Erickson, C., Yokoe, D. S., & Wick, E. (2026). Retrospective validation study of a large language model approach to screening for intraabdominal surgical site infections for quality and safety reporting. BMJ Quality & Safety. https://doi.org/10.1136/bmjqs-2025-019493