Physician Gestalt Compared With Artificial Intelligence Model to Predict Intubation in Critically Ill Patients

Clinical Snapshot

Key Findings

• P Value: p = 0.013 for sensitivity comparison, p < 0.001 for specificity comparison

• Effect Size: Physician sensitivity 0.190 vs Vent.io 0.714; Physician specificity 0.960 vs Vent.io 0.673

• Primary Outcome: Sensitivity and specificity of intubation prediction within 24 hours

• Nnt Or Sensitivity: Vent.io demonstrated 71.4% sensitivity vs physician 19.0% sensitivity for identifying patients requiring intubation

• Confidence Interval: OR for Vent.io vs physician sensitivity: 18.68 (95% CI, 1.87-186.31)

Clinical Application

Physician gestalt readily available but limited sensitivity; ML model requires technological infrastructure and real-time implementation Relevant to Australian ICUs; would require TGA approval for clinical ML model deployment; aligns with ANZICS quality improvement initiatives for predictive analytics in critical care Adult patients in intensive care units where intubation prediction could guide early intervention

Abstract

IMPORTANCE: Accurate prediction of intubation in critically ill patients could enable interventions that improve patient outcomes. However, the performance of intensive care physicians compared with machine learning (ML) models remains unknown. OBJECTIVES: To investigate intensive care physicians' ability to predict the need for intubation within 24 hours and compare their performance against an established ML model, Vent.io. DESIGN, SETTING, AND PARTICIPANTS: This prospective observational study in two ICUs surveyed intensivists to test their ability to predict the need for intubation of adult patients under their care. Physician predictions of intubation were then compared with predictions from Vent.io. MAIN OUTCOMES AND MEASURES: Primary metrics include prediction sensitivity, specificity, and descriptive statistics for both physicians and ML model. Generalized linear mixed models investigated the fixed effect of the predictor (physician vs. Vent.io) on both sensitivity and specificity while accounting for the random effects from different physicians. Similar modeling was used to investigate the relationship between physician confidence and correctness. RESULTS: Overall, physicians were quite confident in their predictions of intubation with a median score of 8 (on a 0-10 point scale, with 0 being not at all confident and 10 being extremely confident) out of the 302 surveys administered. Sensitivity was 0.190 and 0.714 for physicians and Vent.io, respectively. Specificity was 0.960 and 0.673 for physicians and Vent.io, respectively. Generalized linear mixed modeling showed that physician confidence was associated with greater odds of correctly predicting intubation outcome (odds ratio [OR] 1.49; 95% CI, 1.22-1.84; p < 0.001). Vent.io had significantly greater odds of being correct when patients required intubation compared with physicians (OR 18.68; 95% CI, 1.87-186.31; p = 0.013). However, intensive care physicians outperformed Vent.io at correctly predicting when patients did not require intubation (OR 24.80; 95% CI, 13.22-46.52; p < 0.001). CONCLUSIONS AND RELEVANCE: Although predictive performance compared with human experts is promising, Vent.io needs real-time testing in a randomized clinical trial to determine if its deployment can improve clinical outcomes.

References

1. 1.Miller, M. A., Lu, X., Pearce, A., Malhotra, A., & Nemati, S. (2026). Physician gestalt compared with artificial intelligence model to predict intubation in critically ill patients. Critical Care Explorations, Article 1393. https://doi.org/10.1097/CCE.0000000000001393