Applying machine learning and natural language processing to patient safety event reports: Identifying patterns of cardiovascular diagnostic errors

Clinical Snapshot

Key Findings

• P Value: Not reported

• Effect Size: AUROC = 0.914

• Primary Outcome: XGBoost model performance in identifying cardiovascular diagnostic errors

• Nnt Or Sensitivity: Specificity = 0.982, PPV = 0.866, F-1 score = 0.738

• Confidence Interval: Not explicitly reported

Clinical Application

Technically feasible but requires significant computational resources and expertise in ML/NLP implementation Relevant to Australian healthcare systems using incident reporting (e.g., AIMS, state-based systems). Could complement TGA device safety monitoring and support RACGP diagnostic error reduction initiatives Healthcare systems with electronic PSE reporting systems, particularly those managing cardiovascular patients

Abstract

OBJECTIVE: To explore the utility of natural language processing (NLP) and machine learning (ML) techniques to identify unsafe conditions leading to cardiovascular diagnostic errors using patient safety event (PSE) reports data. METHODS: PSE reports from January 2016 to August 2021 from a multi-hospital healthcare system in the mid-Atlantic region of the United States were included in this study. To have the true cardiovascular diagnostic errors labels for PSE reports, each individual PSE report was manually reviewed to find clinician-reported narratives describing current definitions of cardiovascular diagnostic errors. The PSE reports which contained cardiovascular diagnostic errors-related narratives were labeled as one and zero otherwise. Four binary ML models were employed to identify cardiovascular diagnostic errors narratives and common features from annotated PSE reports data: (1) simple logistic regression, (2) elastic net, (3) XGBoost, and (4) deep neural networks. RESULTS: XGBoost outperformed the rest of the models in identifying cardiovascular diagnostic errors -related reports and achieved high performance metrics on the testing data (AUROC = 0.914, specificity = 0.982, PPV = 0.866, accuracy = 0.929, F-1 score = 0.738, and AUPRC = 0.783). Pacemaker emerged as a significant signal for cardiovascular diagnostic errors in our PSE reports. Our analysis demonstrated that ordering MRI for patients with pacemaker was a frequent theme among cardiovascular diagnostic errors-related PSE reports containing the word pacemaker. Order, EKG, cardiac, and chest were the five most important features in identifying cardiovascular diagnostic errors events. These words were utilized in explaining heart conditions, associated care process, and related safety incidents. CONCLUSIONS: Findings from our study demonstrates the feasibility of ML and NLP techniques in identifying cardiovascular diagnostic errors-related reports among the existing PSE data sources. However, validation in external healthcare systems is needed before broader application.

References

1. 1.Tabaie, A., Tran, A. K., Parau, C., Bennett, S. S., Kazi, S., Smith, K., Yosaitis, J., & Miller, K. E. (2026). Applying machine learning and natural language processing to patient safety event reports: Identifying patterns of cardiovascular diagnostic errors. PLoS One. https://doi.org/10.1371/journal.pone.0345693