Developing a natural language processing system using transformer-based models for adverse drug event detection in electronic health records

Clinical Snapshot

Key Findings

• P Value: Not reported; no statistical significance testing between model configurations performed

• Effect Size: n2c2 dataset — best AUPRC: 0.840 (Clinical-Longformer, 4-chunk split); best micro F1: 0.965 (BioBERT, 15-word window); best macro F1: 0.852 (Clinical-Longformer, 10-chunk split). VUMC dataset — best AUPRC: 0.536 (Clinical-Longformer, 6-chunk split); best micro F1: 0.966 (BERT-base-uncased, 6-chunk split); best macro F1: 0.762 (Clinical-Longformer, 4-chunk split)

• Primary Outcome: ADE detection performance measured by AUPRC, micro F1, and macro F1 on the n2c2 2018 benchmark dataset (5-fold cross-validation) and the VUMC external EHR dataset

• Nnt Or Sensitivity: Sensitivity and specificity not reported separately; AUPRC (area under precision-recall curve) is the primary discriminative metric. The AUPRC of 0.536 on the VUMC dataset indicates substantially reduced precision-recall performance in real-world application compared to the benchmark (0.840), suggesting meaningful false positive and false negative burden in external deployment

• Confidence Interval: Not reported for any metric

Clinical Application

Transformer-based NLP models require substantial computational infrastructure (GPU resources), annotated training data, and ongoing maintenance. The annotation burden for creating institution-specific training data is acknowledged as a practical barrier but not quantified. The split-based preprocessing approach identified as superior is computationally more tractable than window-based methods for long clinical notes. Real-time deployment feasibility is not assessed. No Australian EHR data were used in this study. Australian clinical documentation practices, drug naming conventions (including PBS-listed drug names and TGA-approved terminology), and EHR systems (e.g., Epic, Cerner, Sunrise implementations used across Australian health networks) differ from US counterparts. The TGA's Database of Adverse Event Notifications (DAEN) and the RACGP's emphasis on medication safety in primary care provide a policy context for ADE surveillance, but direct application of these models to Australian EHRs would require prospective local validation. Australian health services considering NLP-based pharmacovigilance tools should treat this study as proof-of-concept rather than implementation-ready evidence. The Australian Commission on Safety and Quality in Health Care (ACSQHC) medication safety standards would need to be considered in any deployment framework. Healthcare institutions seeking to implement automated pharmacovigilance systems for ADE detection from unstructured EHR clinical notes, particularly in settings with large volumes of free-text documentation. Most directly applicable to adult inpatient settings with documentation styles similar to US academic medical centres.

Abstract

OBJECTIVE: To develop a transformer-based natural language processing (NLP) system for detecting adverse drug events (ADEs) from clinical notes in electronic health records (EHRs). MATERIALS AND METHODS: We fine-tuned BERT Short-Formers and Clinical-Longformer using the processed dataset from the 2018 National NLP Clinical Challenges (n2c2) shared task Track 2. Two data processing methods, window-based and split-based approaches, were compared to identify the optimal processing method. Model generalizability was evaluated on a dataset extracted from Vanderbilt University Medical Center (VUMC) EHRs. RESULTS: On the n2c2 dataset, the best 5-fold cross-validation AUPRC, micro F1, and macro F1 scores were 0.840 (Clinical-Longformer, 4-chunk split), 0.965 (BioBERT, 15-word window), and 0.852 (Clinical-Longformer, 10-chunk split). On the VUMC dataset, the best AUPRC, micro F1, and macro F1 scores were 0.536 (Clinical-Longformer, 6-chunk split), 0.966 (BERT-base-uncased, 6-chunk split), and 0.762 (Clinical-Longformer, 4-chunk split). DISCUSSION: Transformer-based models demonstrated strong performance for ADE detection, with split-based processing generally outperforming window-based methods. Clinical-Longformer combined with a practical split-based approach showed promise for real-world implementation. Beyond token limits, chunk size substantially influenced model performance, even when text length remained within limits. CONCLUSION: Our findings provide practical guidance for developing transformer-based ADE detection systems from clinical notes. The selection of both text preprocessing strategies and model architectures should be guided by note characteristics and practical considerations such as annotation burden.

References

1. 1.Wu, J., Ruan, X., McNeer, E., Rossow, K. M., & Choi, L. (2026). Developing a natural language processing system using transformer-based models for adverse drug event detection in electronic health records. PLoS ONE. https://doi.org/10.1371/journal.pone.0350516