Automated extraction of fluoropyrimidine treatment and treatment-related toxicities from clinical notes using natural language processing.

Clinical Snapshot

Key Findings

• P Value: Not reported

• Effect Size: Error analysis prompting: Treatment F1=1.000, Toxicity F1=0.965; Rule-based: Treatment F1=1.000, Toxicity F1=0.904

• Primary Outcome: F1 scores for treatment and toxicity extraction across different NLP methods

• Nnt Or Sensitivity: Error analysis prompting achieved 100% precision and recall for treatment extraction, 96.5% F1 for toxicity extraction

• Confidence Interval: Not reported

Clinical Application

High feasibility for implementation in EHR systems, though requires computational resources for LLM-based approaches Relevant to Australian oncology practice for PBS-subsidised fluoropyrimidine monitoring and TGA pharmacovigilance requirements, though validation needed for local clinical documentation patterns Oncology patients receiving fluoropyrimidine therapy in healthcare systems with electronic health records

Abstract

OBJECTIVE: Fluoropyrimidines are widely prescribed for colorectal and breast cancers, but are associated with toxicities such as hand-foot syndrome and cardiotoxicity. Since toxicity documentation is often embedded in clinical notes, we aimed to develop and evaluate natural language processing (NLP) methods to extract treatment and toxicity information. MATERIALS AND METHODS: We constructed a gold-standard dataset of 236 clinical notes from 204,165 adult oncology patients. Domain experts annotated categories related to treatment regimens and toxicities. We developed rule-based, machine learning-based (Random Forest [RF], Support Vector Machine [SVM], Logistic Regression [LR]), deep learning-based (BERT, ClinicalBERT), and large language models (LLM)-based NLP approaches (zero-shot and error analysis prompting). A 5-fold cross validation were conducted to validate each model. RESULTS: Error analysis prompting achieved optimal precision, recall, and F1 scores for treatment (F1 = 1.000) and toxicities extraction (F1 = 0.965), whereas zero-shot perform moderately (treatment F1 = 0.889, toxicities extraction F1 = 0.854) Rule-based reached F1 = 1.000 for treatment and F1 = 0.904 for toxicities extraction. LR and SVM ranked second and fourth for toxicities extraction (LR F1 = 0.914, SVM F1 = 0.903). Deep learning and RF underperformed, with performance of BERT reached F1 = 0.792 for treatment and F1 = 0.837 for toxicities extraction.,ClinicalBERT reached F1 = 0.797 for treatment and F1 = 0.884 for toxicities extraction). RF reached F1 = 0.745 for treatment and F1 = 0.853 for toxicities extraction. DISCUSSION: LMM-based error analysis outperformed all others, followed by machine learning methods. Machine learning and deep learning methods were limited by small training data and showed limited generalizability, particularly for rare categories. CONCLUSION: LLM-based error analysis most effectively extracted fluoropyrimidine treatment and toxicity information from clinical notes, and has strong potential to support oncology research and pharmacovigilance.

References

1. 1.Wu, X., Kreider, M. S., Empey, P. E., Li, C., & Wang, Y. (2026). Automated extraction of fluoropyrimidine treatment and treatment-related toxicities from clinical notes using natural language processing. International Journal of Medical Informatics. https://doi.org/10.1038/s41746-020-0221-y