Benchmarking large language models for de-identification of electronic health record notes

Clinical Snapshot

Key Findings

• P Value: Not reported

• Effect Size: Best LLM-based model (supervised fine-tuning, combined corpus, setting 9): strict F1 = 0.9447; Best baseline model (combined corpus, setting 3): strict F1 = 0.8172; absolute improvement of 0.1275 F1 points (~12.75 percentage points)

• Primary Outcome: De-identification performance (strict F1 micro-average score) across five benchmark EHR datasets using rule-based baseline and LLM-based models in nine experimental settings

• Nnt Or Sensitivity: Sensitivity (recall) and specificity not separately reported in abstract; strict F1 score used as composite accuracy metric. For de-identification tasks, recall for sensitive health information is the critical safety parameter — not reported separately.

• Confidence Interval: Not reported

Clinical Application

Fine-tuned LLM deployment requires substantial computational infrastructure, access to labelled training data, and robust data governance frameworks. The combined-corpus training approach (highest performing) requires multi-institutional data sharing for model development, which presents significant governance and consent challenges. Rule-based and hybrid approaches remain more immediately deployable in resource-constrained settings. Australian relevance is limited but growing. The My Health Record system and state-based EHR platforms (e.g., NSW HealtheNet, Victorian VHIMS) generate large volumes of clinical text requiring de-identification for secondary research use under the My Health Records Act 2012 and Privacy Act 1988. The OAIC's Australian Privacy Principles impose strict requirements on SHI handling. TGA's emerging regulatory framework for Software as a Medical Device (SaMD) may apply to clinical de-identification tools used in diagnostic or treatment contexts. RACGP standards for patient data management would require validation of any deployed de-identification system against Australian-specific identifier types (Medicare numbers, IHI, Australian addresses, state-specific facility codes). The predominantly US-centric benchmark datasets used in this study do not directly validate performance on Australian clinical text, and local validation studies would be required before implementation. The Australian Health Data De-identification Framework (AIHW) provides relevant guidance for local adaptation. Healthcare organisations seeking to de-identify EHR clinical notes for secondary use (research, quality improvement, data sharing). Most directly applicable to institutions using English-language EHR systems with structured clinical documentation similar to US benchmark datasets.

Abstract

OBJECTIVES: The rapid evolution of large language models (LLMs) and their growing application in clinical text processing have created an urgent need for reliable de-identification mechanisms. While LLMs show promise in identifying sensitive health information (SHI), their capabilities require rigorous evaluation. This study aims to conduct a comprehensive benchmarking analysis of various LLM-based, traditional rule-based and hybrid de-identification methods. METHODS: Our benchmark analysis used five datasets (i2b2-2006, MIMIC-2008, i2b2-2014, i2b2-2016 and OpenDeID v1) from different countries. We developed three baseline and eight LLM-based models. The experimental setup encompassed nine different settings using various combinations of training and testing sets to assess model robustness and cross-dataset performance. RESULTS: In the baseline models, the approach trained on the combined corpus of all five datasets (setting 3) significantly outperformed the other settings, achieving a strict F1 micro-average score of 0.8172. Regarding LLM-based models, the supervised fine-tuning approach using the same combined configuration (setting 9) achieved the highest performance with a strict F1 score of 0.9447. DISCUSSION: The harmonisation of corpora ensured standardised data formatting and SHI management across five diverse datasets, highlighting the necessity for uniform categorisation to enhance the reliability of de-identification results. CONCLUSIONS: Our findings indicate that while fine-tuned LLMs offer superior accuracy, the observed performance variability across heterogeneous electronic health record sources poses significant technical challenges. Real-world implementation must address these inconsistencies to overcome the ethical and technical hurdles associated with deploying LLMs for handling sensitive health data.

References

1. 1.Panchal, O., Chang, N.-W., Zhao, Z.-R., Nadar, D. R., Dai, H.-J., & Jonnagaddala, J. (2026). Benchmarking large language models for de-identification of electronic health record notes. BMJ Health & Care Informatics. https://doi.org/10.1136/bmjhci-2025-101894