De-Identification of Free-Text Medical Records Using Large Language Models
Clinical Snapshot
PICO Framework
| P — Population | 150 synthetically generated doctor's letters containing personal health information (PHI) |
| I — Intervention | Large language model-based de-identification (LLaMA-3.1-8B full and quantized variants) |
| C — Comparator | Human annotation performance and different prompting strategies (zero-shot vs few-shot) |
| O — Outcomes | De-identification accuracy (macro F1 score, recall), processing time, and GDPR compliance |
Bottom Line
This study demonstrates promising accuracy for automated de-identification of medical records using large language models, achieving F1 scores above 0.99. However, the exclusive use of synthetic data significantly limits clinical applicability. While the efficiency gains are substantial (seconds vs 13.4 seconds per PHI for humans), the acknowledged false negative risk for sensitive identifiers like names raises concerns for real-world deployment. The quantized model's ability to run locally addresses privacy concerns, but validation on authentic clinical documents with diverse formatting and complexity is essential before clinical implementation. Australian healthcare organisations should await real-world validation studies before adopting these tools for Privacy Act compliance.
Key Findings
P Value: Not reported
Effect Size: Few-shot prompting: F1 = 0.996; Quantized model: F1 = 0.992
Primary Outcome: De-identification accuracy measured by macro F1 score
Nnt Or Sensitivity: Recall: 0.994 (few-shot), 0.987 (quantized model)
Confidence Interval: Not reported
Clinical Application
High technical feasibility with local deployment capability, but validation on real clinical data needed Relevant for Privacy Act 1988 compliance and NHMRC data sharing requirements; could support Australian Digital Health initiatives while maintaining privacy standards Healthcare institutions requiring de-identification of clinical documents for research or data sharing
Abstract
This study evaluates large language models (LLMs) for automated de-identification of free-text medical records. Using 150 synthetically generated, personal health information (PHI)-enriched doctor's letters (23.2 PHIs per letter), we compared the full LLaMA-3.1-8B model (BF16, GPU) with a quantized variant (Q8, CPU) under zero-shot and few-shot prompting. Few-shot prompting improved performance from a macro F1 of 0.989 to 0.996 (recall = 0.994); the quantized Q8 model achieved a macro F1 of 0.992 (recall = 0.987) while enabling fully local, GPU-free inference. Although LLMs perform at a level comparable to human annotation, false negatives remain a concern for highly sensitive PHI, such as names. While human annotators typically require 13.4 seconds per PHI, the LLMs de-identified complete letters within seconds, demonstrating an efficiency and scalability advantage alongside GDPR-compliant, high-precision de-identification.
Related Research
Journal of medical Internet research
Advancing Human-Centered AI in Clinical Decision Support: Sociocognitive Human-in-the-Loop Study in HIV Care
2 Aug 2026
Neurological research
Federated deep learning model for epilepsy seizure detection using electroencephalogram (EEG) signal
2 Aug 2026
Neural networks : the official journal of the International Neural Network Society
MFBD: Model-free backdoor defense based on vision-language pre-trained models
2 Aug 2026
This content is for educational purposes for healthcare professionals only and does not constitute clinical advice. Clinical decisions should be based on individual patient assessment, current guidelines, and appropriate specialist consultation. Editorial Standards · Privacy Policy · Terms of Service