Implicit bias in safety-aligned large language models: A multi-faceted evaluation of clinical decision-making and health equity

Clinical Snapshot

Key Findings

• P Value: p < 0.001 for correlation between implicit associations and discriminatory decisions

• Effect Size: Race bias (Mean = 0.61), Socioeconomic Status bias (Mean = 0.56)

• Primary Outcome: All 10 models exhibited systematic implicit biases across all six categories

• Nnt Or Sensitivity: 100% of models showed measurable bias (Mean IAT Bias > 0)

• Confidence Interval: Not reported in abstract

Clinical Application

Highly feasible - requires awareness and critical evaluation skills rather than new technology Directly relevant to Australian healthcare given TGA consideration of AI medical devices and RACGP digital health guidelines. Supports need for cultural safety frameworks in AI-assisted care Healthcare professionals using LLMs for clinical decision support, particularly in diverse patient populations

Abstract

BACKGROUND: Large language models are increasingly integrated into healthcare for clinical decision support and patient communication. Although these models can pass explicit social bias tests, they may retain implicit biases-latent associations between social groups and attributes-that could influence medical judgment. OBJECTIVE: To systematically evaluate the presence, magnitude, and behavioral impact of implicit biases in large language models within the medical domain across six high-stakes categories: gender, race, socioeconomic status, health conditions, religion, and healthcare systems. DESIGN: A descriptive cross-sectional study using a multi-faceted evaluation framework. SETTING(S): Computational analysis of 10 mainstream global large language models, including proprietary models (ChatGPT-4o, Gemini-2.0-Flash) and open-source models (DeepSeek-V3, Qwen3). METHODS: We constructed 24 medical bias datasets across six categories. Bias was assessed using three methods: (1) the Large Language Model Word Association Test, a prompt-based method for revealing implicit biases; (2) the Large Language Model Relative Decision Test, a strategy for detecting subtle discrimination in situational decision-making; (3) Paired-Prompt Analysis, used to examine whether implicit associations predict discriminatory decisions. RESULTS: All 10 models exhibited systematic implicit biases (Mean IAT Bias > 0) across all categories, with the strongest biases observed in Race (Mean = 0.61) and Socioeconomic Status (Mean = 0.56). Advanced reasoning capabilities (Chain-of-Thought) did not significantly reduce bias magnitude. Crucially, stronger implicit associations significantly predicted discriminatory choices in downstream medical decision tasks (p < 0.001). CONCLUSION: Current safety alignment techniques fail to eliminate implicit biases in large language models within the medical domain. These latent associations translate into biased decision-making, posing risks for health equity. Future development must prioritize representational debiasing over superficial alignment. Furthermore, healthcare professionals must embrace a stance of "AI vigilance": they should critically evaluate algorithmic outputs as fallible "second opinions" rather than objective truths, thereby ensuring that human judgment remains the ultimate safeguard for equitable patient care.

References

1. 1.Jia, Q., Wen, Y., Liu, Y., Zhao, H., Yu, Q., Long, Y., Sun, D., & Yu, Y. (2026). Implicit bias in safety-aligned large language models: A multi-faceted evaluation of clinical decision-making and health equity. PLoS One. https://doi.org/10.1371/journal.pone.0348819