MFBD: Model-free backdoor defense based on vision-language pre-trained models

Clinical Snapshot

Key Findings

• P Value: Not reported

• Effect Size: Not reported in abstract

• Primary Outcome: Detection of poisoned images via cosine similarity between semantic representations of original and saliency-masked image descriptions; classification accuracy preservation on clean data post-defense

• Nnt Or Sensitivity: Detection sensitivity and specificity not reported; no equivalent metric (e.g., AUROC, F1-score for poisoned image detection) provided in the abstract

• Confidence Interval: Not reported

Clinical Application

The method requires deployment of three large pre-trained models (BLIP-2, CLIP, SBERT) as a preprocessing pipeline, which introduces significant computational overhead. Feasibility in resource-constrained clinical AI deployment environments (e.g., point-of-care devices, hospital edge computing) is uncertain. Cloud-based inference pipelines in larger health systems may be more amenable. In the Australian context, the TGA's emerging framework for Software as a Medical Device (SaMD) and AI-enabled medical devices does not yet mandate specific adversarial robustness testing, though the TGA's 2023 discussion paper on AI/ML-based SaMD acknowledges data integrity and model security as emerging concerns. The RACGP and relevant specialist colleges have not issued guidance on backdoor attack mitigation for clinical AI tools. As Australian health systems increasingly adopt AI-assisted diagnostic tools — including those procured through the MBS-linked digital health ecosystem — the security of training pipelines becomes a legitimate governance concern. This research, if validated, could inform procurement and security audit frameworks for clinical AI vendors operating under TGA SaMD regulations. This paper has no direct clinical population. Indirect relevance exists for healthcare AI systems — including medical image classification models (radiology, pathology, dermatology AI) — that may be vulnerable to backdoor poisoning attacks in training pipelines or federated learning environments.

Abstract

Backdoor attacks pose serious security threats to deep neural networks (DNNs), causing networks to output predetermined target labels when specific trigger patterns are embedded in input images. While existing backdoor defense methods utilize additional clean data or victim network models, their effectiveness in defending against various trigger injection methods is limited. We find that vision-language models, such as BLIP-2 and CLIP, are largely unaffected by backdoor triggers in poisoned images. Based on this observation, we propose MFBD, an effective model-free backdoor defense method leveraging the semantic consistency of dual vision-language descriptors. Specifically, we employ BLIP-2 and CLIP to generate descriptions of an image and its saliency masked image, respectively. We then embed both the descriptions and the image labels into a semantic space using SBERT. To distinguish clean and poisoned images, we compute the cosine similarity between their semantic representations. The image exhibiting a low cosine similarity value is classified as a poisoned image. We evaluate a suiute of backdoor defense methods against various attack methods on three datasets. Experimental results demonstrate that our method can effectively mitigate various complex backdoor threats while maintaining high performance across different datasets.

References

1. 1.Huang, R., Hao, M., Wang, H., Xing, Y., & Zhang, Y. (2026). MFBD: Model-free backdoor defense based on vision-language pre-trained models. Neural Networks, 108772. https://doi.org/10.1016/j.neunet.2026.108772