Clinical Plausibility in Large Language Model Robustness Testing for Medicine: A Scoping Review

Clinical Snapshot

Key Findings

• Effect Size: Not applicable (descriptive scoping review)

• Primary Outcome: Only 33% of studies designed robustness tests that clearly mimic plausible clinical scenarios

• Nnt Or Sensitivity: 49% of studies used misleading prompts, 39% used adversarial prompts as primary testing methods

Clinical Application

Findings suggest current robustness testing may inadequately prepare LLMs for real-world clinical deployment Relevant for TGA medical device regulation of AI systems and RACGP guidelines on AI in general practice. Supports need for clinically-grounded validation before PBS or clinical integration Healthcare institutions implementing or evaluating large language models for clinical applications

Abstract

Large language models (LLMs) show promise in medical applications, yet their translation into clinical practice requires rigorous validation. Current robustness testing often employs adversarial approaches borrowed from AI safety, raising questions about their alignment with authentic clinical scenarios. To systematically map methodologies used for robustness testing of LLMs in medical contexts and assess their clinical plausibility. A scoping review was conducted following PRISMA-ScR guidelines, searching PubMed, Embase, Web of Science, IEEE Xplore, ACM Digital Library, arXiv, and MedRxiv from January 2023 to September 2025. Two independent physician reviewers screened 5,331 articles, extracting data on testing methodologies, medical domains, expert involvement, and clinical plausibility. Thirty-three studies met inclusion criteria, predominantly from 2025 (82%). The most common robustness testing approaches were misleading prompts (49%) and adversarial prompts (39%). Only 33% of studies designed tests clearly mimicking plausible clinical scenarios. While 58% reported expert involvement, the depth of integration varied considerably. Studies predominantly addressed mixed medical domains (73%) rather than specialized fields. The emerging literature suggests that LLM robustness testing in medicine often emphasizes technical vulnerability detection, with fewer studies examining clinically plausible scenarios of routine use. Future frameworks should complement adversarial testing with clinically grounded, longitudinal, and specialty focused evaluations to support deployment-relevant inference.

References

1. 1.Chang, Y., Hsieh, M. H., Ju, P. C., Liu, Y. C., & Chang, C. C. (2026). Clinical Plausibility in Large Language Model Robustness Testing for Medicine: A Scoping Review. Journal of Medical Systems. https://doi.org/10.56147/aaiet.1.1.3