Cross-Sectional Evaluation of Medical Disinformation Safeguards in Consumer-Facing Large Language Model Platforms
Clinical Snapshot
PICO Framework
| P — Population | Six consumer-facing large language model platforms (Claude, ChatGPT, Copilot, Meta AI, Grok, and Gemini) |
| I — Intervention | Exposure to prompts designed to elicit health disinformation |
| C — Comparator | Performance comparison across different LLM platforms |
| O — Outcomes | Safeguard effectiveness against generation of health disinformation |
Bottom Line
This cross-sectional study reveals concerning disparities in medical disinformation safeguards across popular consumer LLM platforms. While Claude and ChatGPT demonstrated complete resistance to generating health misinformation, four other major platforms showed substantial vulnerabilities. For Australian clinicians, this highlights the critical need for patient education about reliable health information sources and the risks of seeking medical advice from certain platforms. The findings suggest that not all LLMs are equivalent in their safety measures, which has important implications for patient counselling and digital health literacy initiatives. However, the rapid evolution of these platforms means safeguards may change frequently, requiring ongoing monitoring. Healthcare professionals should be aware of these platform differences when discussing digital health resources with patients and consider recommending more robust platforms for health-related queries.
Key Findings
P Value: Not reported in abstract
Effect Size: Complete resistance (Claude, ChatGPT) vs substantial vulnerabilities (Copilot, Meta AI, Grok, Gemini)
Primary Outcome: Safeguard effectiveness against health disinformation generation
Nnt Or Sensitivity: Binary classification: complete resistance vs substantial vulnerability
Confidence Interval: Not reported in abstract
Clinical Application
Highly feasible - involves commonly used consumer platforms Directly relevant to Australian practice; TGA and ACMA may need to consider regulatory implications for health information accuracy in consumer platforms Healthcare professionals and patients using consumer LLM platforms for health information
Abstract
This cross-sectional evaluation of six consumer-facing large language model platforms found significant heterogeneity in safeguard performance against the generation of health disinformation, with Claude and ChatGPT demonstrating complete resistance across all prompt types, while Copilot, Meta AI, Grok, and Gemini exhibited substantial vulnerabilities.
References
- 1.Modi, N. D., Alex, C. A., Awaty, A. A., Menz, B. D., Bacchi, S., Gradon, K. T., Logan, J. M., Rowland, A., Kalisch Ellett, L., McKinnon, R. A., Wiese, M. D., Sorich, M. J., & Hopkins, A. M. (2026). Cross-Sectional Evaluation of Medical Disinformation Safeguards in Consumer-Facing Large Language Model Platforms. *JMIR Infodemiology*. https://doi.org/10.2196/89831
Related Research
The Journal of rheumatology
Examining the Role of Wearables in Inflammatory Arthritis Care: A Narrative Literature Review
3 Aug 2026
Acta obstetricia et gynecologica Scandinavica
ChatGPT in urogynecology: Comparing large language model responses to human experts
3 Aug 2026
Sleep medicine
Concordance of wearable device sleep metrics with patient-reported sleep quality: A systematic review
3 Aug 2026
This content is for educational purposes for healthcare professionals only and does not constitute clinical advice. Clinical decisions should be based on individual patient assessment, current guidelines, and appropriate specialist consultation. Editorial Standards · Privacy Policy · Terms of Service