Trusting Generative AI for Health Advice: Preregistered Survey Experiment

Clinical Snapshot

Key Findings

• P Value: Not reported — statistical significance implied but not quantified in abstract

• Effect Size: Not reported in abstract — directional effects described qualitatively without standardised effect size metrics (Cohen's d, eta-squared, or partial eta-squared)

• Primary Outcome: Health advice attributed to a human nurse was rated as significantly more credible than advice attributed to either AI source (AI nurse or ChatGPT). Message intuitiveness had effects on perceived credibility comparable to or larger than source type, particularly in high-risk scenarios. Medical skepticism moderated source evaluations such that higher skepticism was associated with greater perceived competence of the AI nurse and lower perceived competence of the human nurse.

• Nnt Or Sensitivity: Not applicable for this perceptual/attitudinal outcome study; no NNT, sensitivity, specificity, or hazard ratio calculable

• Confidence Interval: Not reported

Clinical Application

Findings are immediately applicable to the design of patient-facing AI health interfaces. The observation that message intuitiveness may matter as much as source labelling suggests that how AI health advice is framed — not merely that it is disclosed as AI-generated — is a critical design variable. Clinicians should be aware that medically skeptical patients may be more receptive to AI-sourced guidance, which has implications for therapeutic alliance and shared decision-making. In Australia, the TGA has begun developing regulatory frameworks for AI-enabled software as medical devices (SaMD), and the RACGP has published position statements on AI in general practice. The My Health Record system and expanding telehealth infrastructure create contexts where AI-generated health advice may increasingly reach patients. Australian patients' trust in the universal Medicare system and strong GP-patient relationships may produce different baseline credibility hierarchies than the US sample studied here. The PBS-subsidised medication context and Australia's lower rates of medical malpractice litigation may also shape patient attitudes toward AI health advice differently. RACGP guidance emphasises that AI tools should support, not replace, the therapeutic relationship — consistent with this study's finding that human clinicians retain a credibility advantage. Australian digital health literacy programs (e.g., those supported by the Australian Digital Health Agency) should consider message design principles identified in this research. General adult population engaging with digital health information platforms, particularly those with varying levels of medical skepticism or prior AI experience; most relevant to patient-facing digital health tool designers, health communicators, and clinicians advising patients on AI health information use

Abstract

BACKGROUND: Generative artificial intelligence (AI) systems are increasingly used for health information seeking, yet it remains unclear how the public evaluates AI-generated health advice relative to guidance from credentialed clinicians in digital environments. Understanding the conditions under which AI is perceived as credible is critical as these systems become integrated into digital health ecosystems. OBJECTIVE: This study examined how source type (a human nurse in an online portal, a health care-specialized generative artificial intelligence system framed as an "AI nurse," or ChatGPT [OpenAI], a general-purpose chatbot), message characteristics, contextual risk, values framing, and individual differences in medical skepticism and experience with AI shape credibility evaluations of the provided advice and its purported source. METHODS: In a preregistered online experiment, a national sample of US adult participants (N=1502) was randomly assigned to 1 of 3 source conditions and evaluated health advice across 3 scenarios: low risk (dietary advice for cholesterol), high risk (chest pain triage), and a morally sensitive scenario (egg freezing). Advice type (intuitive vs counterintuitive) was manipulated in the risk scenarios, and ideological framing (neutral, conservative-leaning, and liberal-leaning) was manipulated in the morally sensitive scenario. Primary outcomes included participants' perceived credibility of the advice and beliefs about whether the patient should follow it. Source-level perceptions of competence and benevolence were also assessed. Medical skepticism and prior AI experience were examined as moderators. RESULTS: Advice attributed to a human nurse was rated as more credible than advice attributed to either AI source. Message intuitiveness showed effects comparable to, and sometimes larger than, the effects of source: intuitive advice was perceived as more credible than counterintuitive advice, with this difference amplified in high-risk contexts. In the morally sensitive scenario, ideological framing influenced perceived bias but did not interact significantly with source. Medical skepticism moderated source evaluations: higher skepticism was associated with greater perceived competence of the AI nurse and lower perceived competence of the human nurse. CONCLUSIONS: Generative AI is evaluated within existing credibility frameworks rather than dismissed outright as inferior to human expertise. While licensed clinicians retain a credibility advantage, AI-generated advice is generally perceived as competent and legitimate. Importantly, individuals skeptical of traditional medical authority may evaluate AI-based guidance more favorably, suggesting that AI systems may redistribute-rather than uniformly erode-trust in health advice. As AI tools become embedded in patient-facing health platforms, message design and audience characteristics may shape acceptance more strongly than source labeling alone.

References

1. 1.Landrum, A. R., Verma, N., & Kehrberg, A. (2026). Trusting generative AI for health advice: Preregistered survey experiment. Journal of Medical Internet Research. PubMed ID: 42330316. https://pubmed.ncbi.nlm.nih.gov/42330316