Personal Health Large Language Models and the Negotiation of Medical Authority in Clinical Care: Opportunities, Risks, and Governance

Clinical Snapshot

Key Findings

• Effect Size: Not applicable — no empirical data presented

• Primary Outcome: Conceptual proposal of a triadic model of medical authority (clinician-patient-AI) and a 3-layer governance framework for managing patient-brought PH-LLM narratives in clinical encounters

• Nnt Or Sensitivity: Not applicable — governance/conceptual framework paper; no diagnostic, therapeutic, or prognostic metrics reported

• Confidence Interval: Not applicable — no quantitative analysis performed

Clinical Application

The proposed 3-layer framework is conceptually coherent but operationally untested. Risk-stratified intake, proportionate documentation, and escalation triggers add workflow complexity that may be difficult to implement in high-volume primary care or emergency settings without dedicated resources. The authors acknowledge documentation burden as an unresolved concern. Australia presents a distinctive context for PH-LLM governance. The TGA's Software as a Medical Device (SaMD) framework and the 2023 AI in Health Care guidance are directly relevant but do not yet specifically address patient-facing consumer AI tools that operate outside clinical governance. The My Health Record system creates a unique data infrastructure that PH-LLMs could theoretically interface with, raising specific privacy obligations under the My Health Records Act 2012 and the Privacy Act 1988. The RACGP's Standards for General Practices and its digital health position statements provide a starting point for integrating the proposed framework into Australian primary care. Equity concerns are particularly salient given disparities in digital literacy and device access among Aboriginal and Torres Strait Islander communities, rural populations, and culturally and linguistically diverse patients. The PBS does not currently fund PH-LLM tools, and premium consumer AI health subscriptions risk creating a two-tier system of health information access. Australian health services should monitor the TGA's evolving SaMD classification guidance and consider proactive institutional policies ahead of regulatory mandates. Clinicians in any specialty who encounter patients presenting with AI-generated health narratives, symptom summaries, or self-management recommendations derived from consumer-facing PH-LLMs; particularly relevant for general practitioners, chronic disease specialists, and digital health leads

Abstract

Personal health large language models (PH-LLMs) are patient-facing conversational systems that synthesize user-entered information, patient-generated health data, wearable data, and selected personal health records-where users choose to connect them-into personalized, longitudinal, action-oriented health narratives. Unlike generic health chatbots that mainly provide one-off responses to isolated questions, PH-LLMs may generate continuing interpretations, priorities, and candidate next steps that patients bring into clinical encounters. In contrast to electronic health record-tethered clinical artificial intelligence (AI), they often originate outside institutional oversight and may be selected, used, or trusted by patients before professional review. This viewpoint examines how PH-LLMs may reshape the negotiation of medical authority by contributing to a shift from the traditional dyadic clinician-patient relationship toward a triadic model of negotiated authority, in which clinicians may increasingly need to mediate among clinical evidence, patient values, and algorithmic narratives. PH-LLMs may support patient participation by organizing symptoms, contextualizing home-monitoring and wearable data, improving health literacy, assisting chronic disease self-management, and preparing patients for more collaborative visits. Patient-facing AI narratives may also introduce distinct risks. At the individual level, these include inaccurate or incomplete responses arising from imprecise queries or missing context, misinterpretation of otherwise accurate information in the absence of clinical context, and contextually biased or poorly matched advice across demographic, cultural, linguistic, disability-related, or socioeconomic contexts. At the system level, they include authority conflict when AI recommendations diverge from clinical judgment, fragmentation of clinical truth, privacy and data-governance concerns, diffusion of accountability when harm results from advice produced outside clinical governance, and inequitable access to premium tools and continuous monitoring devices. To address these challenges, we propose a 3-layer clinical governance framework for patient-brought PH-LLM narratives. The first layer, evidence and provenance, makes AI-generated narratives epistemically legible by clarifying platform identity, data sources, temporal anchoring, uncertainty, and privacy-relevant data-use and retention conditions. The second layer, clinical arbitration and workflow integration, uses risk-stratified intake, proportionate documentation, escalation triggers, and equity-preserving workflows to embed PH-LLM outputs into routine care. The third layer, competence and accountability, defines the communication competencies, AI literacy supports, institutional responsibilities, vendor accountability, and risk-proportionate verification duties needed for triadic care. This framework is a conceptual and governance-oriented proposal rather than a validated clinical protocol. Future empirical work should evaluate its feasibility, documentation burden, equity effects, clinical safety impact, and acceptability among patients, clinicians, and health systems. Governed through these interdependent layers, PH-LLMs may serve as supporting infrastructure for safer, person-centered longitudinal care.

References

1. 1.Xie, W., Liu, J., & Liu, S. (2026). Personal health large language models and the negotiation of medical authority in clinical care: Opportunities, risks, and governance. Journal of Medical Internet Research. https://doi.org/10.2196/91727