Large Language Model-Generated Patient Instructions for Prescriptions in Primary Health Care: Preclinical Algorithm Validation

Clinical Snapshot

Key Findings

• P Value: P=.003 (Kruskal-Wallis test)

• Effect Size: ChatGPT-4.0 median 88.4 vs Llama3.1-8B median 66.5 vs Llama3.1-8B-RAG median 79.9

• Primary Outcome: Overall instruction quality scores across three LLM models

• Nnt Or Sensitivity: Error rates: 22.7-22.8% across models; Hallucination rates: 9.1-13.6%

• Confidence Interval: IQR ranges: 22.8, 50.9, and 34.4 respectively

Clinical Application

Requires integration into electronic prescribing systems with mandatory clinician oversight Potential integration with Australian Digital Health platforms; requires TGA consideration for clinical decision support tools; alignment with RACGP digital health guidelines needed Primary care patients requiring medication instructions, particularly those with limited health literacy

Abstract

BACKGROUND: The application of generative artificial intelligence to simplify medication use instructions has the potential to enhance people's health by improving treatment adherence. OBJECTIVE: We evaluated the performance of large language models (LLMs) in generating medication usage instructions to complement prescriptions in primary health care. METHODS: This randomized, blinded experimental preclinical study used prescription-inducing scenarios, assigned to 62 health care professionals, to validate instructions generated by LLMs during electronic prescriptions. The instructions were generated by ChatGPT-4.0 (OpenAI), Llama3.1-8B (Meta), and Llama3.1-8B-RAG (Meta) using retrieval-augmented generation based on patient information leaflets. Performance metrics assessed adequacy, completeness, clarity, language simplification, usefulness, and errors in the generated instructions, with scores to analyze overall and individual metrics. RESULTS: The 3 models yielded high overall scores for producing qualified instructions (ChatGPT-4.0: median 88.4, IQR 22.8; Llama3.1-8B: median 66.5, IQR 50.9; Llama3.1-8B-RAG: median 79.9, IQR 34.4; Kruskal-Wallis test P=.003). Llama3.1-8B-RAG received evaluations with similar overall scores to ChatGPT-4.0 (post hoc test, P=.05) and similar to Llama3.1-8B (post hoc test, P=.44). ChatGPT-4.0 outperformed Llama3.1-8B (Bonferroni test, P<.001). Regarding specific domains, Llama3.1-8B-RAG received scores equivalent to those of ChatGPT-4.0 for adequacy (mean 6.24, SD 2.3 vs mean 6.82, SD 2.1; post hoc test, P=.54); completeness (mean 5.94, SD 2.2 vs 6.55, SD 1.9; post hoc test P=.38), clarity (mean 5.77, SD 2.4 vs mean 6.68, SD 1.9; post hoc test P=.09), and usefulness (mean 5.42, SD 2.4 vs mean 5.96, SD 2.2; post hoc test P=.63). ChatGPT-4.0 received higher scores in the language simplification criterion than Llama3.1-8B-RAG (mean 7.05, SD 1.5 vs mean 5.44, SD 2.6; post hoc test P<.001). Interrater variability in assigning scores ranged from 4.2% (n=3) to 85.8% (n=6) among primary health care professionals. Instructions leading to incorrect use of the medication had similar frequency among the models(ChatGPT-4.0: n=15, 22.7%; Llama3.1-8B: n=19, 22.8%; Llama3.1-8B-RAG: n=19, 22.8%; chi-square test P=.71). The frequencies of hallucination were similar (ChatGPT-4.0: n=7, 10.6%; Llama3.1-8B: n=9, 13.6%; Llama3.1-8B-RAG: n=6, 9.1%; chi-square test P=.67). CONCLUSIONS: The open-source LLM enhanced with external information presented similar performance to the closed-source model, except for ChatGPT4.0, which was superior in language simplification of messages. LLM generation demonstrated potential for instructing patients on medication use. Nonetheless, the introduction of this innovation into the electronic prescribing workflow demands prescriber validation for human oversight of the technology and requires a strategy for LLM performance governance. INTERNATIONAL REGISTERED REPORT IDENTIFIER (IRRID): RR2-10.12688/verixiv.1359.1.

References

1. 1.Silveira Nogueira Reis, Z., Tuler Albergaria, E., Silvina Pagano, A., Martins Lage, E., Ribeiro de Oliveira, F., Dos Santos Dias, C., Almeida Oliveira, J., Miranda Varella Pereira, G., Jose Ramos de Oliveira, I., Franco Mineiro, É., Carvalho Lima Oliveira, I., Dos Reis de Jesus, D., Pereira de Souza Júnior, A., de Carvalho Gomes, I., André Cuevas Gaete, R., Cruz-Correia, R., & Rocha, L. (2026). Large Language Model-Generated Patient Instructions for Prescriptions in Primary Health Care: Preclinical Algorithm Validation. Journal of Medical Internet Research. https://doi.org/10.2196/84444