Large Language Models for Breast and Cervical Cancers Communication: Mixed Methods Evaluation Study Assessing Linguistic Quality, Safety, and Accessibility
Clinical Snapshot
PICO Framework
| P — Population | Large language model (LLM) outputs generated in response to real-world breast and cervical cancer-related questions curated from publicly available medical datasets; no human patient population enrolled |
| I — Intervention | Five general-purpose LLMs (including Llama 3 and Gemma) evaluated for breast and cervical cancer health communication |
| C — Comparator | Three medical-domain LLMs (including MedAlpaca and BioMistral) evaluated under identical conditions |
| O — Outcomes | Primary: linguistic quality (fluency, coherence, accuracy); safety and trustworthiness (toxicity, bias, harm potential); communication accessibility and affectiveness (readability, empathy, clarity). Assessed via expert qualitative ratings and quantitative automated metrics. |
Bottom Line
This mixed-methods study evaluated eight LLMs — five general-purpose and three medical-domain — on their ability to generate safe, accurate, and accessible breast and cervical cancer information. The headline finding is a clinically important trade-off: general-purpose models (Llama 3, Gemma) produce linguistically superior and more empathic outputs but at a reading complexity that may exclude low-health-literacy users, while medical-domain models (MedAlpaca, BioMistral) generate simpler text but with unacceptably higher rates of hallucination, bias, and potentially harmful content. For senior clinicians, the key message is that no currently available LLM is fit-for-purpose as a standalone patient-facing cancer communication tool. The study is methodologically limited by absent inter-rater reliability data, unspecified blinding of expert raters, and evaluation of curated model outputs rather than real patient interactions. Generalisability to Australian clinical settings — particularly for BreastScreen and cervical screening programs serving diverse communities — cannot be assumed. Until hybrid LLM architectures with robust safety guardrails are prospectively validated in real-world patient populations, LLM-generated cancer information should be treated as a supplementary resource requiring mandatory clinical oversight, not an autonomous communication solution.
Key Findings
P Value: Not reported in abstract; Games-Howell post-hoc tests used for pairwise comparisons following significant ANOVA results
Effect Size: Hedges g reported (specific values not available in abstract); Welch ANOVA detected statistically significant between-model differences across metric domains
Primary Outcome: General-purpose LLMs (Llama 3, Gemma) demonstrated superior linguistic quality and communication affectiveness compared to medical-domain LLMs (MedAlpaca, BioMistral), but produced more complex outputs. Medical LLMs generated simpler content but exhibited higher rates of hallucination, bias, and toxicity.
Nnt Or Sensitivity: Not applicable to this evaluation study design; no diagnostic accuracy, therapeutic NNT, or prognostic hazard ratio reported
Confidence Interval: Not reported in abstract
Clinical Application
The practical implication — that no currently available LLM optimally balances linguistic quality, safety, and accessibility for cancer communication — is clinically important. Hybrid modelling strategies are proposed but not yet validated. Deployment of any LLM in patient-facing oncology communication without human clinical oversight would be premature based on this evidence. In Australia, the TGA has not approved any LLM as a therapeutic good or medical device for cancer communication. Cancer Australia and the RACGP have not issued guidelines on LLM use in patient communication. The National Cancer Screening Register (NCSR) and BreastScreen Australia programs rely on standardised, evidence-based communication materials; LLM integration would require TGA regulatory consideration under the Software as a Medical Device (SaMD) framework. The finding that medical LLMs exhibit higher toxicity and hallucination rates is particularly relevant given Australia's culturally and linguistically diverse population and the need for safe, accurate cancer screening communication for Aboriginal and Torres Strait Islander communities. PBS and Medicare implications are not relevant to this evaluation study. This study's findings are most directly applicable to health informaticists, digital health developers, and clinical communication specialists considering deployment of LLM-based tools for patient-facing cancer information. It is not directly applicable to individual patient care decisions.
Abstract
BACKGROUND: Effective communication about breast and cervical cancers remains a public health challenge, with widespread misinformation and barriers to cancer-related language understanding. Large language models (LLMs) offer potential for scalable health communication, yet trade-offs between quality, safety, and accessibility of general-purpose and medical-domain LLMs remain underexplored. OBJECTIVE: This study aimed to propose a comprehensive evaluation framework and systematically assess the performance of LLMs in generating breast and cervical cancer information, with a focus on linguistic quality, safety and trustworthiness, and communication accessibility and affectiveness. METHODS: This mixed methods evaluation study assessed outputs from 5 general-purpose and 3 medical LLMs using real-world breast and cervical cancer-related questions curated from publicly available medical datasets. LLM-generated responses were evaluated in a controlled offline setting. Primary outcomes included linguistic quality (fluency, coherence, and accuracy), safety and trustworthiness (toxicity, bias, and harm potential), and communication accessibility and affectiveness (readability, empathy, and clarity). Qualitative ratings were performed by domain experts, while quantitative metrics were compared across models. Statistical analyses included Welch ANOVA to detect differences in metric scores, Games-Howell tests for pairwise comparisons, and Hedges g to assess effect sizes. RESULTS: General-purpose LLMs, particularly Llama 3 and Gemma, demonstrated superior linguistic quality and affectiveness but often produced complex outputs that may limit accessibility. In contrast, medical LLMs (eg, MedAlpaca and BioMistral) generated simpler content suitable for broader audiences but scored lower in safety and empathy due to higher levels of hallucination, bias, and toxicity. CONCLUSIONS: While LLMs show promise for improving digital cancer communication, our findings reveal a trade-off between domain specialization and overall communication quality and safety. Future development of health-focused LLMs should prioritize hybrid modeling strategies to enhance trust, clarity, and clinical relevance in patient-facing tools.
References
- 1.Saha, A., Churchill, V., Rodriguez, A. D., Kursuncu, U., & Idris, M. Y. (2026). Large language models for breast and cervical cancers communication: Mixed methods evaluation study assessing linguistic quality, safety, and accessibility. JMIR Cancer. https://doi.org/10.2196/64406
Related Research
The American journal of pathology
An Integrated Machine Learning and Genomic Framework for Precise Detection of Gastric Cancer
3 Aug 2026
Current opinion in chemical biology
Prediction of protein-protein interactions and co-complex models with deep learning
3 Aug 2026
European journal of radiology
A CT-based deep learning model to differentiate between benign and malignant adrenal lesions
3 Aug 2026
This content is for educational purposes for healthcare professionals only and does not constitute clinical advice. Clinical decisions should be based on individual patient assessment, current guidelines, and appropriate specialist consultation. Editorial Standards · Privacy Policy · Terms of Service