Performance of Large Language Models in Answering Osteoporosis-Related Frequently Asked Questions: A Systematic Comparative Evaluation Based on International Associations
Clinical Snapshot
PICO Framework
| P — Population | Osteoporosis-related frequently asked questions from three international osteoporosis associations |
| I — Intervention | Six different large language models generating responses to standardized questions |
| C — Comparator | Performance comparison between different LLM models |
| O — Outcomes | Accuracy and comprehensiveness of responses measured on 5-point Likert scales by two independent osteoporosis experts |
Bottom Line
This comparative evaluation demonstrates that current large language models, particularly Claude Sonnet 4.5, show promising potential as auxiliary tools for osteoporosis patient education and clinical support. The study employed rigorous methodology with good inter-rater reliability and found significant performance differences between models. However, substantial variability across question types suggests these tools require careful clinical oversight and cannot replace professional medical judgment. While LLMs demonstrated solid medical knowledge, the lack of real-world clinical validation and potential for generating incorrect medical advice necessitates cautious implementation. For Australian clinicians, these findings suggest LLMs could support patient education initiatives and clinical decision-making when used as adjunctive tools, but regulatory frameworks and professional guidelines for their clinical application remain essential. The technology shows promise for improving accessibility of osteoporosis information, particularly valuable given Australia's aging population and high osteoporotic fracture burden.
Key Findings
P Value: Significant differences found using Kruskal-Wallis tests
Effect Size: Significant differences observed among models (specific values not provided in abstract)
Primary Outcome: Claude Sonnet 4.5 demonstrated highest overall performance on osteoporosis FAQs
Nnt Or Sensitivity: Inter-rater reliability: κ = 0.668 (accuracy), κ = 0.702 (comprehensiveness)
Confidence Interval: Not reported in abstract
Clinical Application
High feasibility as LLMs are readily accessible, though requires careful implementation with clinical oversight Relevant to Australian practice given international association sources; could support RACGP osteoporosis guidelines implementation and patient education, though TGA regulatory considerations for medical advice tools would apply Healthcare providers and patients seeking osteoporosis information, particularly relevant for endocrinologists, rheumatologists, and general practitioners
Abstract
To systematically evaluate and compare the performance of large language models (LLMs) in answering osteoporosis-related frequently asked questions (FAQs) derived from international osteoporosis-related associations. A standardized question bank was constructed based on FAQs summarized from three international osteoporosis-related associations. Six LLMs were prompted to generate responses to all questions under uniform conditions. Two osteoporosis experts independently and blindly evaluated all responses using 5-point Likert scales for accuracy and comprehensiveness. Inter-rater reliability was assessed using Cohen's κ coefficient. Nonparametric statistical analyses, including Kruskal-Wallis tests with Dunn's post hoc comparisons and Wilcoxon signed-rank tests, were performed to compare models' performance. A total of 528 responses were generated from 88 association-based questions. Inter-rater agreement was good for both accuracy and comprehensiveness (κ = 0.668 and 0.702, respectively). Significant differences were observed among models, with Claude Sonnet 4.5 demonstrating the highest overall performance on association-based FAQs. Although most models showed solid medical knowledge and basic reasoning ability, substantial variability was observed across question types and prompt formulations. Current LLMs demonstrate promising potential as auxiliary tools for osteoporosis-related health management. However, it is necessary to optimize the strategies for integrating medical knowledge and designing prompts, so as to better improve the accessibility of its clinical application.
References
- 1.Wu, X., Li, J., Liu, S., Li, Y., Wang, Q., Luo, C., Shi, L., Zheng, Y., Meng, L., Fan, M., Sheng, Z., & Ouyang, N. (2026). Performance of Large Language Models in Answering Osteoporosis-Related Frequently Asked Questions: A Systematic Comparative Evaluation Based on International Associations. Calcified Tissue International. https://doi.org/10.1038/s42256-025-01113-8
Related Research
European journal of radiology
A CT-based deep learning model to differentiate between benign and malignant adrenal lesions
3 Aug 2026
Journal of chemical information and modeling
A Unified Molecular Graph and Protein Language Model Framework for Predicting Human Drug-Hormone Receptor Interactions with Structure-Aware Validation
28 July 2026
Neurological sciences : official journal of the Italian Neurological Society and of the Italian Society of Clinical Neurophysiology
Differential effects of hormone therapy formulations on Parkinson's disease risk: a systematic review and meta-analysis
23 July 2026
This content is for educational purposes for healthcare professionals only and does not constitute clinical advice. Clinical decisions should be based on individual patient assessment, current guidelines, and appropriate specialist consultation. Editorial Standards · Privacy Policy · Terms of Service