The Alberta Quality Assessment Tool: Risk of Bias (AQAT:RoB) for the Evaluation of Medical Large Language Model Question-Answer Studies: Development and Pilot Validation

Clinical Snapshot

Key Findings

• P Value: Not reported

• Effect Size: Cohen's κ = 0.70 (substantial agreement)

• Primary Outcome: Inter-rater reliability of AQAT:RoB tool

• Nnt Or Sensitivity: 86.1% agreement between independent evaluators

• Confidence Interval: Not reported

Clinical Application

High - structured checklist format suitable for systematic reviews and quality assessments Highly relevant for TGA evaluation of medical AI tools, NHMRC research assessment, and clinical guideline development incorporating LLM evidence Researchers, clinicians, and regulatory bodies evaluating medical LLM question-answer studies

Abstract

BACKGROUND: Despite the transformative potential of large language models (LLMs) in health care, the rapid development of these tools has outpaced their rigorous evaluation. While artificial intelligence-specific reporting guidelines have been developed to address standardized reporting of artificial intelligence studies, there is currently no specific tool available for risk of bias assessment of LLM question-answer (QA) studies. Existing risk-of-bias tools for medical research are not well suited to the unique challenges of evaluating LLM-QA studies, which creates a critical gap in assessing their safety and effectiveness. OBJECTIVE: This study aims to develop the Alberta Quality Assessment Tool: Risk of Bias (AQAT:RoB) for LLM-QA studies to systematically evaluate the validity and risk of bias in LLM-QA studies. METHODS: We conducted 2 literature reviews. The first was on quality assessment tools for LLM-QA studies, and the second was on LLM-QA studies, which informed the first draft of the AQAT:RoB. The draft AQAT:ROB was further refined through a prespecified iterative process of modified Delphi, consensus meeting, and validation. The first Delphi process occurred between May 1 and May 20, 2025, and the first consensus meeting was held on May 22. The first round of validation was completed by 4 evaluators, who were not part of the consensus meeting, on 16 randomly selected studies. As this first round of validation surpassed our a priori threshold of ≥80% agreement and a Cohen κ of ≥0.61 between evaluators, no further rounds of development and validation were undertaken. A second Delphi process occurred between February 20 and February 23, 2026, to vote on postpilot changes in response to peer review. RESULTS: The AQAT:RoB consists of 5 high-level domains (Questions, Reference Answers, LLM Answers, Evaluators, Outcomes). These domains are subdivided into 9 subdomains. Each subdomain includes at least one "Support for Judgment" and at least one "Type of Bias" and is to be rated "low," "high," or "unclear" for risk of bias. A pilot evaluation was completed by internal validators who were not part of the consensus discussion and were asked to complete the AQAT:RoB form for each assigned study. Each of the 16 studies was evaluated by 2 evaluators independently. Pilot validation showed a percent agreement of 86.1% and a Cohen κ of 0.70 between assessors. CONCLUSIONS: The AQAT:RoB demonstrates promising initial reliability for assessing the validity or risk of bias in LLM-QA studies. The tool will benefit from future refinements, external validation, and periodic updates to keep pace with evolving technology.

References

1. 1.Ye, C., Mitchell, J. R., Baumgart, D. C., Ma, Z., Fung, A. L., Orellana, D. G., Chowdhury, J., Abass, A., Katz, S., Jaremko, J. L., Boulanger, P., Barber, C. E. H., Lemermeyer, G., Jabbari, H., Mou, L., Mirzaei, M., Githumbi, M. W. B., Tandon, P., Goebel, R., Clark, R., Hung, W., Abbasi, M., Maleki, F., Klarenbach, S., & Abdalla, M. (2026). The Alberta Quality Assessment Tool: Risk of Bias (AQAT:RoB) for the Evaluation of Medical Large Language Model Question-Answer Studies: Development and Pilot Validation. Journal of Medical Internet Research. https://doi.org/10.1016/j.jclinepi.2024.111370