Using a large language model artificial intelligence agent to improve the efficiency of clinical quality measure evidence evaluation: a case study.
Clinical Snapshot
PICO Framework
| P — Population | Clinical quality measure evidence evaluation process for pneumonia diagnostic performance measures in the Centers for Medicare & Medicaid Services Consensus-Based Entity endorsement process |
| I — Intervention | Large language model-based artificial intelligence agent implemented within a structured Claim-Argument-Evidence System (CAES) |
| C — Comparator | Subject matter expert manual evaluation |
| O — Outcomes | Agreement between CAES and expert assessments, time efficiency, claim status accuracy, evidence quality assessment |
Bottom Line
This pilot study demonstrates the potential for large language model technology to dramatically reduce the time required for clinical quality measure evidence evaluation, from thousands of hours to single digits. However, the 69% agreement rate with expert evaluators and 14% disagreement rate highlight significant limitations in current technology. The study's single-case design and lack of statistical analysis limit confidence in the findings. While promising for efficiency gains, the technology requires substantial further development and validation before implementation in clinical quality improvement processes. Healthcare organisations should view this as early-stage proof-of-concept rather than ready-for-deployment technology. The substantial time savings potential warrants continued research, but current limitations in contextual interpretation and evidence synthesis accuracy must be addressed before clinical application.
Key Findings
P Value: Not reported
Effect Size: 69% agreement rate between CAES and expert evaluations
Primary Outcome: Agreement between CAES and subject matter expert assessments
Nnt Or Sensitivity: Time reduction from estimated 2400 labour hours to approximately 5 hours for evidence evaluation
Confidence Interval: Not reported
Clinical Application
Requires significant technical infrastructure and validation before implementation Limited direct applicability to Australian context; would require adaptation for NHMRC guidelines, ACSQHC standards, and local evidence synthesis processes Healthcare quality improvement organisations, clinical guideline developers, evidence synthesis teams
Abstract
OBJECTIVES: To evaluate the feasibility and performance of a large language model (LLM)-based artificial intelligence (AI) agent, implemented within a structured Claim-Argument-Evidence System (CAES), for supporting the review of clinical quality measure (CQM) evidence in the Centers for Medicare & Medicaid Services Consensus-Based Entity (CBE) endorsement process. METHODS: The CBE conducted a pilot study using a previously endorsed measure. CAES extracted claims and citations from a submitted diagnostic performance measure for pneumonia, automatically retrieved additional relevant evidence from PubMed abstracts and assessed the quality, confidence and agreement of evidence supporting each claim. The system's assessments were compared with the judgement of a subject matter expert (SME). RESULTS: CAES completed the assessment in approximately 5 hours. The SME agreed with the CAES-assigned claim statuses for 69% of claims, was neutral for 11% and disagreed for 14%. Disagreements primarily stemmed from the need for contextual interpretation beyond abstracts. DISCUSSION: Manual evaluation of CQM evidence requires significant time and resources, estimated at over 2400 labour hours per review cycle, limiting efficiency and transparency.The AI agent evaluated 64 claims and 355 claim-evidence pairs related to the pneumonia diagnosis measure. It assigned claim statuses based on evidence strength and generated justifications. CONCLUSION: This pilot demonstrated the feasibility and potential of LLM-based AI agents to improve the efficiency and transparency of evidence review for CQMs. Further development is needed to incorporate additional data sources and extend applicability across the measure development lifecycle.
References
- 1.Geppert, J., Jones, B., Bellay, J., & Bryan, G. (2026). Using a large language model artificial intelligence agent to improve the efficiency of clinical quality measure evidence evaluation: a case study. BMJ Health & Care Informatics. https://doi.org/10.1136/bmjhci-2025-101621
Related Research
Journal of medical Internet research
Advancing Human-Centered AI in Clinical Decision Support: Sociocognitive Human-in-the-Loop Study in HIV Care
2 Aug 2026
Neurological research
Federated deep learning model for epilepsy seizure detection using electroencephalogram (EEG) signal
2 Aug 2026
Neural networks : the official journal of the International Neural Network Society
MFBD: Model-free backdoor defense based on vision-language pre-trained models
2 Aug 2026
This content is for educational purposes for healthcare professionals only and does not constitute clinical advice. Clinical decisions should be based on individual patient assessment, current guidelines, and appropriate specialist consultation. Editorial Standards · Privacy Policy · Terms of Service