Research Appraisals
Evidence-based critical appraisals of the latest medical research, systematically evaluated using Oxford CEBM methodology.
Showing 2 appraisals
Journal of the American Medical Informatics Association : JAMIA
Alert fatigue measurement in clinical decision support: a systematic review
BACKGROUND: Alert fatigue is defined as alert dismissals due to excessive or irrelevant alerts and is frequently cited as a barrier to clinical decision support system use and impact. However, the criteria for determining the presence or absence of alert fatigue are poorly defined. The objective of this systematic review of systematic reviews was to identify operationalized definitions and measures of alert fatigue or alert-related metrics. METHODS: Systematic reviews reporting at least one alert-related metric or measure/operationalization of alert fatigue for physician-directed electronic alerts were included. The Cochrane Library, Embase, and PubMed were searched from database start to 2024. The Revised Assessment of Multiple Systematic Reviews was used to assess study quality and risk of bias. Data were synthesized narratively and with descriptive statistics. RESULTS: A total of 22 studies were included in the review. Studies reported between 1 and 11 alert metrics. Studies were most often of medium quality. Reporting of primary study characteristics was frequently judged to be insufficient. Only one article reported an operational definition of alert fatigue. The most common alert metrics were quantity, override rate, and acceptance rate. DISCUSSION: Alert fatigue measurement methods are not clearly or consistently defined in systematic reviews related to alert fatigue in clinical decision support. Reporting of other primary study characteristics is often limited. We recommend that future efforts use a significant, sustained decrease in appropriate alert response rates from an established baseline as a measure of alert fatigue.
2 Aug 2026
Read appraisal →Joint Commission journal on quality and patient safety
Large Language Models as Physician Recommenders: Limitations, Alternatives, and the Path Toward Hybrid AI Systems
Digital tools are increasingly used by patients to access health information and navigate care, including choosing clinicians, but the evidence supporting "best doctor" recommendations varies widely across available methods. This manuscript compares the emerging use of consumer-facing large language models (LLMs) for this purpose with four alternative physician-selection strategies: patient networks, expert referrals, consumer rating/search platforms, and outcome-informed artificial intelligence / machine learning (AI/ML) recommender systems. Consumer rating platforms mainly reflect patient experiences rather than technical skills and often show weak or inconsistent links with objective performance measures. General-purpose LLMs may produce convincing rationales that highlight visibility and reputation, but they remain limited by hallucinations, prompt sensitivity, and uncertainty about whether their explanations truly reflect the underlying reasoning. Outcome-informed AI/ML recommender systems could, in theory, incorporate risk-adjusted clinical performance, but they face significant challenges, including unreliable physician-level measurement, incomplete data, residual confounding, and limited transparency, which can undermine trust and informed consent. We suggest that a hybrid system should be seen as a guiding concept and a testable approach rather than a final solution. Such a system could integrate proven, auditable performance measures with transparent patient-preference filters and verifiable, patient-facing explanations. We present key design principles, including clinical relevance, explainability, autonomy, equity, and governance, in line with current US Food and Drug Administration (FDA) guidance related to clinical decision support and adaptive AI life cycle management.
1 July 2026
Read appraisal →