Research Appraisals
Evidence-based critical appraisals of the latest medical research, systematically evaluated using Oxford CEBM methodology.
Showing 3 appraisals
JMIR research protocols
Effectiveness of Guideline-Based Clinical Decision Support Systems: Protocol for a Systematic Review
BACKGROUND: Clinical guidelines (CGs) standardize care through evidence-based recommendations, while clinical decision support systems (CDSS) can assist in applying these guidelines to individual patients. The scientific basis for the decisions offered by decision support systems is often not explicitly stated or not clearly specified in the literature on CDSS. Therefore, a systematic examination of the literature is needed to map the current state of CDSS, with a particular focus on the integration of CGs. OBJECTIVE: This study aims to systematically collect, describe, and synthesize evidence of randomized controlled studies of interventions using CDSS with a well-defined integration of evidence-based CGs and evaluating direct medical outcomes. METHODS: This systematic review adheres to the PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) and PRISMA-ScR (Preferred Reporting Items for Systematic Reviews and Meta-Analyses extension for Scoping Reviews) checklists. The eligibility criteria for this review are defined using the patient, intervention, control, outcome, and study design framework, including studies involving patients with any medical condition or disease. Study interventions need to include guideline-based CDSS, encompassing all types of interventions used for treatment. Each intervention must provide a sufficiently accessible technical description, including the types of data and algorithms used for decision-support procedures. The guidelines used within these CDSS interventions must be derived from a clearly defined, evidence-based guideline development process published by a discernible guideline-producing body. Studies must use a randomized controlled study design. Only studies evaluating the effectiveness of the interventions on direct medical outcomes are included. Web of Science, including MEDLINE, and Scopus will be searched with search expressions aligned with the eligibility criteria. RESULTS: On August 11, 2022, the initial search was conducted on Web of Science and Scopus. From a total of 6203 records, 1347 were removed prior to screening as duplicates, 2506 records were excluded during the first screening step, and 2291 were excluded during the second step. Next, 41 papers were excluded based on full-text review, and 18 papers were finally included in the review following this initial search. This review explores whether CDSS based on CGs can improve clinical outcomes, although their effectiveness may vary depending on various factors. Potential limitations, such as high study heterogeneity, have already been identified. An update of the review has been started in April 2025. CONCLUSIONS: To our knowledge, this is the first rigorous systematic review on the effectiveness of guideline-based decision support systems in which the technical integration and algorithmic embedding of CGs have been described or can be inferred from secondary literature. With this review, we aim to address this gap by providing a detailed analysis of existing research and identifying best practices, challenges, and areas for future investigation.
14 June 2026
Read appraisal →Research synthesis methods
Transforming evidence synthesis: A systematic review of the evolution of automated meta-analysis in the age of AI
Exponential growth in scientific literature has heightened the demand for efficient evidence-based synthesis, driving the rise of the field of automated meta-analysis (AMA) powered by natural language processing and machine learning. This PRISMA systematic review introduces a structured framework for assessing the current state of AMA, based on screening 13,216 papers (2006-2024) and analyzing 61 studies across diverse domains. Findings reveal a predominant focus on automating data processing (52.5%), such as extraction and statistical modeling, while only 16.4% address advanced synthesis stages. Just one study (approximately 2%) explored preliminary full-process automation, highlighting a critical gap that limits AMA's capacity for comprehensive synthesis. Despite recent breakthroughs in large language models and advanced AI, their integration into statistical modeling and higher-order synthesis, such as heterogeneity assessment and bias evaluation, remains underdeveloped. This has constrained AMA's potential for fully autonomous meta-analysis (MA). From our dataset spanning medical (67.2%) and non-medical (32.8%) applications, we found that AMA has exhibited distinct implementation patterns and varying degrees of effectiveness in actually improving efficiency, scalability, and reproducibility. While automation has enhanced specific meta-analytic tasks, achieving seamless, end-to-end automation remains an open challenge. As AI systems advance in reasoning and contextual understanding, addressing these gaps is now imperative. Future efforts must focus on bridging automation across all MA stages, refining interpretability, and ensuring methodological robustness to fully realize AMA's potential for scalable, domain-agnostic synthesis.
6 May 2026
Read appraisal →Journal of medical Internet research
Beyond GPT-4: The Rapidly Evolving Potential of Large Language Models for Clinical Guideline Improvement
This commentary reviews the study by Jones et al, which evaluated whether GPT-4 could improve the readability of injectable medication guidelines while preserving important safety information. The study found that GPT-4 produced modest readability gains comparable to manual revision, but also introduced omissions and meaning changes in a minority of sections. These findings highlight both the potential and limitations of early large language models (LLMs) in clinical contexts. However, this study reflects the capabilities of a specific model in a rapidly evolving domain. Since the release of GPT-4, advances in multistep reasoning, model-critique workflows, and structured validation have substantially improved the ability of newer systems to detect omissions, maintain factual fidelity, and support controlled editing. As a result, some documented limitations may stem from the constraints of a single-model, single-pass workflow rather than intrinsic flaws in LLM-assisted guideline revision. This commentary highlights the need for evaluation frameworks that can keep pace with LLM progress and emphasizes that clinical oversight and user-centered testing remain essential. Updated research using contemporary models is needed to determine how emerging architectures can more safely support clarity, consistency, and maintenance of clinical guidelines.
12 Apr 2026
Read appraisal →