Research Appraisals
Evidence-based critical appraisals of the latest medical research, systematically evaluated using Oxford CEBM methodology.
Showing 2 appraisals
BMC medical research methodology
When to stop reviewing: validation of stop criteria in ASReview.
BACKGROUND: Systematic reviews are essential for evidence-based research, but they often require a great deal of time and effort. Although title and abstract (T&A) screening is just one part of the review process, it can be very time-consuming when search strategies retrieve large numbers of records. Given the exponential growth of scientific publications in recent decades, tools such as ASReview, which use machine learning (ML) for active learning-based screening, aim to reduce the workload. However, since ASReview helps users prioritise potentially relevant studies rather than supporting them in screening all records, a key question arises: at what point can the screening process safely stop without risking the omission of relevant studies when the entire dataset is not being reviewed? METHODS: This simulation study tested three proposed stop criteria for terminating screening in ASReview without loss of relevant data: (1) stopping after a calculated number of relevant studies based on an initial sample; (2) stopping after a fixed number of consecutively studies deemed irrelevant; (3) stopping after a predefined percentage of the dataset has been screened. A total of 35,000 automated title and abstract screenings were conducted using five datasets from the SYNERGY repository. Key outcomes included the percentage of studies screened until the last relevant study was found and the number of relevant studies missed under each stop criterion. RESULTS: The proportion of the dataset that needed to be screened to identify all relevant studies (as pre-classified in the SYNERGY dataset) varied greatly across datasets, ranging from 2.9% to 76.9% on average. None of the tested stop criteria could consistently identify all relevant studies across all datasets. Stop criterion 1 was reliable in only 2% of simulations. Stop criterion 2 showed high variability, with thresholds ranging from 2% to 61%, depending on the dataset. Stop criterion 3 failed to define a universal percentage applicable across datasets. CONCLUSIONS: ASReview can reduce screening workload by prioritizing potentially relevant studies through ML-based ranking, thereby allowing researchers to identify relevant studies earlier in the screening process. However, no stop criterion reliably ensures that all relevant studies are identified. Early stopping may result in missed studies, depending on dataset characteristics. Current stop criteria should be applied cautiously and potentially combined with quality assurance measures. Further research is needed to develop more robust and generalizable stopping rules. TRIAL REGISTRATION: Not applicable - this is a simulation study, not a registered systematic review.
13 May 2026
Read appraisal →International journal of epidemiology
Structured tools for assessing quality and risk of bias in Mendelian randomization studies: an updated systematic review
BACKGROUND: The growing use of Mendelian randomization (MR) has heightened the need for rigorous quality and bias assessment tools. A previous systematic review included studies published up to July 2021 identified 14 structured instruments for conducting, evaluating, and reporting MR studies. However, methodological developments have accelerated in the years since. METHODS: We updated the previous systematic review to include tools published between July 2021 and January 2025, applying the same search strategy and eligibility criteria. Two reviewers independently screened articles, extracted data, and mapped tool content to bias domains. RESULTS: We identified 15 additional articles, bringing the total to 29 tools. Of these 29, 19 provided structured evaluation tools. Twelve of the 19 evaluation tools were newly added in the present review, which addressed broader methodological domains beyond core instrumental variable assumptions, including genetic instrument selection, population stratification, sensitivity analyses, and dataset considerations. However, substantial variation in bias domains, structure, and scoring methods across tools persists. Key gaps remain in the assessment of linkage disequilibrium, missing data, and dynastic effects. CONCLUSIONS: While the number of structured tools has increased in recent years, the lack of standardization across tools makes it difficult to compare across systematic reviews of MR studies. Developing more complete and standardized evaluation frameworks and properly testing these tools in practice are important next steps to improve the overall quality of MR research.
20 Apr 2026
Read appraisal →