When to stop reviewing: validation of stop criteria in ASReview.

Clinical Snapshot

Key Findings

• P Value: Not applicable - descriptive simulation

• Effect Size: Stop criterion 1 reliable in only 2% of simulations; criterion 2 showed 2-61% threshold variability

• Primary Outcome: None of the three stop criteria could consistently identify all relevant studies across datasets

• Nnt Or Sensitivity: Dataset screening requirements ranged from 2.9% to 76.9% to find all relevant studies

• Confidence Interval: Not reported - simulation study

Clinical Application

Highly feasible - addresses practical workflow decisions in systematic reviews Relevant to NHMRC systematic review guidelines, Cochrane Australia, and evidence synthesis for clinical practice guidelines Systematic review teams, evidence synthesis researchers, guideline developers

Abstract

BACKGROUND: Systematic reviews are essential for evidence-based research, but they often require a great deal of time and effort. Although title and abstract (T&A) screening is just one part of the review process, it can be very time-consuming when search strategies retrieve large numbers of records. Given the exponential growth of scientific publications in recent decades, tools such as ASReview, which use machine learning (ML) for active learning-based screening, aim to reduce the workload. However, since ASReview helps users prioritise potentially relevant studies rather than supporting them in screening all records, a key question arises: at what point can the screening process safely stop without risking the omission of relevant studies when the entire dataset is not being reviewed? METHODS: This simulation study tested three proposed stop criteria for terminating screening in ASReview without loss of relevant data: (1) stopping after a calculated number of relevant studies based on an initial sample; (2) stopping after a fixed number of consecutively studies deemed irrelevant; (3) stopping after a predefined percentage of the dataset has been screened. A total of 35,000 automated title and abstract screenings were conducted using five datasets from the SYNERGY repository. Key outcomes included the percentage of studies screened until the last relevant study was found and the number of relevant studies missed under each stop criterion. RESULTS: The proportion of the dataset that needed to be screened to identify all relevant studies (as pre-classified in the SYNERGY dataset) varied greatly across datasets, ranging from 2.9% to 76.9% on average. None of the tested stop criteria could consistently identify all relevant studies across all datasets. Stop criterion 1 was reliable in only 2% of simulations. Stop criterion 2 showed high variability, with thresholds ranging from 2% to 61%, depending on the dataset. Stop criterion 3 failed to define a universal percentage applicable across datasets. CONCLUSIONS: ASReview can reduce screening workload by prioritizing potentially relevant studies through ML-based ranking, thereby allowing researchers to identify relevant studies earlier in the screening process. However, no stop criterion reliably ensures that all relevant studies are identified. Early stopping may result in missed studies, depending on dataset characteristics. Current stop criteria should be applied cautiously and potentially combined with quality assurance measures. Further research is needed to develop more robust and generalizable stopping rules. TRIAL REGISTRATION: Not applicable - this is a simulation study, not a registered systematic review.

References

1. 1.Kempny, C., Annac, K., Wahidie, D., Yilmaz-Aslan, Y., & Brzoska, P. (2024). When to stop reviewing: validation of stop criteria in ASReview. BMC Medical Research Methodology, 24, Article 699. https://doi.org/10.1186/s13643-024-02699-7