Impact of COVID-19-related data drift on machine-learning prognostic models predicting 30-day opioid-related emergency department visits, hospitalisation or mortality: a population-level administrative data study in Alberta, Canada

Clinical Snapshot

Key Findings

• P Value: Not reported

• Effect Size: Event rates: 2.1% (pre-pandemic), 2.4% (pandemic), 2.3% (endemic), 2.8% (2023 validation)

• Primary Outcome: 30-day opioid-related emergency department visits, hospitalisation, or mortality

• Nnt Or Sensitivity: Discrimination (AUC): Pre-pandemic 0.81, Pandemic 0.74, Endemic 0.83 on 2023 validation; 2020-2021 validation AUC 0.71

• Confidence Interval: Not reported

Clinical Application

Requires integration with administrative health data systems and machine learning infrastructure; may be challenging in smaller healthcare systems Highly relevant given Australia's opioid prescribing patterns and PBS data availability; could inform TGA post-market surveillance and RACGP opioid prescribing guidelines Adults receiving opioid prescriptions in primary care, emergency departments, and specialist settings

Abstract

OBJECTIVE: To develop machine-learning (ML) models during the COVID-19 pandemic and adjacent time periods to evaluate the impact of data drift on model performance. DESIGN: This prognostic study used population-level administrative health data to develop ML prediction models. SETTING: Alberta, Canada during 2019-2023. PARTICIPANTS: All patients over 18 who received at least one opioid dispensation from a community pharmacy within the province of Alberta between 2019-2023. EXPOSURE: Each opioid dispensation served as the unit-of-analysis. MAIN OUTCOMES/MEASURES: Opioid-related outcomes were identified from linked health administrative datasets. Light Gradient Boosting-machine models were developed on pre-pandemic, pandemic and endemic data and temporally validated on 2023 data (pre-pandemic model was also validated on 2020-2021 data) to predict the risk of emergency department visit, hospitalisation or mortality within 30-days of an opioid dispensation. We described key feature distributions across the study time period and changes in model prediction performance on the validation sets using relevant metrics. RESULTS: Among 1.2 million study participants representing over 13 million opioid dispensations, there were 59 809 (2.1%), 134 402 (2.4%) and 62 143 (2.3%) events reported in the pre-pandemic (2019), pandemic (2020 and 2021) and endemic (2022) time periods, respectively (estimated 2023 validation set pre-test probability of 2.8%). Notable differences in key features were observed in the 2020-2021 model relative to other years. In the 2023 validation set, discrimination performance was highest for the pre-pandemic and endemic models compared with the pandemic model (0.81, 0.83, 0.74, respectively). A similar trend regarding changes from pre-test to post-test probabilities in higher categories of predicted risk (23%, 40%, 16%) was observed. 2020-2021 had the lowest discrimination performance (0.71) and uninformative post-test probabilities (<10%). CONCLUSION: COVID-19 pandemic health data contributed to significant ML drift. Although ML approaches allow for quick re-training to mitigate drift, health regulators should approach ML prediction with caution when using pandemic-times data.

References

1. 1.Sharma, V., Li, W., Joon, T., Dubois, C., Lau, D., Jess, E., Lindeman, C., Kain, N. A., Ye, M., Semenchuk, M., Eurich, D. T., & Samanani, S. (2026). Impact of COVID-19-related data drift on machine-learning prognostic models predicting 30-day opioid-related emergency department visits, hospitalisation or mortality: a population-level administrative data study in Alberta, Canada. BMJ Open. https://doi.org/10.1136/bmjopen-2025-103318