Optimising supervised machine learning algorithms predicting cigarette cravings and lapses for a smoking cessation just-in-time adaptive intervention (JITAI).
Clinical Snapshot
PICO Framework
| P — Population | 37 adults attempting smoking cessation |
| I — Intervention | Ecological momentary assessment (EMA) with varying frequencies (3-16 prompts/day) for 10 days |
| C — Comparator | Different EMA configurations (frequency, predictor count, training data source) |
| O — Outcomes | Machine learning algorithm performance (F1-score, ROC-AUC) for predicting smoking lapses and cravings |
Bottom Line
This study evaluated machine learning algorithms for predicting smoking lapses and cravings using ecological momentary assessment data from 37 participants over 10 days. While the methodology was sound, algorithm performance was modest with substantial inter-individual variability. Lapse prediction slightly outperformed craving prediction, but many configurations fell below acceptable performance thresholds. Interestingly, more intensive EMA sampling and larger predictor sets did not consistently improve performance, suggesting simpler approaches may be equally effective. The findings highlight the challenges of real-world behavioural prediction and suggest that machine learning alone may be insufficient for just-in-time adaptive interventions in smoking cessation. The small sample size and short follow-up period limit generalisability, and the substantial individual variation suggests personalised approaches may be necessary. For Australian clinicians, these results suggest caution in implementing EMA-based prediction systems without further validation and likely combination with other intervention strategies.
Key Findings
P Value: Not reported in abstract
Effect Size: Lapse prediction: F1-score 0.436 (IQR 0.180-0.625), ROC-AUC 0.659 (IQR 0.514-0.809); Craving prediction: F1-score 0.400 (IQR 0.048-0.649), ROC-AUC 0.628 (IQR 0.510-0.729)
Primary Outcome: Machine learning algorithm performance for predicting smoking lapses and cravings
Nnt Or Sensitivity: Substantial proportion of configurations fell below minimum performance thresholds, particularly for F1-score
Confidence Interval: Performance varied substantially across individuals with wide interquartile ranges
Clinical Application
Limited by modest algorithm performance and substantial inter-individual variability; may require combination with rule-based approaches Relevant to Australian smoking cessation services, but would need validation in local population; PBS-funded smoking cessation aids could potentially incorporate JITAI approaches if performance improves Adults attempting smoking cessation who can engage with smartphone-based EMA systems
Abstract
This study aimed to optimise the balance between participant burden and algorithm performance for predicting high-risk moments in a smoking cessation just-in-time adaptive intervention (JITAI) by systematically varying ecological momentary assessment (EMA) prompt frequency, predictor count, and training data source. Thirty-seven participants completed 16 EMAs per day for the first 10 days of their smoking cessation attempt, reporting mood, context, behaviour, cravings, and smoking lapses. Random forest algorithms predicting lapses and cravings were evaluated in terms of F1-score and ROC-AUC via mixed effects models accounting for clustering within individuals. Performance across out-of-sample individuals ranged from excellent to poor but was, on average, modest. Lapse prediction outperformed craving prediction, particularly for ROC-AUC (Median F1-score: Lapses 0.436 [IQR 0.180-0.625], Cravings 0.400 [IQR 0.048-0.649]; Median ROC-AUC: Lapses 0.659 [IQR 0.514-0.809], Cravings 0.628 [IQR: 0.510-0.729]). A substantial proportion of configurations fell below commonly used minimum performance thresholds, particularly for F1-score. Reducing EMA frequency had outcome- and metric-dependent effects. Lapse F1-scores improved with fewer prompts (16 EMAs: 0.254 [IQR 0.081-0.500], 3 EMAs: 0.588 [IQR 0.353-0.667]), while ROC-AUC showed a slight, inconsistent decline (16 EMAs: 0.661 [IQR 0.520-876], 4 EMAs: 0.613 [IQR 0.494-0.786], 3 EMAs: 0.704 [IQR 0.567-0.809]). For cravings, both metrics declined with fewer prompts (F1-score: 16 EMAs: 0.470 [IQR 0.141-0.745]; 3 EMAs: 0.333 [IQR 0.000-0.600]; ROC-AUC: 16 EMAs 0.700 [IQR 0.582-0.811], 3 EMAs 0.544 [IQR 0.421-0.676]). Feature reduction had negligible impact on lapse prediction (F1-score: all features 0.435, selected features 0.441; ROC-AUC: all 0.660, selected 0.657), but slightly reduced craving performance (F1-score: all 0.410 [IQR 0.117-0.646], selected 0.400 [IQR 0.000-0.650]; ROC-AUC: all 0.632, selected 0.622). Including participant-specific data improved lapse F1-scores (None 0.286 [IQR 0.000-0.571], 30 pc 0.542 [IQR: 0.329-0.667]), but not ROC-AUC (None 0.655 [IQR: 0.512-0.786], 30 pc 0.694 [IQR 0.513-0.852]); and impaired craving ROC-AUC (None 0.650 [IQR: 0.544-0.734], 30 pc 0.614 [IQR 0.493-0.730]; F1-score: None 0.424 [IQR 0.143-0.649], 30 pc 0.400 [IQR 0.000-0.703]). Overall, EMA-based machine learning detected lapse risk but showed modest overall performance and substantial inter-individual variability. Algorithms using higher EMA density, larger predictor sets, and participant-specific training data did not consistently outperform over more parsimonious approaches. However, machine learning prediction alone is unlikely to be sufficient for real-world JITAI implementation and may be best combined with complementary rules-based approaches.
References
- 1.Leppin, C., Brown, J., Garnett, C., Kale, D., Okpako, T., Simons, D., & Perski, O. (2026). Optimising supervised machine learning algorithms predicting cigarette cravings and lapses for a smoking cessation just-in-time adaptive intervention (JITAI). PLoS One. https://doi.org/10.1093/ntr/ntab174
Related Research
The Journal of rheumatology
Examining the Role of Wearables in Inflammatory Arthritis Care: A Narrative Literature Review
3 Aug 2026
Journal of medical Internet research
Advancing Human-Centered AI in Clinical Decision Support: Sociocognitive Human-in-the-Loop Study in HIV Care
2 Aug 2026
Journal of psychiatric research
Systematic review of machine learning and deep learning models for EEG-based detection of depression.
2 Aug 2026
This content is for educational purposes for healthcare professionals only and does not constitute clinical advice. Clinical decisions should be based on individual patient assessment, current guidelines, and appropriate specialist consultation. Editorial Standards · Privacy Policy · Terms of Service