Development, Feasibility, Acceptability, and Usability of an Artificial Intelligence-Powered Chatbot (Suzy) to Support Patients in Substance Use Disorder Recovery: Multiphase Study

Clinical Snapshot

Key Findings

• P Value: Not applicable for qualitative study

• Effect Size: High usability demonstrated across standardized measures

• Primary Outcome: Feasibility, acceptability, and usability of AI-powered chatbot for SUD recovery support

• Nnt Or Sensitivity: SUS score 93 (benchmark 68), NPS 63 (benchmark 35), SEQ 6.5/7

• Confidence Interval: Not applicable for qualitative study

Clinical Application

Technically feasible with appropriate safety protocols, though requires integration with existing care workflows and staff training Relevant to Australian SUD services facing similar workforce challenges. Would require adaptation to local clinical guidelines, privacy regulations, and integration with existing digital health infrastructure Patients with substance use disorders receiving outpatient treatment, particularly those needing enhanced support between clinical contacts

Abstract

BACKGROUND: Substance use disorder (SUD) remains a major public health crisis in the United States, with significant challenges in treatment access, retention, and workforce capacity. SUD care teams, including addiction medicine physicians and peer recovery coaches (PRCs), support patients receiving SUD treatment but face heavy workloads and burnout. Artificial intelligence (AI) innovations, particularly large language model (LLM)-based chatbots, may extend PRC support and provide patients with on-demand recovery support between clinic visits and PRC contacts. However, evidence on their development, feasibility, acceptability, and usability in addiction services remains limited. OBJECTIVE: This study describes the development, feasibility, acceptability, and usability of an AI-powered health coaching chatbot (Suzy) designed to support patients in SUD recovery. METHODS: A total of 2 clinicians, 5 researchers, and 2 technology developers led a small, multiphase pilot study. In the formative phase, they conducted focus groups and qualitative in-depth interviews with 12 health care professionals and 8 patients with substance use histories to specify chatbot functions and develop a rule-based chatbot. In phase 2, they conducted usability testing of the rule-based chatbot with 8 patients who reported substance use and completed standardized tasks, surveys, and qualitative interviews. Measures included the System Usability Scale (SUS), Net Promoter Score (NPS), and Single Ease of Use Question (SEQ). In phase 3, they developed an LLM-based chatbot co-designed and fine-tuned with PRCs and other SUD experts. RESULTS: Rule-based chatbot functions included craving management, appointment reminders, resource referrals, care team contacts, and goal setting. Usability task testing supported feasibility. In this small pilot sample, quantitative and qualitative feedback indicated acceptability and usability, with an average SUS score of 93 (benchmark 68), an NPS of 63 (benchmark 35), and a mean SEQ score of 6.5/7. Patients valued Suzy's approachable, nonjudgmental language and features that promoted accountability, self-monitoring, and 24/7 availability, while emphasizing that chatbots should supplement but not replace human support. The LLM-based chatbot development emphasized information accuracy, safety escalation protocols to mitigate risks of inappropriate chatbot responses, human-in-the-loop features, and expanded conversational flexibility and personal tailoring. CONCLUSIONS: In this pilot study, a rule-based chatbot designed to support SUD care demonstrated feasibility, usability, and acceptability. LLM-based chatbot development required more robust safety and emergency reporting features, while offering more patient-responsive conversational functions. By providing on-demand coaching, referrals, and reminders, Suzy may extend the reach of care teams, alleviate provider burden, and enhance patient engagement. Additional work is needed to understand how to best integrate Suzy into patients' recovery journeys to ensure human support remains accessible and prioritized. LLM evaluation was based on expert testing and safety review. Clinical effectiveness, including the impact on substance use, was not evaluated. Next steps include evaluating the LLM chatbot in real-world settings with larger samples and assessing its efficacy in reducing substance use.

References

1. 1.Comulada, W. S., Swendeman, D., Ho, Y. X., Streck, J. M., Lewis, D.-M., Rezai, R., Warren, D., Pachas, G., Chandler, M., Jackson, J. L., & Gelberg, L. (2026). Development, feasibility, acceptability, and usability of an artificial intelligence-powered chatbot (Suzy) to support patients in substance use disorder recovery: Multiphase study. JMIR Formative Research. https://doi.org/10.2196/84683