Off-the-Shelf Large Language Models for Guiding Pharmacoepidemiological Study Design

Clinical Snapshot

Key Findings

• P Value: Not reported

• Effect Size: Coherence >90% for most components; relevance variable (65-97% agreement)

• Primary Outcome: AI coherence and relevance in pharmacoepidemiological study design support

• Nnt Or Sensitivity: Coding accuracy: ATC 50%, CPT 22.2%, ICD 20%

• Confidence Interval: Not reported

Clinical Application

High - uses readily available AI tools, but requires expert oversight Relevant for TGA post-market surveillance and PBS safety studies, but coding systems may differ from Australian standards Pharmacoepidemiologists and regulatory scientists designing observational studies

Abstract

This study aimed to assess the ability of two off-the-shelf large language models, ChatGPT and Gemini, to support the design of pharmacoepidemiological studies. We assessed 48 study protocols of pharmacoepidemiological studies published between 2018 and 2024, covering various study types, including disease epidemiology, drug utilization, safety, and effectiveness. The coherence (i.e., "Is the response coherent and well-formed, or is it difficult to understand?") and relevance (i.e., "Is the response relevant and informative, or is it lacking in substance?") of the large language models' responses were evaluated by human experts across seven key study design components. Coding accuracy was assessed. Both large language models demonstrated high coherence, with over 90% of study components rated as "Strongly agree" by experts for most categories. ChatGPT achieved the highest coherence for "Index date" (97.9%) and "Study design" (95.8%). Gemini excelled in "Study outcome" (93.9%) and "Study exposure" (95.9%). Relevance, however, was more variable, with ChatGPT aligning with expert recommendations in over 90% of cases for "Index date" and "Study design" but showing lower agreement for covariates (65%) and follow-up (70%). Coding agreement percentages reveal varying levels of concordance, with the Anatomical Therapeutic Chemical classification system coding system demonstrating the highest agreement at 50% with experts. In contrast, the Current Procedural Terminology and International Classification of Diseases systems showed agreements of 22.2% and 20%, respectively. While ChatGPT and Gemini show promise in certain tasks supporting pharmacoepidemiological study design, their limitations in relevance and coding accuracy highlight the need for critical oversight by domain experts.

References

1. 1.Ompad, G., Wurst, K., Ramcharran, D., Hviid, A., Bate, A., & Sessa, M. (2024). Off-the-shelf large language models for guiding pharmacoepidemiological study design. Clinical Pharmacology and Therapeutics. https://doi.org/10.1002/cpt.70039