Auditing the impact of social media's policy shift on anti-vaccine discourse: A large language model-driven empirical study
Clinical Snapshot
PICO Framework
| P — Population | Twitter/X users posting vaccine-related content during November 16-30, 2022 |
| I — Intervention | Termination of X's misinformation policy on November 23, 2022 |
| C — Comparator | Seven-day period before policy termination (November 16-22, 2022) |
| O — Outcomes | Prevalence, thematic composition, and engagement metrics of anti-vaccine discourse |
Bottom Line
This study provides evidence that removing content moderation policies on social media platforms can rapidly increase anti-vaccine discourse volume and amplification. The 60% increase in anti-vaccine tweets following X's policy termination suggests content moderation may effectively constrain misinformation spread. However, methodological limitations including unclear sampling, insufficient confounder adjustment, and limited validation of automated classification methods weaken confidence in these findings. While the natural experiment design is valuable, the short observation period and potential for unmeasured confounding limit causal inference. For Australian healthcare professionals, these findings highlight the importance of proactive vaccine communication strategies and potential collaboration with social media platforms, but should be interpreted cautiously given the study's limitations. The shift toward health concern themes rather than conspiracy narratives may inform targeted public health messaging approaches.
Key Findings
P Value: Not reported in abstract
Effect Size: OR = 1.60
Primary Outcome: 60% increase in anti-vaccine tweets following policy termination
Nnt Or Sensitivity: Increased amplification primarily through retweets rather than new content creation
Confidence Interval: 95% CI: 1.50-1.72
Clinical Application
Findings inform content moderation strategies but require platform-specific implementation Relevant for Australian health authorities monitoring social media vaccine discourse, particularly given TGA's role in vaccine safety communication and potential coordination with social media platforms Healthcare professionals, public health authorities, and policymakers concerned with vaccine misinformation
Abstract
The sudden termination of X's (formerly Twitter) misinformation policy on November 23, 2022, provides an opportunity to assess the effects of lifting content moderation restrictions on vaccine-related discourse. This study examines changes in the prevalence, thematic composition, and engagement of anti-vaccine discourse following X's policy shift, analyzing tweets from a seven-day period before and after the policy termination (November 16-30, 2022), excluding the announcement date itself from regression analyses. Using GPT-4o for stance classification, thematic categorization, and stance consistency assessment, with validation through external benchmarks and cross-annotator agreement, we find that anti-vaccine tweets increased significantly post-policy (OR = 1.60, 95% CI: 1.50-1.72), particularly via retweets, suggesting content amplification. Sensitivity analyses excluding highly retweeted content revealed that the policy change was also associated with increased creation of new anti-vaccine content. Thematically, health concerns over vaccination became more prominent, while conspiracy-related and anti-mandate narratives declined in relative prevalence. Stance consistency in quote tweets increased, indicating reinforced ideological alignment in anti-vaccine discourse. These results suggest that content moderation policies may constrain both the volume and amplification of anti-vaccine content, with policy removal associated with rapid shifts in discourse patterns.
References
- 1.Li, Y., Chen, T., Zhao, Y., Ke, W., Pang, P., McKay, D., Chang, S., & Baxter, N. (2026). Auditing the impact of social media's policy shift on anti-vaccine discourse: A large language model-driven empirical study. PLoS One. https://doi.org/10.1371/journal.pone.0346568
Related Research
JMIR formative research
Teaching Health Informatics in a Skills-First Era: A Curriculum Mapping Approach Aligned With Workforce Demand
31 July 2026
Proceedings of the National Academy of Sciences of the United States of America
Race-conscious admissions algorithms and the law.
28 July 2026
Journal of medical Internet research
Digital Transformation in Health Care: Are We on the Right Track?
22 July 2026
This content is for educational purposes for healthcare professionals only and does not constitute clinical advice. Clinical decisions should be based on individual patient assessment, current guidelines, and appropriate specialist consultation. Editorial Standards · Privacy Policy · Terms of Service