Using randomization to compare AI and expert-generated formative assessment questions in medical education
Clinical Snapshot
PICO Framework
| P — Population | Medical students at a single center |
| I — Intervention | AI-generated formative assessment questions (one per day for 28 days) |
| C — Comparator | Expert-generated formative assessment questions (one per day for 28 days) |
| O — Outcomes | Student performance (proportion correct responses), subjective evaluation of question quality, perceived difficulty ratings |
Bottom Line
This randomized study suggests AI-generated formative assessment questions perform similarly to expert-generated questions in medical education, with students showing comparable performance and perceptions. However, AI questions were rated as easier, raising concerns about assessment rigor. The study's methodological limitations—including unclear randomization procedures, missing sample sizes, lack of confidence intervals, and single-center design—significantly limit confidence in the findings. Key educational outcomes like learning retention and knowledge transfer were not assessed. While the research addresses an important question about AI in medical education, the evidence quality is insufficient to support widespread implementation. Medical educators should await more robust studies with proper methodological reporting, multi-center validation, and comprehensive outcome measures before adopting AI-generated assessment tools. The finding that AI questions may be perceived as easier warrants particular attention in maintaining assessment standards.
Key Findings
P Value: 0.18 for perception comparison
Effect Size: Not reported
Primary Outcome: No significant difference in student perceptions between AI and expert-generated questions
Nnt Or Sensitivity: 53% of AI questions rated easy/very easy vs 31% of expert questions
Confidence Interval: Not provided
Clinical Application
High feasibility for implementation in medical schools with appropriate AI tools and oversight mechanisms Relevant to Australian medical schools but would need validation within LCME/AMC accreditation frameworks and consideration of local curriculum standards Medical students in formative assessment settings, though generalizability uncertain due to single-center design
Abstract
BACKGROUND: AI-generated content is being used across the education spectrum and is beneficial for creating complex multiple-choice questions, such as those used in medical education. However, evaluating AI-generated content is challenging, and existing testing and evaluation methods are falling short. This study uses randomization to compare medical students' performance on and subjective evaluation of AI versus expert-generated questions. We hypothesized that there would be no difference in student performance or subjective evaluation of AI vs expert-generated questions. METHODS: We designed a single-center, randomized study where medical student participants received either one AI- or expert-generated question per day over four weeks. RESULTS: Our study showed that participants had similar perceptions of AI versus expert-generated questions (p = 0.18), with no significant difference between the distributions of the proportion of correct responses. The cumulative proportion of questions answered correctly over 28 days was consistent across the two question sets. However, participants rated 53% of AI-generated questions as very easy or easy compared with only 31% of the expert-generated questions. DISCUSSION: Randomization was crucial to show that AI-generated questions were nearly imperceptible from expert-generated questions, demonstrating the need for additional evaluation methods to compare AI- and expert-generated medical education content.
Related Research
Clinical gastroenterology and hepatology : the official clinical practice journal of the American Gastroenterological Association
Artificial Intelligence Tools for Gastrointestinal Research: A Practical Guide
2 Aug 2026
Artificial intelligence in medicine
Artificial intelligence language models for medical text analysis: A systematic review
2 Aug 2026
Surgery
Artificial intelligence to revolutionize surgical decision-making is still just around the corner.
1 Aug 2026
This content is for educational purposes for healthcare professionals only and does not constitute clinical advice. Clinical decisions should be based on individual patient assessment, current guidelines, and appropriate specialist consultation. Editorial Standards · Privacy Policy · Terms of Service