Research Appraisals
Evidence-based critical appraisals of the latest medical research, systematically evaluated using Oxford CEBM methodology.
Showing 2 appraisals
JMIR research protocols
Large Language Models in German Continuing Medical Education Assessments: Protocol for a Fully Crossed Experimental Study
BACKGROUND: Continuing medical education (CME) is a legal and ethical obligation for physicians in Germany. The rapid rise of large language models (LLMs) such as ChatGPT, Gemini, Claude, and Grok raises concerns about the integrity of CME assessments, as LLMs can already pass German CME tests. OBJECTIVE: This study aims to determine whether the choice of document format (searchable PDF, protected PDF, raster PDF, or vector PDF) and LLM influences the ability of LLMs to solve CME test questions at rates exceeding the passing threshold specified for each CME module (typically 70%). METHODS: In a fully crossed within-subjects repeated-measures design, 18 expired CME articles from 3 major German publishers across 6 specialties will be converted into 3 cheating-impeding PDF formats and processed alongside the original PDF files by 4 current LLMs (GPT-5, Claude Sonnet 4, Grok-4, and Gemini 3). This results in 16 model-format combinations. Each model will answer every article 3 times per file-format condition, with outcomes derived from aggregated run-level results. The primary outcome is the proportion of correctly answered questions; the secondary outcome is the pass/fail rate. RESULTS: The study has been approved by the Witten/Herdecke University Ethics Committee (S-260/2025; dated August 10, 2025) and is preregistered at the Open Science Framework. The study is supported by internal departmental resources only, and no external funding was received. Because this protocol evaluates LLMs using expired CME materials, no human participants are being recruited. Data collection is planned to begin in June 2026 and is expected to last approximately 4 weeks. At the time of manuscript submission, no data have been collected or analyzed. Results are expected to be available after the completion of data collection and statistical analysis in 2026. The analyses will quantify performance differences across document formats; these findings may inform the feasibility of nonsearchable document formats as a temporary measure to reduce LLM-enabled cheating risks in CME contexts. CONCLUSIONS: By quantifying how document format constrains LLM performance, this study aims to evaluate simple technical safeguards that may reduce artificial intelligence-assisted manipulation of CME tests and inform regulators and CME providers about how to balance assessment validity, accessibility, and responsible LLM integration into postgraduate medical education.
30 July 2026
Read appraisal →Medical education online
AI-assisted case-based learning and flipped classroom to improve clinical decision-making: a randomized controlled trial in reproductive medicine
BACKGROUND: Efficient training of reproductive medicine clinicians is critical in the context of declining global fertility and increasing infertility. Traditional lecture‑based instruction often fails to sufficiently develop clinical decision‑making skills within limited residency rotations. Innovative strategies that integrate artificial intelligence (AI) with case‑based learning (CBL) and flipped classroom (FC) formats may enhance clinical reasoning, however rigorous evidence in reproductive medicine education remains limited. METHODS: We conducted a randomized controlled trial involving 50 obstetrics and gynecology residents at the First Hospital of Jilin University. Participants were randomly assigned to an AI‑assisted CBL+FC group or a traditional lecture control group. The AI‑assisted CBL+FC group completed pre‑class interactive case work with virtual standardized patients on the DoctorU platform and case analyses on the Superstar Learning platform, followed by interactive in‑class discussions. Primary outcomes included post‑course theoretical knowledge tests, Mini‑Clinical Evaluation Exercise (Mini‑CEX), and Objective Structured Clinical Examination (OSCE) scores. Secondary outcomes assessed learner motivation, clinical thinking, self‑directed learning, and perceived course effectiveness using a 5‑point Likert scale. RESULTS: Baseline characteristics were comparable between groups. After the intervention, the AI‑assisted CBL+FC group achieved significantly higher theoretical test scores than the control group. The AI‑assisted CBL+FC group also demonstrated superior overall clinical competence in Mini‑CEX assessments and higher OSCE total scores. Participants in the AI‑assisted CBL+FC group reported greater improvements in learning motivation, clinical reasoning, self‑directed learning, and perceived course effectiveness. CONCLUSIONS: The AI‑assisted CBL+FC instructional model significantly enhances theoretical knowledge, clinical decision‑making skills, and learner engagement among reproductive medicine residents. This blended learning model offers an efficacious and generalizable methodology for training practitioners to address the evolving clinical requirements within contemporary fertility care.
28 May 2026
Read appraisal →