Assessment of the Large Language Models in Creating Dental Board-Style Questions: A Prospective Cross-Sectional Study
Clinical Snapshot
PICO Framework
| P — Population | Five large language models (ChatGPT-4o, Claude 3.5 Sonnet, Copilot Pro, Gemini 1.5 Pro, Mistral Large 2) as of August 2024 |
| I — Intervention | Generation of dental board-style multiple-choice questions based on recent clinical guidelines from Journal of American Dental Association |
| C — Comparator | Comparison between different LLM models |
| O — Outcomes | Question quality assessed by two investigators using 5 criteria (clarity, relevance, suitability, distractor, rationale) on 10-point Likert scale |
Bottom Line
This cross-sectional study demonstrates that contemporary large language models can generate high-quality dental board-style questions with median quality scores exceeding 8-9 across multiple evaluation criteria. All five tested models (ChatGPT-4o, Claude 3.5 Sonnet, Copilot Pro, Gemini 1.5 Pro, Mistral Large 2) showed strong performance in clarity and relevance, with Claude 3.5 Sonnet providing superior answer rationales. The substantial inter-rater reliability (kappa 0.7-0.8) supports the validity of the evaluation framework. While these findings suggest LLMs could assist dental educators in question development, several limitations warrant consideration: the study lacks validation against actual examination performance, provides limited confidence intervals, and was conducted using US clinical guidelines in a Vietnamese context. Australian dental educators should consider local curriculum requirements and AHPRA standards when implementing such tools. The rapid evolution of LLM capabilities also means these findings may quickly become outdated, requiring ongoing evaluation of newer model versions.
Key Findings
P Value: p < 0.05 for within-LLM criterion differences; p < 0.01 for Claude 3.5 Sonnet rationale superiority
Effect Size: Median scores >9 for clarity, relevance, and rationale; 8-9 for suitability and distractor across all LLMs
Primary Outcome: Question quality across five evaluation criteria using 10-point Likert scale
Nnt Or Sensitivity: Inter-rater reliability kappa = 0.7-0.8 (substantial agreement)
Confidence Interval: Not reported
Clinical Application
High feasibility given widespread LLM availability, though requires expertise in prompt engineering and quality assurance Relevant to Australian dental education but requires adaptation to AHPRA/Dental Board standards and local clinical guidelines rather than US-based sources Dental educators, examination boards, and institutions developing assessment materials
Abstract
INTRODUCTION: Although some studies have investigated the application of large language models (LLMs) in generating dental-related multiple-choice questions (MCQs), they have primarily focused on ChatGPT and Gemini. This study aims to evaluate and compare the performance of five of the LLMs in generating dental board-style questions. MATERIALS AND METHODS: This prospective cross-sectional study evaluated five of the advanced LLMs as of August 2024, including ChatGPT-4o (OpenAI), Claude 3.5 Sonnet (Anthropic), Copilot Pro (Microsoft), Gemini 1.5 Pro (Google) and Mistral Large 2 (Mistral AI). The five most recent clinical guidelines published by The Journal of the American Dental Association were used to generate a total of 350 questions (70 questions per LLM). Each question was independently evaluated by two investigators based on five criteria: clarity, relevance, suitability, distractor and rationale, using a 10-point Likert scale. RESULT: Inter-rater reliability was substantial (kappa score: 0.7-0.8). Median scores for clarity, relevance and rationale were above 9 across all five LLMs. Suitability and distractor had median scores ranging from 8 to 9. Within each LLM, clarity and relevance scored higher than other criteria (p < 0.05). No significant difference was observed between models regarding clarity, relevance and suitability (p > 0.05). Claude 3.5 Sonnet outperformed other models in providing rationales for answers (p < 0.01). CONCLUSION: LLMs demonstrate strong capabilities in generating high-quality, clinically relevant dental board-style questions. Among them, Claude 3.5 Sonnet exhibited the best performance in providing rationales for answers.
References
- 1.Nguyen, V. A., & Nguyen, T. T. (2026). Assessment of the Large Language Models in Creating Dental Board-Style Questions: A Prospective Cross-Sectional Study. European Journal of Dental Education. https://doi.org/10.1111/eje.70015
Related Research
International dental journal
Accuracy of Large Language Models in Answering Dental Examination Questions: A Systematic Review and Meta-Analysis
2 Aug 2026
JMIR medical informatics
Large Language Models for Endodontic Symptom Assessment and Treatment Planning Using Image-Free Clinical Records: Comparative Evaluation Study
25 July 2026
JMIR medical education
Evaluating the Pediatric Behavior Guidance of Students Based on Actual Clinical Transcripts Scored by Faculty and Large Language Models: Pilot Comparative Study
14 June 2026
This content is for educational purposes for healthcare professionals only and does not constitute clinical advice. Clinical decisions should be based on individual patient assessment, current guidelines, and appropriate specialist consultation. Editorial Standards · Privacy Policy · Terms of Service