Research Appraisals
Evidence-based critical appraisals of the latest medical research, systematically evaluated using Oxford CEBM methodology.
Showing 2 appraisals
European journal of dental education : official journal of the Association for Dental Education in Europe
Assessment of the Large Language Models in Creating Dental Board-Style Questions: A Prospective Cross-Sectional Study
INTRODUCTION: Although some studies have investigated the application of large language models (LLMs) in generating dental-related multiple-choice questions (MCQs), they have primarily focused on ChatGPT and Gemini. This study aims to evaluate and compare the performance of five of the LLMs in generating dental board-style questions. MATERIALS AND METHODS: This prospective cross-sectional study evaluated five of the advanced LLMs as of August 2024, including ChatGPT-4o (OpenAI), Claude 3.5 Sonnet (Anthropic), Copilot Pro (Microsoft), Gemini 1.5 Pro (Google) and Mistral Large 2 (Mistral AI). The five most recent clinical guidelines published by The Journal of the American Dental Association were used to generate a total of 350 questions (70 questions per LLM). Each question was independently evaluated by two investigators based on five criteria: clarity, relevance, suitability, distractor and rationale, using a 10-point Likert scale. RESULT: Inter-rater reliability was substantial (kappa score: 0.7-0.8). Median scores for clarity, relevance and rationale were above 9 across all five LLMs. Suitability and distractor had median scores ranging from 8 to 9. Within each LLM, clarity and relevance scored higher than other criteria (p < 0.05). No significant difference was observed between models regarding clarity, relevance and suitability (p > 0.05). Claude 3.5 Sonnet outperformed other models in providing rationales for answers (p < 0.01). CONCLUSION: LLMs demonstrate strong capabilities in generating high-quality, clinically relevant dental board-style questions. Among them, Claude 3.5 Sonnet exhibited the best performance in providing rationales for answers.
6 May 2026
Read appraisal →Behavior research methods
A modified expectation-maximization algorithm for accelerated item response theory model estimation with large datasets
The expectation-maximization (EM) algorithm is widely used for parameter estimation in item response theory (IRT) modeling. However, when applied to datasets with large numbers of individuals and items, the standard EM algorithm can be slow to converge, with computationally expensive E-steps. We propose a modified EM algorithm to accelerate estimation for unidimensional two-parameter logistic IRT models. The modified algorithm uses a two-stage structure with partial-step updating over data subsets to reduce convergence time, while maintaining comparable accuracy and precision. The first two simulation studies evaluated its performance relative to standard EM, focusing on convergence time, parameter recovery, and standard error estimation across varying subset sizes and item counts. The third study demonstrated its scalability and runtime advantage in a large-scale testing scenario involving one million respondents and 100 items. The fourth study evaluated robustness under departures from unidimensionality. The proposed algorithm showed time reductions under smaller subsets with 36 items, consistent reductions across all subset sizes with 54 items, and the largest reduction (60%) with 40-item forms constructed from 100 items, while maintaining comparable estimation performance. These results highlight the algorithm's potential for large-scale applications involving tens of thousands of respondents and moderate-to-large item pools, with modifications that can be integrated into existing EM routines. We conclude with directions for operational use and potential extensions to multidimensional IRT estimation.
23 Apr 2026
Read appraisal →