Research Appraisals
Evidence-based critical appraisals of the latest medical research, systematically evaluated using Oxford CEBM methodology.
Showing 7 appraisals
International dental journal
Accuracy of Large Language Models in Answering Dental Examination Questions: A Systematic Review and Meta-Analysis
INTRODUCTION: Large language models (LLMs), including OpenAI's GPT family accessed via interfaces such as ChatGPT and Microsoft Copilot, as well as non-GPT systems such as Google Gemini, are increasingly applied in healthcare and dental education. However, the accuracy of these systems in specialized tasks such as answering dental examination questions remains unclear. METHODS: This systematic review and meta-analysis evaluated LLM performance in answering dental questions. Databases searched were PubMed, Embase, Scopus, and Web of Science. Data on question type and number, LLM versions, and accuracy rates were extracted. Pooled accuracy was estimated using a random-effects model; heterogeneity and publication bias were assessed. RESULTS: A total of 39 studies were included, with ChatGPT-4 being the most frequently evaluated model. The pooled accuracy for LLMs was 63.7% (95% CI: 60.3%-67.1%), with high heterogeneity (I² = 91.5%). Subgroup analysis revealed ChatGPT-4 and Copilot (a GPT-based interface) achieved the highest pooled accuracies (∼73% and ∼75%, respectively). Direct comparisons confirmed ChatGPT-4 significantly outperformed earlier versions and some competitor models. Sensitivity analyses supported the robustness of findings. CONCLUSION: LLMs demonstrate moderate accuracy in answering dental examination questions and are currently insufficient for autonomous clinical decision-making. When their limitations are explicitly recognized, however, these systems may serve as valuable adjuncts in dental education and examination preparation. Methodological strategies such as structured prompting and retrieval-augmented approaches warrant further investigation but were not the primary focus of the present analysis.
2 Aug 2026
Read appraisal →JMIR medical informatics
Large Language Models for Endodontic Symptom Assessment and Treatment Planning Using Image-Free Clinical Records: Comparative Evaluation Study
BACKGROUND: Accurate assessment of pulpal status is essential for achieving successful endodontic outcomes. However, direct evaluation remains inherently challenging because the pulp is surrounded by calcified tissue, necessitating reliance on clinical and radiographic examinations for diagnostic and prognostic decision-making. These procedures demand substantial clinical expertise and time, and less-experienced clinicians often face challenges that may lead to errors in diagnosis and treatment planning. Recent advancements in large language models (LLMs) offer promising opportunities to enhance clinical reasoning by facilitating the integration of evidence and supporting methodical diagnostic decision-making. OBJECTIVE: This study aimed to evaluate the clinical applicability of LLMs by comparing their text-based clinical screening performance and the clinical validity of their treatment plan responses with those of human evaluators. METHODS: Between January 2011 and December 2022, 100 clinical cases involving primary endodontic disease were randomly selected from the clinical records of outpatients who visited the Department of Conservative Dentistry or Advanced General Dentistry (AGD) at Yonsei University Dental Hospital. Four prompt types, combining 2 variables (language and role), were used as input for 4 LLMs. Both LLMs and human evaluators (AGD specialists, AGD residents, endodontic residents, and senior dental students) assessed the cases using text-based clinical records. Radiographic images were not directly provided. Screening performance was evaluated using a 0-to-2-point concordance scale, and treatment plan validity and relevance were assessed using a 5-point Likert scale. RESULTS: Among the 4 LLMs evaluated, ChatGPT achieved the highest mean concordance score on Korean-doctor prompts (mean 0.98, SD 0.82). However, this score did not reach the partially correct criterion of 1 on the 0 to 2-point scale. Clova X recorded the lowest mean score on English-patient prompts (mean 0.23, SD 0.63). Across both diagnostic categories, AGD specialists demonstrated the highest diagnostic accuracy (pulpal: 0.70; periapical: 0.65), with higher sensitivity but lower specificity than those exhibited by the other groups. ChatGPT also showed favorable performance among the LLMs, with accuracies of 0.65 (95% CI 0.55-0.74) for pulpal disease and 0.57 (95% CI 0.47-0.69) for periapical disease, which were comparable to those of AGD and endodontic residents. CONCLUSIONS: Under image-free clinical record review conditions, ChatGPT 4.0 showed relatively higher and more consistent performance in symptom screening and treatment planning compared to the other LLMs evaluated. However, its highest mean score of 0.98 (SD 0.82) did not reach the partially correct criterion of 1 on the 0 to 2-point scale. Hallucinations generated by LLMs and experience-dependent interpretation biases among human evaluators remain key challenges that require attention. Therefore, continuous clinical supervision and comprehensive user training are necessary for the safe and effective clinical application.
25 July 2026
Read appraisal →JMIR medical education
Evaluating the Pediatric Behavior Guidance of Students Based on Actual Clinical Transcripts Scored by Faculty and Large Language Models: Pilot Comparative Study
BACKGROUND: Personalized feedback improves the clinical pediatric behavior guidance performance of students but is prohibitively time-consuming to provide. Large language models (LLMs) can automate the process of evaluating clinical sessions but are limited to text-only input and consistency issues. OBJECTIVE: This study compared the use of text-only transcripts against the use of video recordings for evaluating the clinical behavior guidance performance of dental students. Additionally, the consistency and accuracy of LLMs in evaluating the transcripts were compared against a human assessor. METHODS: This study was conducted by using 40 video-recorded clinical encounters involving final-year dental students who were managing patients aged between 4 and 12 years at the Faculty of Dentistry, National University of Singapore. The videos were scored by using a previously validated pediatric behavior guidance scale. Clinical encounters were transcribed verbatim and scored by a study member using a modified version of the scale (nonverbal components removed). The time taken to rate the transcripts was recorded. Video scores were compared with transcript scores. Both the free-to-use version and the paid version of the ChatGPT LLM were also used to score the transcripts; consistency was evaluated and compared against the human assessor. RESULTS: The average time taken to rate the transcripts (mean 12, range 3-25 min) was significantly (P<.001) lower than the average video length (mean 73, range 37-120 min). Comparing transcript scores with video scores resulted in a consistency intraclass correlation coefficient of 0.830 (95% CI 0.679-0.910; P<.001), demonstrating good reliability. Comparing transcript scores with the free-to-use LLM's and paid LLM's scores yielded an absolute agreement intraclass correlation coefficient of 0.729 (95% CI 0.475-0.859; P<.001) and 0.670 (95% CI 0.377-0.825; P<.001), respectively, demonstrating moderate agreement. The LLMs were inconsistent, producing variable scores with the same prompt. The free-to-use and paid versions produced the same score for all 3 runs in only 7 (18%) and 4 (10%) of the 40 clinical encounters, respectively. CONCLUSIONS: Using transcripts to evaluate students' clinical behavior guidance was time-saving for faculty, demonstrated good agreement with video-based evaluation, and could improve clinical teaching. Although LLMs can automate the task, improvements are needed to improve their consistency and accuracy.
14 June 2026
Read appraisal →Scientific reports
3D‑printed training model for periodontal splinting: a randomized preclinical study
Preclinical simulation is essential for students to practice before clinical treatment. A low-cost 3D-printed model was developed with repositionable mobile maxillary anterior teeth to train periodontal splinting and evaluate learner perception and repositioning accuracy. In a two-period randomized cross-over course 43 students in their fourth year completed two palatal slot splints from the right to the left maxillary canine (FDI 13-23) in randomized material order. They used a polyethylene ribbon and a pre-impregnated unidirectional glass-fibre bundle. Outcomes included an electronic questionnaire using an anchored 6‑point ordinal scale from 1 for "Excellent" to 6 for "Unsatisfactory" following the German school grading system and geometric deviation of each tooth to an ideal position. The course was rated overall as good (Md 2.0, IQR 1.0) and highly clinically relevant (Md 1.0, IQR 1.0). Self-reported preparedness for splinting improved from poor (Md 5.0, IQR 1.0) before the course to good (Md 2.0, IQR 1.0; p < 0.001) after the course. In this pilot study, no statistically detectable positional deviations between materials or between the first and second session in this cohort were observed. This inexpensive, reproducible 3D-printed model enables realistic periodontal splinting training under simulated mobility conditions and may support structured preparation for clinical teaching.
25 May 2026
Read appraisal →JMIR formative research
Feasibility of Large Language Model-Based Standardized Virtual Patients to Support Clinical Decision-Making Training in Operative Dentistry: Mixed Methods Study
BACKGROUND: Clinical decision-making training in operative dentistry commonly relies on real or standardized patients to develop undergraduate students' ability to deliver safe, effective, and patient-centered care. However, training with real or standardized patients can be limited in scalability, cost-effectiveness, and accessibility. Large language models, with their human-like language capabilities, may have the potential to simulate patients in clinical encounters and help overcome some limitations associated with traditional training approaches. OBJECTIVE: This study aimed to evaluate the feasibility of using large language model-based standardized virtual patients to support undergraduate dental students' clinical decision-making training in operative dentistry. METHODS: This mixed methods cross-sectional feasibility study was conducted during a simulation-based clinical decision-making training session in the Operative Dentistry and Cariology course at the College of Dentistry, King Saud bin Abdulaziz University for Health Sciences, Riyadh, Saudi Arabia. Eligible participants were second-year undergraduate dental students enrolled in the course. A convenience sampling approach was used, with all eligible students (N=50) invited to participate. A total of 41 students completed the study, 23 (56%) of whom were male. The students were divided into 8 groups. Each group interacted with 2 standardized virtual patients powered by ChatGPT-4o (OpenAI) through the Chatbase platform to complete comprehensive history-taking and then reviewed the standardized virtual patients' intraoral photographs and bitewing radiographs. For each standardized virtual patient, students as a group recorded diagnoses, performed a risk assessment, and formulated a treatment plan. Students then completed the Student Satisfaction and Self-Confidence in Learning questionnaire. The quality of the standardized virtual patient responses and overall dialogue realism were evaluated using the Dialogue Authenticity Scale. The dialogues were also thematically analyzed to identify authenticity-undermining response features and explore their context and underlying causes. RESULTS: Students perceived the simulation-based training session positively, with all questionnaire items showing high median scores (4.00-5.00 on a 5-point scale), and both item-level IQRs and 95% CIs spanning no more than 1.0 scale point. In addition, standardized virtual patient responses were largely authentic, with an overall median authenticity rating of 4.50 (IQR 4.00-5.00; 95% CI 4.00-5.00) on a 6-point scale across all interactions. However, several authenticity-undermining response features were identified, including responses that were inconsistent with typical human behavior, contained information beyond a patient's likely knowledge, or were factually incorrect. CONCLUSIONS: This proof-of-concept study supports the feasibility of implementing large language model-based standardized virtual patients in undergraduate simulation-based clinical decision-making training in operative dentistry. In a dental context where this application has been only minimally evaluated, this study provides early evidence of positive student perceptions, acceptability, and largely authentic dialogue, while also identifying important performance limitations. Further research is warranted to optimize performance and to evaluate the educational effectiveness of this approach in improving undergraduate students' clinical skills and knowledge.
20 May 2026
Read appraisal →European journal of dental education : official journal of the Association for Dental Education in Europe
Assessment of the Large Language Models in Creating Dental Board-Style Questions: A Prospective Cross-Sectional Study
INTRODUCTION: Although some studies have investigated the application of large language models (LLMs) in generating dental-related multiple-choice questions (MCQs), they have primarily focused on ChatGPT and Gemini. This study aims to evaluate and compare the performance of five of the LLMs in generating dental board-style questions. MATERIALS AND METHODS: This prospective cross-sectional study evaluated five of the advanced LLMs as of August 2024, including ChatGPT-4o (OpenAI), Claude 3.5 Sonnet (Anthropic), Copilot Pro (Microsoft), Gemini 1.5 Pro (Google) and Mistral Large 2 (Mistral AI). The five most recent clinical guidelines published by The Journal of the American Dental Association were used to generate a total of 350 questions (70 questions per LLM). Each question was independently evaluated by two investigators based on five criteria: clarity, relevance, suitability, distractor and rationale, using a 10-point Likert scale. RESULT: Inter-rater reliability was substantial (kappa score: 0.7-0.8). Median scores for clarity, relevance and rationale were above 9 across all five LLMs. Suitability and distractor had median scores ranging from 8 to 9. Within each LLM, clarity and relevance scored higher than other criteria (p < 0.05). No significant difference was observed between models regarding clarity, relevance and suitability (p > 0.05). Claude 3.5 Sonnet outperformed other models in providing rationales for answers (p < 0.01). CONCLUSION: LLMs demonstrate strong capabilities in generating high-quality, clinically relevant dental board-style questions. Among them, Claude 3.5 Sonnet exhibited the best performance in providing rationales for answers.
6 May 2026
Read appraisal →JMIR formative research
Evaluation of GPT-5 in Periodontitis Staging and Grading: Retrospective Observational Study
BACKGROUND: Periodontitis is a chronic gum disease affecting approximately 42% of adults aged 30 years and older in the United States. Training dental students to accurately diagnose and manage periodontitis is a critical component of dental education and clinical care. Recent advances in large language models offer new opportunities to support both domains, yet their performance in periodontal diagnosis remains largely unexplored, particularly for newer models such as GPT-5. OBJECTIVE: This study conducted an exploratory evaluation of GPT-5's ability to stage and grade periodontitis. METHODS: A total of 25 publicly available clinical cases explicitly reporting periodontitis stage and grade were identified through Google and PubMed searches. Each case description was entered into GPT-5 using a zero-shot prompting approach to assess guideline-based reasoning without exemplar conditioning. The model's predictions were compared with the published reference diagnoses. Performance was measured using accuracy, 95% CI, unweighted Cohen κ, and weighted Cohen κ. RESULTS: Across these cases, GPT-5 showed marked class-dependent performance and a tendency to overestimate disease severity. Grading performance was notably imbalanced, with high recall for grade C but substantially lower discrimination for grade B. GPT-5 achieved a staging accuracy of 68% (95% CI 48.4%-82.8%) and a grading accuracy of 77.3% (95% CI 56.6%-89.9%), with corresponding Cohen κ values of 0.454 (95% CI 11.0%-75.6%) and 0.179 (95% CI -15.8% to 63.8%), respectively. While staging performance showed fair agreement beyond chance, the low κ for grading indicates poor agreement and limited reliability in distinguishing periodontal disease severity. CONCLUSIONS: These findings suggest that although GPT-5 demonstrates potential for guideline-based periodontitis staging and grading, its current diagnostic performance, particularly for periodontitis grading, limits its use in clinical assessment and educational training. Meaningful application in periodontal diagnosis and training will require substantial improvements in reliability and rigorous validation in larger, more diverse, and prospectively collected datasets.
7 Apr 2026
Read appraisal →