Research Appraisals
Evidence-based critical appraisals of the latest medical research, systematically evaluated using Oxford CEBM methodology.
Showing 3 appraisals
JMIR medical education
Evaluating the Pediatric Behavior Guidance of Students Based on Actual Clinical Transcripts Scored by Faculty and Large Language Models: Pilot Comparative Study
BACKGROUND: Personalized feedback improves the clinical pediatric behavior guidance performance of students but is prohibitively time-consuming to provide. Large language models (LLMs) can automate the process of evaluating clinical sessions but are limited to text-only input and consistency issues. OBJECTIVE: This study compared the use of text-only transcripts against the use of video recordings for evaluating the clinical behavior guidance performance of dental students. Additionally, the consistency and accuracy of LLMs in evaluating the transcripts were compared against a human assessor. METHODS: This study was conducted by using 40 video-recorded clinical encounters involving final-year dental students who were managing patients aged between 4 and 12 years at the Faculty of Dentistry, National University of Singapore. The videos were scored by using a previously validated pediatric behavior guidance scale. Clinical encounters were transcribed verbatim and scored by a study member using a modified version of the scale (nonverbal components removed). The time taken to rate the transcripts was recorded. Video scores were compared with transcript scores. Both the free-to-use version and the paid version of the ChatGPT LLM were also used to score the transcripts; consistency was evaluated and compared against the human assessor. RESULTS: The average time taken to rate the transcripts (mean 12, range 3-25 min) was significantly (P<.001) lower than the average video length (mean 73, range 37-120 min). Comparing transcript scores with video scores resulted in a consistency intraclass correlation coefficient of 0.830 (95% CI 0.679-0.910; P<.001), demonstrating good reliability. Comparing transcript scores with the free-to-use LLM's and paid LLM's scores yielded an absolute agreement intraclass correlation coefficient of 0.729 (95% CI 0.475-0.859; P<.001) and 0.670 (95% CI 0.377-0.825; P<.001), respectively, demonstrating moderate agreement. The LLMs were inconsistent, producing variable scores with the same prompt. The free-to-use and paid versions produced the same score for all 3 runs in only 7 (18%) and 4 (10%) of the 40 clinical encounters, respectively. CONCLUSIONS: Using transcripts to evaluate students' clinical behavior guidance was time-saving for faculty, demonstrated good agreement with video-based evaluation, and could improve clinical teaching. Although LLMs can automate the task, improvements are needed to improve their consistency and accuracy.
14 June 2026
Read appraisal →PloS one
Evaluation of large language models in medical examinations: A scoping review protocol
INTRODUCTION: Standardized medical examinations, used to assess trainee clinical competencies, provide a rigorous means to verify LLM accuracy and reliability in medical contexts. Although current evaluations use these exams to test LLMs' clinical reasoning, significant performance variations occur across different clinical scenarios. Existing methods struggle to adapt to evolving research needs. This study synthesizes prior research on LLMs in medical exams, highlighting current limitations and proposing future research directions. METHODS AND ANALYSIS: The formulation of the protocol was guided by the standards set forth in the JBI Manual for Evidence Synthesis. Following the establishment of precise inclusion/exclusion criteria and search strategies, we will execute systematic searches in the PubMed and Web of Science Core Collection databases. The method encompasses literature review, data extraction, analytical frameworks, and process mapping. By employing this method, researchers maintain methodological rigor during the entire research process. ETHICS AND DISSEMINATION: This protocol describes a method for performing a scoping review. The investigation focuses on the organized synthesis and examination of previously published research. It does not include human/animal experimentation or sensitive data collection. Ethical approval is not required for this literature-based study.
24 Apr 2026
Read appraisal →Annals of plastic surgery
Evaluating GPT-4 Performance on Plastic Surgery Oral Examination Vignettes: A Quantitative and Qualitative Analysis
BACKGROUND: The use of artificial intelligence (AI) in medicine is rapidly evolving. However, its role in plastic and reconstructive surgery remains underexplored. Plastic surgery requires nuanced, dynamic decision making, and individualized care, making AI integration challenging. This study evaluates GPT-4's ability to respond to American Board of Plastic Surgery (ABPS) Oral Board-style clinical vignettes and assesses its potential as a decision making and educational adjunct. METHODS: Twelve clinical vignettes spanning aesthetic, reconstructive, hand, craniofacial, pediatric, and trauma surgery were input into GPT-4. Each response was scored by 2 board-certified plastic surgeons across 6 clinical domains: Case Introduction, Diagnosis, Treatment Planning, Patient Counseling, Operative Steps, and Complication Management. Domains were scored on a 0-4 scale; scores <2 were considered failing. Interrater reliability was assessed via weighted κ . A structured qualitative interview was conducted. RESULTS: GPT-4 achieved mean domain scores ranging from 2.3 to 3.7 across vignettes, with passing rates of 91.7% (rater 1) and 83.3% (rater 2). The highest-performing domains were Diagnosis (84.4%) and Complication Handling (82.3%), and the lowest-performing domains were seen in Case Introductions (62.5%). Interrater agreement was strong across domains ( κ = 0.70-0.89). Qualitative findings emphasized GPT-4's accuracy in guideline-based cases, limitations in trauma triage, and potential as an educational resource. CONCLUSION: GPT-4 demonstrates concordance with board-level clinical reasoning in structured plastic surgery scenarios. However, limitations in heuristic judgment and real-time adaptability underscore the need for optimization with specialty-specific training datasets and performance rubrics. With appropriate curation, GPT-4 could serve as a valuable supplement in surgical education and oral board preparation.
13 Apr 2026
Read appraisal →