Research Appraisals
Evidence-based critical appraisals of the latest medical research, systematically evaluated using Oxford CEBM methodology.
Showing 10 appraisals
World journal of urology
Assessing multiple chatGPT versions on novel content in the Taiwan urology board examination: accuracy, speed, and domain-specific performance
PURPOSE: Large language models demonstrate increasing utility in healthcare; however, their capacity to handle newly developed medical content beyond training data boundaries remains unclear. This study evaluated the performance of four ChatGPT models on 120 newly developed questions from the 2024 Taiwan Urology Board Examination, all created after the models' October 2023 training cutoff, thereby reducing the risk of prior model exposure. METHODS: Model performance was assessed using accuracy and processing time. Of 150 examination items, the first 120 newly developed questions constituted the primary analysis set; the remaining 30 previously used archival questions were excluded to reduce potential bias from prior exposure. The primary paired accuracy comparison evaluated o1-preview versus GPT-4o across all 120 questions. Additional analyses across 12 urological subspecialties and question-complexity strata were treated as exploratory. RESULTS: Across all 120 questions, o1-preview achieved 66.7% accuracy and significantly outperformed GPT-4o (55.8%; p = 0.012), although with longer processing time (19.20 vs. 14.94 s). Performance varied significantly across 12 urological subspecialties (p < 0.001). Unlike the other tested models, o1-preview showed slightly higher accuracy on high-complexity than on low-complexity questions (68.3% vs. 65.0%). On high-complexity surgical anatomy items, o1-preview achieved 80.0% accuracy, whereas GPT-4o and GPT-4o mini both scored 0.0% (p = 0.024). CONCLUSION: On this newly developed post-cutoff, text-based urology board question set, o1-preview achieved higher overall accuracy than GPT-4o, particularly on high-complexity surgical anatomy items, at the cost of longer response times. These findings suggest that reasoning-oriented models may offer advantages for selected specialty-level assessment tasks, although generalizability to other languages, modalities, and clinical settings requires further validation.
30 May 2026
Read appraisal →Journal of robotic surgery
Extended reality and emerging artificial intelligence in orthopedic surgical training: a scoping review of educational outcomes
Immersive technologies have gained increasing relevance in orthopedic surgical education; however, the scope, outcomes, and maturity of extended reality (XR) and artificial intelligence (AI) applications remain heterogeneous. To map the educational applications, learner populations, outcome measures, and research gaps related to the use of XR and emerging AI tools in orthopedic surgical training. A scoping review was conducted following Joanna Briggs Institute methodology and reported according to PRISMA-ScR guidelines. PubMed, Scopus, and ScienceDirect were searched for English and Spanish studies published between 2015 and 2025. Fifty-four studies involving 3,066 participants were included. Virtual reality (VR) was the predominant modality (83.3%), followed by augmented reality (31.4%), while AI-based applications were infrequently reported. XR-based training was most commonly evaluated in trauma surgery, arthroscopy, and arthroplasty, primarily among medical students and residents. Most studies reported improvements in simulation-based technical performance metrics, whereas evidence on clinical outcomes, long-term skill retention, and cost-effectiveness was limited. XR-particularly VR-represents the most mature immersive technology in orthopedic surgical education. AI applications remain emergent and primarily supportive. Future research should prioritize standardized outcome measures, multicenter designs, and evaluation of long-term educational and clinical impact.
25 May 2026
Read appraisal →BMJ open
Performance of large language models (GPT-5.2, Gemini 3 Pro, Claude Sonnet 4.6 and Grok 4.1) on the Fellowship of The Royal College of Surgeons Urology Part A examination.
OBJECTIVES: To assess and compare the performance of four contemporary frontier large language models (LLMs)-GPT-5.2 (OpenAI), Gemini 3 Pro (Google DeepMind), Claude Sonnet 4.6 (Anthropic) and Grok 4.1 (xAI)-on a simulated Fellowship of The Royal College of Surgeons Urology (FRCS(Urol)) Part A examination, evaluating overall accuracy, subspecialty-level performance, output consistency and response time. DESIGN: Controlled comparative evaluation study using a standardised simulation framework with repeated independent testing runs per model. SETTING: All models were accessed via their respective consumer-facing interfaces. No clinical setting or patient data were involved. Testing was conducted under uniform conditions with conversational memory disabled across all sessions. PARTICIPANTS: Four large language models were evaluated. No human participants were involved. Models were selected to represent the current frontier of publicly accessible LLMs from four distinct commercial developers. No models were excluded following selection. INTERVENTIONS: Each model was presented with 240 FRCS (Urol) Part A single best answer questions, mapped to the Joint Committee on Intercollegiate Examinations' Urology Syllabus Blueprint (2023). A standardised prompt was delivered at the start of each session. Each model completed five independent examination runs. No fine-tuning or system-level modification was applied to any model. PRIMARY AND SECONDARY OUTCOME MEASURES: The primary outcome was overall examination accuracy for each model, benchmarked against an indicative pass threshold for the FRCS (Urol) Part A examination. Secondary outcomes were performance across 18 individual urology subspecialty topics; response time reported as mean total and per-question elapsed time; and consistency of performance quantified by SD and 95% CIs derived from a sequential Monte Carlo sampling procedure. All outcomes were prospectively planned and fully measured as specified. RESULTS: Three of four models exceeded the indicative 74% pass threshold: Gemini 3 Pro (82.4%±0.9%; 95% CI 81.3 to 83.6%), Claude Sonnet 4.6 (79.3%±1.1%; 95% CI 77.9 to 80.6%) and GPT-5.2 (76.1%±2.4%; 95% CI 73.1 to 79.1%). Grok 4.1 failed (70.4%±0.6%; 95% CI 69.6 to 71.2%), with its entire CI below 74%. All models completed the assessment in under 3 min. Strong performance was observed in research methodology (90-98%) and andrology (92-98%), with the weakest results in paediatric urology (38.7-54.7%) and testicular cancer (48.2-67.3%). Substantial within-model output instability was identified across several domains, most notably GPT-5.2 in female urology (SD±22.8%) and anatomy (SD±14.2%). CONCLUSIONS: Three of four frontier LLMs achieved scores consistent with passing the FRCS (Urol) Part A examination, representing a substantial advance since ChatGPT-3.5. Aggregate accuracy alone, however, obscures important subspecialty weaknesses and output instability. LLMs should be regarded as adjunctive revision aids rather than authoritative knowledge sources and always used alongside expert-led teaching. Future work should evaluate performance on Part B and viva-style assessments.
15 May 2026
Read appraisal →Journal of robotic surgery
Robotic versus laparoscopic intracorporeal anastomosis learning curve: a systematic review
The learning curve for robotic colorectal surgery is increasingly characterised. However, the specific learning curve for intracorporeal anastomosis (IA) remains poorly defined. This review compares robotic and laparoscopic IA with respect to operative time, cases required for proficiency, complication rates, and conversion to extracorporeal anastomosis (EA). A systematic review was conducted in accordance with PRISMA guidelines. The protocol was registered on Open science framework ( https://doi.org/10.17605/OSF.IO/K4MYZ ). On March 18, 2025, OVID and PubMed were searched using the search terms; "laparoscopic," "robotic," "anastomosis," "learning," "education," and "training." Yielding a total of 273 results from PubMed (179) and OVID (94). After screening a total of thirteen studies were included. Thirteen studies met inclusion criteria, including eight comparative analyses and six directly comparing robotic and laparoscopic techniques. Four studies evaluated robotic learning curves without a comparator group, while one was a literature review incorporating institutional experience. Most studies were retrospective (n = 8), with two prospective clinical studies and two experimental simulation-based studies. Hybrid approaches (robotic mobilisation with laparoscopic EA) contributed to heterogeneity (Reitz et al. 2018, https://doi.org/10.1007/s00464-018-6074-7 ). Learning curve assessment was heterogeneous, with most studies using cumulative sum (CUSUM) or risk-adjusted CUSUM (RA-CUSUM) analyses based on operative time or technical performance metrics. Proficiency in robotic IA was reported after approximately 11-29 cases in some series and around 20 cases in others, although larger studies of robotic colorectal surgery demonstrated broader ranges of up to 100 cases. Robotic right hemicolectomy with IA demonstrated a shorter learning curve and reduced anastomosis time compared with laparoscopic IA. Complication rates, conversion rates, and oncological outcomes were comparable between approaches (Gachabayov et al. 2019, Surg Technol Int, 34:163-168). Robotic intracorporeal anastomosis, particularly in right hemicolectomy, appears to have a shorter and less technically demanding learning curve compared with laparoscopic techniques, without compromising safety or oncological outcomes. These findings support a potential role for robotic platforms in facilitating adoption of intracorporeal anastomosis. However, current evidence is limited by retrospective design, small cohorts, and inconsistent definitions of proficiency (Gachabayov et al. 2019, Surg Technol Int, 34:163-168; Van Eetvelde et al. 2022, https://doi.org/10.1007/s11701-022-01514-6 ).
14 May 2026
Read appraisal →World journal of urology
Quantifying ergonomic challenges for urologists operating flexible ureteroscopes through artificial intelligence-based posture estimation
PURPOSE: The ergonomic challenges faced by surgeons during flexible ureteroscopy have yet to be thoroughly evaluated using objective methods. However, robot-assisted ureteroscopy has emerged as a promising treatment for urinary stones. The present study quantitatively assesses operative posture during transurethral ureteroscopy via an artificial intelligence-based posture estimation framework, and identifies kinematic differences associated with surgical expertise. METHODS: Expert and novice urologists performed standardized flexible ureteroscopic tasks in a simulated transurethral ureteroscopy environment using kidney phantoms containing artificial stones. Upper-body movements were continuously video-recorded and joint coordinates of the shoulders, elbows, and wrists were extracted using an artificial intelligence-based pose estimation system. Kinematic parameters were quantitatively analyzed and compared between groups. RESULTS: Experts completed observation of the entire renal pelvis significantly faster than novices (median 58 vs. 102 s, p < 0.001). Accumulated travel distance was markedly less in experts for the left shoulder (125 vs. 300 cm, p < 0.001), right elbow (324 vs. 1035 cm, p = 0.028), left elbow (349 vs. 772 cm, p = 0.019), and left wrist (729 vs. 3798 cm, p < 0.001). Experts had smaller movement areas at the left shoulder (6.29 vs. 40.5 cm2, p < 0.001) and right elbow (17.3 vs. 281 cm2, p = 0.040), and reduced movement ranges across multiple joints. Their angle fluctuation ranges were also narrower for the right shoulder (32.0 vs. 76.3°, p = 0.028) and left elbow (70.4 vs. 122.4°, p < 0.001). CONCLUSIONS: The artificial intelligence-based posture analysis objectively demonstrated ergonomic advantages associated with surgical expertise during flexible ureteroscopy and revealed persistent ergonomic risks to the wrist and elbow.
11 May 2026
Read appraisal →JMIR research protocols
Evaluating the Role of Surgical Telementoring in the Acquisition of Surgical Skills in Laparoscopic Cholecystectomy: Protocol for a Pilot Randomized Controlled Trial
BACKGROUND: Developing surgical skills among residents requires time and resources, often in conflict with the goals of efficiency and hospital economics in surgical practice. Optimizing resident training may prevent a decline in quality. Skilled surgeons are vital to effective and safe surgical performance, yet the training trajectory for surgical residents is often inefficient, particularly in the era of minimally invasive surgery. A framework that efficiently disseminates surgical skills is essential to cultivate skilled future surgeons within an acceptable time frame. Surgical telementoring (ST) aligns well with minimally invasive surgery because the mentee operating in the field shares the telementor's perspective. Despite growing experience with telemedicine, data on educational outcomes remain limited. OBJECTIVE: We aim to conduct a pilot randomized controlled trial (RCT). Data collection will inform a future full-scale RCT evaluating whether real-time ST, combined with structured postoperative coaching, is superior to traditional on-site mentoring in developing laparoscopic cholecystectomy skills among surgical residents. METHODS: Eligible residents will be randomly assigned in a 1:1 ratio to either the intervention group (real-time ST and postoperative coaching) or the control group (traditional intraoperative hands-on teaching). This research adheres to the extended SPIRIT (Standard Protocol Items: Recommendations for Interventional Trials) statements for pilot RCTs and follows the intention-to-treat principle. Residents in both groups will perform 5 consecutive procedures under standardized intraoperative conditions. The primary outcomes are the Global Operative Assessment of Laparoscopic Skills (GOALS) score and the Non-Technical Skills for Surgeons (NOTSS). We hypothesize that the intervention group will show a 3- to 5-point improvement in the GOALS score compared with the control group. To achieve 0.8 statistical power, a P value <.05, and a 20% dropout rate in a definitive RCT, 12 residents per group are required. We present a pilot RCT with 5 residents per group. Challenges encountered during the trial will be reported, and amendments will be proposed to assess the need for a future definitive RCT. RESULTS: The primary outcomes will be technical and nontechnical surgical skills, measured by the GOALS and NOTSS scores, respectively. Patient-reported experience measures and satisfaction scores of eligible mentees and mentors will be analyzed as secondary outcomes. The ST system configuration at the 2 inclusion centers was completed in December 2023. Ethics approval and the most recent update report to ClinicalTrials.gov were obtained on October 23, 2023, and December 8, 2024, respectively. Recruitment of residents to the study commenced in spring 2024. Recruitment is expected to be completed in May 2026, with data analysis in November 2026 and manuscript submission of results in January 2027. CONCLUSIONS: The SURGTEACH (Surgical Telementoring in Acquisition of Surgical Skills of Laparoscopic Cholecystectomy) trial is the first pilot RCT evaluating ST for educational outcomes. This research may provide high-quality evidence to improve surgical education, particularly in rural hospitals.
18 Apr 2026
Read appraisal →Annals of plastic surgery
Evaluating GPT-4 Performance on Plastic Surgery Oral Examination Vignettes: A Quantitative and Qualitative Analysis
BACKGROUND: The use of artificial intelligence (AI) in medicine is rapidly evolving. However, its role in plastic and reconstructive surgery remains underexplored. Plastic surgery requires nuanced, dynamic decision making, and individualized care, making AI integration challenging. This study evaluates GPT-4's ability to respond to American Board of Plastic Surgery (ABPS) Oral Board-style clinical vignettes and assesses its potential as a decision making and educational adjunct. METHODS: Twelve clinical vignettes spanning aesthetic, reconstructive, hand, craniofacial, pediatric, and trauma surgery were input into GPT-4. Each response was scored by 2 board-certified plastic surgeons across 6 clinical domains: Case Introduction, Diagnosis, Treatment Planning, Patient Counseling, Operative Steps, and Complication Management. Domains were scored on a 0-4 scale; scores <2 were considered failing. Interrater reliability was assessed via weighted κ . A structured qualitative interview was conducted. RESULTS: GPT-4 achieved mean domain scores ranging from 2.3 to 3.7 across vignettes, with passing rates of 91.7% (rater 1) and 83.3% (rater 2). The highest-performing domains were Diagnosis (84.4%) and Complication Handling (82.3%), and the lowest-performing domains were seen in Case Introductions (62.5%). Interrater agreement was strong across domains ( κ = 0.70-0.89). Qualitative findings emphasized GPT-4's accuracy in guideline-based cases, limitations in trauma triage, and potential as an educational resource. CONCLUSION: GPT-4 demonstrates concordance with board-level clinical reasoning in structured plastic surgery scenarios. However, limitations in heuristic judgment and real-time adaptability underscore the need for optimization with specialty-specific training datasets and performance rubrics. With appropriate curation, GPT-4 could serve as a valuable supplement in surgical education and oral board preparation.
13 Apr 2026
Read appraisal →Archives of gynecology and obstetrics
Comparative validation of a cost-effective in-house 3D-printed LLETZ simulator versus a commercial training model.
PURPOSE: Traditional surgical training methods such as "learning-by-doing" raise ethical and methodological concerns. To improve training and reduce patient risk, a 3D-printed LLETZ simulator was developed. While initial studies showed advantages over a conventional model, they were limited to medical students and lacked comparison with a commercially available simulator. METHODS: A single-center study was conducted at the University Hospital Wuerzburg. 60 medical students without prior LLETZ experience and 10 gynecology residents with prior exposure were randomly assigned to train on either a commercial or the novel in-house simulator. Each participant performed five electrosurgical excisions. Performance was evaluated using LEEP scores, resection status (R0 resections), and blinded video assessments by two senior clinicians. Additionally, participants completed questionnaires, to capture subjective training impressions. RESULTS: The in-house simulator demonstrated superior performance compared to the commercial model. When analyzed separately, changes in LEEP scores over the five attempts were statistically significant for both simulators. The effect size was larger for the in-house simulator (η2 = 0.227) than for the conventional simulator (η2 = 0.10). Within-group analysis revealed no significant pairwise differences across all attempts for the commercial simulator. In contrast, several pairwise comparisons remained statistically significant for the in-house simulator (attempts 1 vs. 3, 1 vs. 4, 2 vs. 5, and 1 vs. 5), all with large effect sizes (Cohen's d > 1.1). Between-group comparison of individual LEEP scores showed a statistically significant difference in the fifth attempt (p = 0.002), with a large effect size (Cohen's d = 1.01) favoring the in-house simulator. Higher R0 resection rates were observed with the in-house simulator in the third (100% vs. 83.3%) and fourth attempts (96.7% vs. 73.3%). Blinded video assessments by two senior experts confirmed these findings, demonstrating higher checklist, Global Rating Scale (GRS), and overall mean scores for the in-house simulator from the third attempt onward (all p < 0.05; all Cohen's d > 0.8). Participant feedback further supported these results, indicating improved confidence, technical skills, and perceived educational value. CONCLUSION: This study demonstrated that the novel in-house 3D-printed simulator significantly outperformed the commercial model in objective surgical performance, learning progression and user satisfaction.
5 Apr 2026
Read appraisal →J Surg Educ
Effect of Artificial Intelligence-Augmented Human Instruction on Feedback Frequency and Surgical Performance During Simulation Training
OBJECTIVE: To determine whether personalized feedback from a human instructor receiving artificial intelligence (AI) error data will result in reduced feedback frequency and improvement of surgical skill compared to AI instruction. As feedback was only provided following AI error detection, a reduced feedback frequency is associated with fewer errors in performance. We hypothesized that AI-augmented personalized instruction would result in reduced feedback frequency and improvement in technical skill. DESIGN: This cross-sectional cohort study was a follow-up of a randomized controlled trial, which utilized the NeuroVR, an immersive virtual reality neurosurgical simulator. Participants were stratified by year in medical school and block randomized to receive one of 3 educational interventions as they performed simulated procedures on the NeuroVR: AI tutor instruction, scripted human instruction, and AI-augmented personalized instruction. Performance was assessed by the feedback frequency and technical skill performance metrics. CLINICALTRIALS: gov ID: NCT06273579. SETTING: Neurosurgical Simulation and Artificial Intelligence Learning Centre, McGill University, Montreal, Canada. PARTICIPANTS: Volunteer sample of medical students from 4 Quebec universities in preparatory, first, or second year without prior use of the NeuroVR. Eighty-eight students participated in the study with 87 included in the final analysis; 1 was excluded due to technical issues. RESULTS: By the third repetition, the AI-augmented personalized instruction group received significantly fewer total instructions (incidence rate ratio [IRR], 1.50 [95% CI, 1.16-1.94] instructions; p < 0.001), and high aspirator force instructions (IRR, 1.71 [95% CI, 1.15-2.55] instructions; p = 0.002), compared to the second repetition. Compared to AI tutor instruction, AI-augmented personalized instruction resulted in a significantly lower rate of healthy tissue removal (p = 0.01), instrument tip separation distance (mean ratio, 1.25 [95% CI, 1.05-1.50] mm; p = 0.008), and aspirator force (mean ratio, 1.68 [95% CI, 1.23-2.31] N; p < 0.001). AI-augmented personalized instruction showed a significant improvement from baseline in all subsequent repetitions for all performance metrics. CONCLUSIONS: This cohort study demonstrated that AI-augmented personalized instruction resulted in less frequent feedback, indicating fewer errors in trainee performance, and an improvement in simulated surgical skills.
1 Nov 2025
Read appraisal →Am Surg
Enhancing Surgical Education Through Artificial Intelligence in the Era of Digital Surgery
Artificial Intelligence (AI) is transforming surgical education, playing a pivotal role in the era of digital surgery. This review, conducted from January 2023 to December 2024, examines the integration of AI into surgical training, focusing on its benefits and limitations. Thirteen studies were included, with emphasis on AI's impact in General Surgery training. Digital surgery leverages AI-powered tools such as virtual reality (VR) and augmented reality (AR) to create immersive, simulated environments where residents can practice surgical techniques safely and effectively. These platforms improve technical skills, hand-eye coordination, and error reduction, while personalized AI-driven feedback systems enhance learning and skill assessment. Digital surgery also bridges the gap between theoretical knowledge and clinical application, fostering innovative training methodologies. Despite challenges such as financial constraints and ethical concerns, AI within digital surgery emerges as a transformative force, redefining surgical education and equipping trainees with the expertise needed for modern health care.
1 Nov 2025
Read appraisal →