Research Appraisals
Evidence-based critical appraisals of the latest medical research, systematically evaluated using Oxford CEBM methodology.
Showing 7 appraisals
PloS one
Pronunciation assessment in foreign language learning: Reliability and scoring bias in human-generative AI evaluation
This study examines the reliability and scoring bias of generative AI (Gen-AI)-based pronunciation assessment compared with human raters, addressing whether AI-generated scores can be trusted in real educational settings. Sixty students participated in a 12-week program. A total of 180 pronunciation samples were evaluated across eight subcomponents (individual phonemes, stress, rhythm, intonation, linking, reduction, fluency, and clarity) by three standardized human raters and Gen-AI using the same 7-point rubric. Quantitative analyses (intraclass correlation coefficients, paired-samples t-tests, and Pearson correlations) assessed reliability and bias, while semi-structured interviews with raters provided explanatory qualitative insights. Gen-AI demonstrated moderate reliability with human raters across most components, showing the highest agreement in fluency and the weakest in individual phonemes. However, Gen-AI consistently assigned significantly higher scores than human raters across all subcomponents. Qualitative findings revealed that discrepancies originated from Gen-AI's limited discriminative power, systematic flaws in handling missing data, decontextualized scoring approach, lack of sensitivity to L1 interference, and inability to interpret pragmatic context. While Gen-AI cannot fully replace human expertise, it can function as a complementary tool for formative assessment and autonomous practice. A hybrid assessment model integrating Gen-AI's efficiency with human raters' contextual and interpretive insights is recommended for effective foreign language pronunciation education.
31 July 2026
Read appraisal →PloS one
Enhancing the sustainability of cultural identity in science curricula through artificial intelligence as an innovative educational approach
This study investigates the role of AI as an innovative educational approach in promoting the sustainability of cultural identity within science curricula. To achieve this aim, a qualitative research design was employed, using semi-structured interviews and classroom observations as primary data collection methods. The study sample consisted of forty-seven science teachers from the Al-Qassim and Al-Ahsa regions in Saudi Arabia, selected purposively based on their experience with technology-enhanced instruction. The collected data were analyzed using systematic qualitative procedures involving open and axial coding, which facilitated the emergence of key categories related to AI integration, cultural relevance, and instructional practices. Accordingly, the analytical process is best understood as an inductive thematic qualitative approach rather than a full grounded theory methodology, as it does not include theoretical sampling, data saturation, or iterative cycles of data collection. This clarification ensures methodological transparency and alignment between design and analysis. Findings derived from both teachers' perceptions and classroom observations indicate that AI is not perceived merely as a tool for knowledge transmission, but rather as a supportive instructional mechanism that enables the contextualization of scientific concepts within local cultural settings. In addition, AI-supported instructional practices appear to foster more interactive and collaborative classroom environments, while also contributing to more responsive approaches to assessment. However, it is important to emphasize that these findings are based on teachers' reported perceptions and observed instructional practices rather than direct measurement of student learning outcomes. Despite this limitation, classroom observations suggest that integrating culturally relevant examples, analyzing cultural texts, and designing collaborative, culturally responsive learning activities may enhance student engagement and strengthen their connection to cultural identity within science learning contexts. Overall, the study highlights the potential of AI to bridge scientific knowledge and local cultural heritage in science education. It contributes to the growing body of literature on AI in education by emphasizing its role in fostering culturally responsive pedagogy and enhancing meaningful learning experiences. The study further underscores the importance of providing teachers with targeted professional development in AI integration, developing culturally responsive instructional materials, and designing assessment practices that address both conceptual understanding and cultural dimensions of learning.
24 July 2026
Read appraisal →Journal of medical Internet research
AI-Powered Simulation for Nursing Education: Mixed Methods Systematic Review
BACKGROUND: Traditional simulation-based nursing education is often constrained by high costs, resource intensity, and limited scalability. AI-powered simulations offer dynamic, scalable, and personalized alternatives. However, the empirical evidence regarding their pedagogical effectiveness and learner acceptance remains fragmented. OBJECTIVE: This study aimed to systematically evaluate and synthesize evidence on the effectiveness and learner perceptions of AI-powered simulations in nursing education. METHODS: Following PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) guidelines, we systematically searched 11 electronic databases (PubMed, CINAHL, Embase, Web of Science, Cochrane Library, Scopus, SinoMed, CNKI, Wanfang, VIP, and Google Scholar) for studies published between January 2014 and September 2025. Two independent reviewers performed study selection, data extraction, and quality appraisal using design-specific tools (risk of bias 2 tool [RoB 2; Cochrane Bias Methods Group] for randomized controlled trials [RCTs], Risk Of Bias in Nonrandomized Studies of Interventions [ROBINS-I; Cochrane Bias Methods Group] for nonrandomized studies, Mixed Methods Appraisal Tool [MMAT] for mixed methods, Joanna Briggs Institute [JBI] for qualitative, and Agency for Healthcare Research and Quality [AHRQ] for cross-sectional studies). Quantitative data were synthesized narratively, and qualitative findings were integrated using JBI meta-aggregation. A convergent segregated approach with joint display was used to generate meta-inferences. RESULTS: Nineteen studies involving 1253 participants (primarily prelicensure nursing students, with some interdisciplinary cohorts) were included. AI modalities comprised generative AI/large language models (n=7), AI-driven virtual patients/mannequins (n=5), AI-enhanced virtual/mixed reality (n=5), and chatbots (n=2). Three studies were RCTs, 4 were quasiexperimental with control groups, 3 were uncontrolled pre-post studies, 4 were mixed methods, 4 were qualitative, and one was a cross-sectional survey. Quantitative synthesis showed that evidence from RCTs and controlled quasiexperimental studies indicates significant improvements in cognitive knowledge and affective outcomes, including self-efficacy and communication confidence; however, effects on complex psychomotor skills were inconsistent, with one RCT finding AI-assisted simulation inferior to standardized patient simulation. Findings from uncontrolled designs are preliminary. Qualitative meta-aggregation revealed that learners valued safe, repeatable, nonjudgmental practice environments that reduced anxiety and bridged the theory-practice gap. Persistent challenges included technical frustrations, "robotic" interactions, lack of nonverbal cues, and system instability, collectively constituting an "authenticity gap." CONCLUSIONS: AI-powered simulations show promise for developing foundational clinical reasoning and communication skills in nursing education, though the evidence base is limited by the predominance of uncontrolled designs, reliance on self-reported measures, and absence of longitudinal data on skill retention or clinical transfer. Due to current technological limitations in replicating physical and emotional authenticity, AI should be implemented as a complementary tool alongside traditional simulation methods and clinical placements, rather than as a replacement. Future research should prioritize longitudinal outcomes, standardized competency measures, RCTs with active comparators, and implementation strategies addressing technical barriers.
22 July 2026
Read appraisal →International journal of nursing education scholarship
Leadership education for nurse practitioners: a systematic review of interventions, pedagogies and gaps
Nurse practitioners (NPs) often lead interprofessional healthcare teams, yet many receive little or no formal leadership training. As healthcare systems grow in complexity and demand greater accountability, the need for strong and effective NP leadership has intensified. However, leadership development within NP education remains poorly defined and often lacks theoretical grounding. This systematic review explores interventions designed to build NP leadership skills. Following PRISMA and BEME Guide No. 13 guidelines, a search (July 2022, updated April 2024) identified peer-reviewed English-language articles from 1980 onward. Studies involving NPs at any training or practice stage were included if they described a leadership intervention. Of 3,058 articles, eight met inclusion criteria. Most studies originated in the U.S., featured isolated leadership interventions, and lacked NP clinical faculty involvement. Experiential learning and mentorship were common. Leadership domains emphasized cognitive and emotional intelligence, with little focus on leader character development. Evaluation of leadership interventions were limited. Overall, few theoretically informed, robust NP leadership interventions exist. Future efforts should prioritize comprehensive, longitudinal, and theory-based approaches to strengthen leadership capacity within NP education and practice.
24 June 2026
Read appraisal →Medical education online
Cross-cultural adaptation and psychometric evaluation of the artificial intelligence learning intention scale (AILIS) among medical sciences students: a methodological study
BACKGROUND: This study aimed to adapt and psychometrically evaluate the Artificial Intelligence Learning Intention Scale (AILIS) among Iranian medical sciences students. METHODS: This methodological psychometric study was conducted among 800 medical sciences students at Babol University of Medical Sciences, Iran. The AILIS was translated and culturally adapted into Persian using a standardized cross-cultural adaptation process. Construct validity was examined through exploratory factor analysis (EFA) and confirmatory factor analysis (CFA). Reliability was assessed using Cronbach's alpha, McDonald's omega, and the intraclass correlation coefficient (ICC). Convergent and discriminant validity were evaluated using composite reliability (CR) and average variance extracted (AVE). Measurement invariance across gender and structural stability were assessed using multi-group confirmatory factor analysis and exploratory graph analysis (EGA). RESULTS: EFA supported a four-factor structure consisting of Epistemic Capacity, Psychological Attitudes, Facilitating Environments, and Psychological and Behavioral Outcomes, explaining 44.55% of the total variance, with factor loadings greater than 0.40. Confirmatory factor analysis indicated an acceptable model fit, supporting the proposed factor structure (χ²/df = 3.42, RMSEA = 0.078, CFI = 0.932). Internal consistency and test-retest reliability were satisfactory, with Cronbach's alpha ranging from 0.854 to 0.904, McDonald's omega from 0.854 to 0.899, and ICC from 0.750 to 0.860. Composite reliability values were adequate, while AVE values indicated acceptable but partially limited evidence of convergent validity. Discriminant validity was generally supported. In addition, multi-group analysis demonstrated measurement invariance across gender, and exploratory graph analysis confirmed the structural stability of the four-factor model. CONCLUSION: The Persian AILIS is a valid and reliable instrument for assessing AI learning intention among medical sciences students. Beyond its psychometric robustness, the instrument may support curriculum planning, evaluation of AI-related educational interventions, and evidence-based decision-making in medical sciences education by providing a standardized measure of students' intentions to engage with AI learning.
20 June 2026
Read appraisal →Journal of medical Internet research
GenAI-Supported Virtual Patients in Health Care Education: Systematic Review
BACKGROUND: Generative artificial intelligence (GenAI) is enhancing virtual patient simulations in health care education by enabling dynamic, adaptive interactions, reshaping how clinical skills are taught. A synthesis of the current evidence is needed to guide implementation and future research, given the pace of technological advancement. OBJECTIVE: This systematic review aims to synthesize empirical research on the design, implementation, and educational impact of GenAI-supported virtual patients in health care education. METHODS: A systematic search was conducted across 5 databases (CINAHL, Medline, Embase, Scopus, and Web of Science) from their inception to March 19, 2026. Reference lists of included studies and relevant systematic reviews were also screened. Peer-reviewed studies in English that evaluated GenAI-supported virtual patients using quantitative or mixed methods were included. Two reviewers independently screened studies and extracted data. Study quality and risk of bias were assessed critically using JBI (Joanna Briggs Institute) checklists, with disagreements resolved by consensus. RESULTS: A total of 15 studies met the inclusion criteria (total participants N=645), spanning health care disciplines, including nursing, medicine, pharmacy, radiography, and medical first-responder training. The virtual patients varied in design; input modalities included text (9 studies), voice (5 studies), or hybrid (1 study); output was text (9 studies), speech (5 studies), or both (1 study); 6 studies used 3D-embodied avatars, while 9 used nonembodied interfaces. A total of 13 studies used OpenAI GPT models (eg, ChatGPT), 1 used a fine-tuned model from a different provider, and 1 evaluated multiple model families (Claude, GPT, and open-source). Further, 6 studies used controlled experimental designs, including 3 randomized controlled trials (RCTs); the remainder were cross-sectional or prepost evaluations. Primary outcomes included user perceptions (14 studies), communication skills (4 studies), clinical reasoning (3 studies), and performance (7 studies). In controlled comparisons, GenAI-supported virtual patients consistently improved outcomes relative to control conditions: for example, enhanced clinical decision-making (RCT, n=21), ophthalmology history-taking skills (RCT, n=26), and medical history-taking performance (crossover RCT, n=20). The evidence base is characterized by brief intervention durations, a predominant reliance on single-session interactions, and a general lack of underpinning educational theory. No meta-analysis was performed due to the limited number of studies and significant heterogeneity in designs, interventions, and outcome measures. CONCLUSIONS: The evidence supports the feasibility and acceptability of GenAI-supported virtual patients, with positive learner perceptions and promising outcomes for skills development. However, critical limitations remain in emotional-behavioral complexity, simulation adaptability, and research design rigor (eg, limited use of control groups and validated instruments). The review offers educators, instructional designers, and policymakers actionable insights for integrating dynamic, artificial intelligence-driven simulations while identifying crucial gaps-such as the need for theoretical grounding, longitudinal studies, and standardized design protocols-that must be addressed for safe and effective implementation.
11 May 2026
Read appraisal →Journal of medical Internet research
Artificial Intelligence in Health Professions Education: Qualitative Study of Student Experiences
BACKGROUND: Artificial intelligence (AI) is increasingly integrated into education and health care, raising questions about how students use these technologies and how AI influences their learning. In health education, understanding these trends is particularly important because student learning directly impacts future clinical skills. OBJECTIVE: This study aimed to explore the use of AI tools by health sciences students at the University of Ottawa. More specifically, it sought to identify the most frequently used AI tools, describe students' usage habits, determine which tools support knowledge acquisition and skill development, and gather students' recommendations for effective strategies to raise awareness and train their peers on the responsible use of AI. METHODS: A qualitative approach was used with students from 10 health professions who reported using AI in their studies. Data were collected through semistructured interviews and an open-ended qualitative online survey. Inductive thematic analysis within an interpretive paradigm was applied to capture patterns, perceptions, and emergent themes. RESULTS: A total of 51 health professions students participated in the study. Most were women between the ages of 20 and 29 years. ChatGPT (OpenAI) emerged as the most frequently used AI tool. Students perceived AI as a complementary tool that facilitated knowledge acquisition, skill development, writing, and problem-solving. AI adoption was driven by curiosity, peer influence, and the desire to improve work efficiency. Students critically evaluated AI results, integrated the tools into their learning processes, and emphasized the importance of technical skills, critical thinking, and digital literacy. Peer learning, hands-on demonstrations, and access to online resources were recommended for effective AI training. CONCLUSIONS: This research demonstrates that health professions students actively use AI tools, particularly ChatGPT, to support learning, skill development, and academic tasks. Although AI is valuable as an educational aid and its use varies by student and context, this highlights the need for structured guidance, critical evaluation skills, and peer-supported training. These findings highlight the importance of thoughtfully integrating AI into educational programs to enhance learning outcomes, foster skill acquisition, and ensure responsible and effective adoption.
4 Apr 2026
Read appraisal →