Research Appraisals
Evidence-based critical appraisals of the latest medical research, systematically evaluated using Oxford CEBM methodology.
Showing 8 appraisals
JMIR nursing
Assessing Digital Literacy Among Nursing Faculty Under the Artificial Intelligence-Enhanced Technological Pedagogical Content Knowledge Framework: Cross-Sectional Survey Analysis
BACKGROUND: Digital literacy has become a core competence for nursing faculty to adapt to the digital transformation of education, improve teaching quality, and cultivate innovative nursing talents. OBJECTIVE: This study aimed to investigate the digital literacy status and influencing factors among nursing faculty in colleges and universities in Fujian province, China. METHODS: A cross-sectional survey analysis was conducted among 339 participants from July 2023 to June 2024 by selecting nursing faculty from 3 medical universities and 5 colleges in Fujian province. A general information questionnaire and a teacher digital literacy questionnaire (designed based on the artificial intelligence-enhanced technological pedagogical content knowledge theoretical model) were used to conduct the survey. The questionnaires included the following dimensions: an intelligent integration ethics layer (digital social responsibility), attention to the moral boundaries of applying technology; an awareness layer (digital awareness), the willingness to actively adapt to technological changes; a knowledge layer (digital technology knowledge and skills), mastery of the integration point of intelligent tools and subject teaching; an ability layer (digital application), the ability to design intelligent teaching plans; and a thinking layer (professional development; innovative thinking for the critical integration of technology). Statistical analysis included descriptive statistics, a 1-way ANOVA, Pearson correlation analysis, and multiple linear regression (using SPSS 28.0 software). RESULTS: The overall digital literacy of the nursing faculty was at a medium level (mean score 101.92, SD 16.47), with the highest score in digital awareness (mean 20.21, SD 9.43) and the lowest score in digital technology knowledge and skills (mean 9.68, SD 2.92). For the dimension of digital technology knowledge and skills, both age (β=-.142; P=.009) and years of teaching (β=-.147; P=.006) were significant negative predictors. Regarding digital application, age was found to be a significant negative predictor (β=-0.124; P=.02) but teaching experience was a positive predictor (β=0.123, P=.02). Similarly, age also had a significant negative impact on the professional development dimension (β=-.153; P=.005) and the overall digital literacy level (β=-.136; P=.01). CONCLUSIONS: The digital literacy of nursing faculty is at a medium level, with especially low scores in technical knowledge. Age and teaching experience are key influencing factors. Recommendations to promote balanced development of awareness and skills include providing targeted training for senior faculty, integrating courses with clinical practice, optimizing the allocation of digital resources, promoting interdisciplinary cooperation, and strengthening ethical and safety training.
14 June 2026
Read appraisal →Postgraduate medical journal
Peer review in the age of artificial intelligence: a comparative study of human and AI-generated review reports
BACKGROUND: Peer review is central to maintaining scientific quality and helps editors make decisions. However, the volume of scientific publications continues to rise, placing pressure on the peer review system. With the rise of generative AI, its role in supporting peer review is gaining attention. This study aims to compare human-written and AI-generated peer review reports. METHODS: We analysed 398 peer review reports linked to 119 research articles published in BMJ Open in 2024. Publicly available reports and manuscripts were included. Editorials, corrections, and protocols were excluded. AI reports were generated using ChatGPT. All reports were anonymized and assessed by two independent reviewers. We conducted a hybrid thematic analysis. Frequencies of themes were calculated and compared by reviewer type. For quantitative comparison, we used the Mann-Whitney U test to assess differences in review quality scores and Fisher's exact test to compare the distribution of themes. All analyses were conducted using R software. RESULTS: Human reviewers gave more detailed and diverse comments. They addressed deeper issues like interpretation, originality, and applicability. AI reviews covered more sections but focused on routine or structural elements. AI outperformed slightly in format-related domains. Co-occurrence analysis showed human reviews linked diverse themes, while AI comments were structurally clustered. Shannon Index confirmed that human reviews were more thematically diverse. CONCLUSIONS: AI can support peer review by screening for basic errors. However, it lacks insight, critical judgment, and contextual awareness. Human input remains essential for meaningful review. Review-specific AI tools that preserve confidentiality are needed for future integration.
22 May 2026
Read appraisal →Medical education online
Using randomization to compare AI and expert-generated formative assessment questions in medical education
BACKGROUND: AI-generated content is being used across the education spectrum and is beneficial for creating complex multiple-choice questions, such as those used in medical education. However, evaluating AI-generated content is challenging, and existing testing and evaluation methods are falling short. This study uses randomization to compare medical students' performance on and subjective evaluation of AI versus expert-generated questions. We hypothesized that there would be no difference in student performance or subjective evaluation of AI vs expert-generated questions. METHODS: We designed a single-center, randomized study where medical student participants received either one AI- or expert-generated question per day over four weeks. RESULTS: Our study showed that participants had similar perceptions of AI versus expert-generated questions (p = 0.18), with no significant difference between the distributions of the proportion of correct responses. The cumulative proportion of questions answered correctly over 28 days was consistent across the two question sets. However, participants rated 53% of AI-generated questions as very easy or easy compared with only 31% of the expert-generated questions. DISCUSSION: Randomization was crucial to show that AI-generated questions were nearly imperceptible from expert-generated questions, demonstrating the need for additional evaluation methods to compare AI- and expert-generated medical education content.
12 May 2026
Read appraisal →Graefe's archive for clinical and experimental ophthalmology = Albrecht von Graefes Archiv fur klinische und experimentelle Ophthalmologie
Comparing the performance of four mainstream large language models on medical literature review generation: a human expert evaluation in SMILE surgery
PURPOSE: To systematically evaluate and compare the performance of four leading large language models (LLMs) in generating medical literature reviews across topics of varying research maturity, thereby providing insights for their effective and responsible application in academic writing. METHODS: In this comparative study, using standardized prompts, we instructed four leading LLMs (GPT-4, Gemini 2.5 Pro, Grok-3, and DeepSeek R1) to generate literature reviews on nine topics related to small incision lenticule extraction (SMILE) surgery. These topics were categorized into three groups by research maturity: well-researched, controversial, and open. Seven ophthalmology experts evaluated the generated content across four dimensions: quality, accuracy, bias, and relevance, while all references were verified for authenticity. Performance differences among models were evaluated using group comparison tests followed by post-hoc analysis. RESULTS: Significant performance variations were identified across all four models and dimensions (p < 0.001). Specifically, Gemini ranked highest in content quality, accuracy, and bias control. In contrast, DeepSeek, despite its high-quality score, received the lowest relevance score. Grok-3 demonstrated the highest reference authenticity (p < 0.001), whereas GPT-4's was the lowest (p < 0.001). All models showed diminished performance on open topics and exhibited severe reference fabrication ("hallucinations"). CONCLUSION: Rather than excelling universally, LLMs exhibit distinct and task-specific strengths that mandate a task-driven, hybrid strategy in tool selection. Reference fabrication was found to be a pervasive issue across all models, regardless of the task topic, elevating human verification from a best practice to an essential safeguard for academic integrity.
9 May 2026
Read appraisal →PloS one
Gender disparities in Italian academic medicine: A cross-sectional study of clinicians in the 2024 stanford top 2% scientists database
BACKGROUND: Women remain underrepresented in senior and influential research positions despite gradual improvements over recent decades.The aim of this study was to assess gender representation among highly cited Italian clinical scientists and to examine differences across academic fields, institutions, and geographic regions. METHODS: We conducted a cross-sectional analysis of Italian clinical scientists included in the 2024 Stanford Top 2 Percent Scientists database. Gender was determined from given names, and demographic and academic information was obtained from national physician registries and publicly available curriculum vitae. Characteristics were compared by gender, and a logistic regression model was used to examine factors associated with female representation. RESULTS: A total of 3389 clinical scientists were identified, of whom 824 were women. Women represented approximately one quarter of highly cited Italian clinical researchers. They were younger and had more recent years of first publication compared with men. No significant gender differences were observed in academic affiliation or geographic distribution. Representation varied across fields, with surgical specialties demonstrating the lowest proportions of women and toxicology the highest. However, in most disciplines, representation converged toward the overall average when sample sizes increased, indicating that academic fields alone contributed minimally to gender representation. In the multivariable model, only toxicology showed significantly greater female representation, while more recent year of first publication was independently associated with being a woman.. CONCLUSIONS: Women remain underrepresented among highly cited Italian clinical scientists, despite evidence of gradual improvement in representation over time. The limited influence of academic fields suggests that broader structural and cultural factors likely contribute to these gaps. These findings highlight the need for targeted strategies to promote equal representation and support the career advancement of women in academic medicine.
28 Apr 2026
Read appraisal →PloS one
The application of large language models in meteorology graduate research: current status, impact, and prospects
With the rapid development of generative artificial intelligence, large language models (LLMs) have gradually integrated into various fields, demonstrating significant potential, particularly in meteorological research. This study explores the current application, advantages, challenges, and future development trends of LLMs in the scientific work of meteorology graduate students at NUIST. Through surveys and case analysis, the study finds that LLMs are primarily applied in literature review, data processing, code development, and academic writing in meteorological research. The results show that LLMs significantly enhance research efficiency, particularly in code development and literature translation, saving considerable time for graduate students. However, challenges remain in areas such as the accuracy of professional knowledge, creative inspiration, and interdisciplinary integration. The study also reveals concerns over data security, academic integrity, and model limitations when using LLMs. Future applications of LLMs in meteorology need further optimization in terms of professional knowledge accuracy and data processing capabilities. This paper provides both theoretical support and practical guidance for the responsible integration of LLMs into meteorological research and education.
27 Apr 2026
Read appraisal →PloS one
Cognitive load and pedagogical tension in multi-platform online learning: Evidence from Chinese higher education
The proliferation of digital tools has transformed higher education into a complex, multi-platform online learning environment. This study investigates the paradoxical effects of this multi-platform environment on the student experience within Chinese higher education. Through the qualitative and quantitative data from a cross-sectional survey of 8,616 university students, we analyzed platform usage patterns, perceived benefits and challenges. The findings reveal a significant paradox: while students express positive attitudes towards platforms' educational value, acknowledging benefits like enhanced self-directed learning and improved teacher-student interaction, they concurrently report substantial practical burdens. A serial mediation model confirmed that platform multiplicity increases extraneous cognitive load, which in turn elevates tool fatigue, ultimately leading to a more negative perception of the learning experience. This study identifies a core "pedagogical tension" between the intended benefits of educational technology and the lived reality of a fragmented, high-friction user experience that encourages instrumental engagement over deep learning. These findings underscore the urgent need for institutions and instructors to adopt a more strategic, integrated approach to educational technology to reduce fragmentation and prioritize a seamless student learning experience.
25 Apr 2026
Read appraisal →PloS one
Comparative analysis of text readability and writing styles in AI-generated vs. Human-written academic abstracts
Research article abstracts are vital in scientific publications for readers to assess a study's significance. The increasing use of AI tools, such as Kimi, ChatGPT and DeepSeek, to generate abstracts raises concerns about their readability and writing styles compared to human-written ones. The study aims to compare the differences in text readability and writing styles between human-written against AI-generated abstracts. A total of 150 abstracts of high-impact journal articles in the field of linguistics and computer science, 75 from each discipline, and another 150 AI-generated abstracts from the same corpus of articles served as the source texts for analysis. The Readability Scoring System, a computational tool, yielded readability and writing style metrics, while expert evaluation was performed to assess the quality of AI-generated academic abstracts. The quantitative data generated were analysed using SPSS 27 with non-parametric statistical methods. Key findings revealed: (1) AI-generated abstracts exhibited significantly lower readability across eight metrics, indicating greater complexity and lower readability; (2) Discipline-specific analysis showed five differing metrics in linguistics and eight in computer science; (3) Interdisciplinary comparisons revealed non-significant differences across nine readability metrics, highlighting AI's potential to mimic natural writing. However, it still faces challenges in generating lexically diverse content. These results underscored the current limitations of AI in generating readable and human-like abstracts, especially in technical fields.
10 Apr 2026
Read appraisal →