Peer review in the age of artificial intelligence: a comparative study of human and AI-generated review reports

Clinical Snapshot

Key Findings

• P Value: Statistical significance achieved using Mann-Whitney U and Fisher's exact tests (specific p-values not provided)

• Effect Size: Shannon Index confirmed higher thematic diversity in human reviews (specific values not reported)

• Primary Outcome: Human reviewers provided more detailed, diverse comments addressing deeper issues like interpretation and originality

• Nnt Or Sensitivity: AI slightly outperformed in format-related domains but lacked critical judgment capabilities

• Confidence Interval: Not reported

Clinical Application

AI tools could be implemented for initial screening but require human oversight for meaningful review Relevant to Australian medical journals and research institutions. Could inform NHMRC and university research integrity policies regarding AI use in peer review Medical journal editors, peer reviewers, and academic institutions involved in scholarly publishing

Abstract

BACKGROUND: Peer review is central to maintaining scientific quality and helps editors make decisions. However, the volume of scientific publications continues to rise, placing pressure on the peer review system. With the rise of generative AI, its role in supporting peer review is gaining attention. This study aims to compare human-written and AI-generated peer review reports. METHODS: We analysed 398 peer review reports linked to 119 research articles published in BMJ Open in 2024. Publicly available reports and manuscripts were included. Editorials, corrections, and protocols were excluded. AI reports were generated using ChatGPT. All reports were anonymized and assessed by two independent reviewers. We conducted a hybrid thematic analysis. Frequencies of themes were calculated and compared by reviewer type. For quantitative comparison, we used the Mann-Whitney U test to assess differences in review quality scores and Fisher's exact test to compare the distribution of themes. All analyses were conducted using R software. RESULTS: Human reviewers gave more detailed and diverse comments. They addressed deeper issues like interpretation, originality, and applicability. AI reviews covered more sections but focused on routine or structural elements. AI outperformed slightly in format-related domains. Co-occurrence analysis showed human reviews linked diverse themes, while AI comments were structurally clustered. Shannon Index confirmed that human reviews were more thematically diverse. CONCLUSIONS: AI can support peer review by screening for basic errors. However, it lacks insight, critical judgment, and contextual awareness. Human input remains essential for meaningful review. Review-specific AI tools that preserve confidentiality are needed for future integration.

References

1. 1.Rajakumar, H. K., Sankaran, K. A., Ashok, M. P., & Rachoori, S. (2026). Peer review in the age of artificial intelligence: a comparative study of human and AI-generated review reports. Postgraduate Medical Journal. https://doi.org/10.1093/postmj/qgag005