The role of large language models in the writing of surgical reviews: fact or fantasy?
Clinical Snapshot
PICO Framework
| P — Population | Short scientific review articles on a liver surgery theme, generated by ten AI-based text tools (five chatbots, five content creators) or written by four surgeons in training |
| I — Intervention | AI-generated surgical review texts (large language models and content creation tools) |
| C — Comparator | Human-written surgical review texts authored by surgeons in training |
| O — Outcomes | Formal and content quality scores assessed by experienced liver surgeons; reference accuracy (PubMed listing, citation correctness); readability (Flesch Reading Ease score); plagiarism resemblance (PlagScan); and ability of expert reviewers to correctly identify AI versus human authorship |
Bottom Line
This small comparative study from Hannover Medical School evaluated ten AI text generators against four surgical trainees producing short liver surgery reviews, assessed by three blinded expert surgeons. The headline finding — that experienced surgeons were misled into accepting AI text as human in approximately one in four cases — is attention-grabbing but must be interpreted with considerable caution given the study's substantial methodological limitations. The sample of 14 texts is far too small to support robust conclusions, no inferential statistics or confidence intervals are reported, and the human comparator group (surgeons in training) does not represent typical published authorship. The most clinically actionable finding is the reference quality gap: only 46% of AI-generated references were PubMed-listed, versus 100% for human authors, with only one AI tool achieving fully correct citation formatting. This hallucination risk is well-documented and represents the most immediate practical concern for editors, peer reviewers, and authors using AI assistance. The study's findings are consistent with the broader literature on LLM limitations but add limited new quantitative evidence. Surgical journals and training programs should mandate reference verification for all AI-assisted manuscripts. The rapidly evolving AI landscape means these specific findings may already be partially outdated.
Key Findings
P Value: Not reported
Effect Size: Reference quality: PubMed-listed references 100% (human) vs 46% (AI); only one AI-generated text achieved 100% formally correct citations. Readability: Flesch Reading Ease 26.2±5.1 (human) vs 17.7±6.1 (AI). Word count: 1054 (human) vs 874 (AI)
Primary Outcome: Expert correct identification of AI versus human authorship: 78.57% overall accuracy, with experienced surgeons misidentifying AI-generated text as human in 26.7% of cases
Nnt Or Sensitivity: Sensitivity of expert detection of AI authorship: approximately 73.3% (AI texts correctly identified as AI); specificity not formally calculable from reported data. Plagiarism resemblance: 6.4%±1.3 (human) vs 7.6%±4.5 (AI) — difference not statistically tested
Confidence Interval: Not reported
Clinical Application
The study's findings are directly applicable to editorial and peer review workflows. The reference fabrication finding (only 46% of AI-generated references PubMed-listed) has immediate practical implications: manual verification of all citations in AI-assisted manuscripts is essential before submission or acceptance. Readability scoring tools (Flesch Reading Ease) are freely available and could be incorporated into editorial screening. PlagScan and equivalent tools are already used by most surgical journals. In the Australian context, the findings are relevant to RACS (Royal Australasian College of Surgeons) training programs and academic surgical departments. The Medical Journal of Australia and ANZ Journal of Surgery, along with international journals publishing Australian surgical research, are increasingly developing AI disclosure policies. The Australian Code for the Responsible Conduct of Research (NHMRC/ARC, 2018) requires accurate attribution and reference integrity — the 54% rate of non-PubMed-listed AI references would constitute a significant compliance risk under this framework. The TGA and PBS are not directly relevant to this methodological study. RACGP and specialist college curricula may benefit from incorporating AI literacy and critical appraisal of AI-generated content into training frameworks. Surgical trainees, academic surgeons, journal editors, and peer reviewers engaged in the production or evaluation of surgical scientific literature, particularly short review articles
Abstract
BACKGROUND: Amidst the current enthusiasm concerning artificial intelligence and its possible application in the composition of different kinds of scientific and non-scientific written documents, we evaluated the usage of artificial intelligence for writing surgical short reviews. METHODS: In order to assess the formal and content quality of AI-generated texts compared to human written texts, ten AI-based text generators (five chatbots and five content creators) and four surgeons in training received the same prompt for a short scientific article on a liver surgery theme. All texts were anonymized and subsequently evaluated by three experienced liver surgeons based on a pre-defined scoring scheme, as well as for quality of references and readability according to readability indices. Furthermore, all texts were tested for plagiarism using PlagScan. RESULTS: Overall percentage of correct assessment for AI/non-AI generation by experienced surgeons lay at 78.57%. Human written text had a mean word count of 1054 versus 874 in AI-generated text, with a higher mean Flesh Reading Ease Score (FRE, 26.2 ± 5.1 versus 17.7 ± 6.1). References were PubMed-listed in 100% for human versus 46% for AI-generated text, with only one non-human text reaching 100% formally correct citation of references. PlagScan found 6.4%±1.3 mean resemblance to existing texts for human versus 7.6%±4.5 for AI-generated text. DISCUSSION: Overall, AI could already mislead experienced scientific surgeons in 26.7% of cases into believing it to be human. However, formal requirements, especially considering referencing, are still in great need of improvement with only one of AI-generated articles fulfilling our quality requirements.
References
- 1.DeTemple, D. E., Störzer, S., Arbabzadah, S., Riddermann, A., Gronau, F., Timrott, K., Bektas, H., & Kleine, M. (2026). The role of large language models in the writing of surgical reviews: fact or fantasy? Langenbeck's Archives of Surgery. https://doi.org/10.2196/67114
Related Research
Minimally invasive therapy & allied technologies : MITAT : official journal of the Society for Minimally Invasive Therapy
Ventral hernia repair in emergency settings. A machine learning model to predict post-operative complications.
3 Aug 2026
Journal of pediatric surgery
Interpretable deep learning model for pediatric strangulated small bowel obstruction on CT: A multicenter study
3 Aug 2026
European journal of radiology
A CT-based deep learning model to differentiate between benign and malignant adrenal lesions
3 Aug 2026
This content is for educational purposes for healthcare professionals only and does not constitute clinical advice. Clinical decisions should be based on individual patient assessment, current guidelines, and appropriate specialist consultation. Editorial Standards · Privacy Policy · Terms of Service