Comparative analysis of text readability and writing styles in AI-generated vs. Human-written academic abstracts

Clinical Snapshot

Key Findings

• P Value: Described as 'significant' but specific values not given

• Effect Size: Not specified in abstract

• Primary Outcome: AI-generated abstracts showed significantly lower readability across eight metrics

• Nnt Or Sensitivity: Five differing metrics in linguistics, eight in computer science disciplines

• Confidence Interval: Not provided

Clinical Application

Findings suggest caution needed when using current AI tools for academic abstract generation Limited direct relevance to Australian clinical practice; may inform academic medical writing and journal editorial policies in Australian universities Academic researchers and journal editors in technical fields, particularly those using AI writing assistance

Abstract

Research article abstracts are vital in scientific publications for readers to assess a study's significance. The increasing use of AI tools, such as Kimi, ChatGPT and DeepSeek, to generate abstracts raises concerns about their readability and writing styles compared to human-written ones. The study aims to compare the differences in text readability and writing styles between human-written against AI-generated abstracts. A total of 150 abstracts of high-impact journal articles in the field of linguistics and computer science, 75 from each discipline, and another 150 AI-generated abstracts from the same corpus of articles served as the source texts for analysis. The Readability Scoring System, a computational tool, yielded readability and writing style metrics, while expert evaluation was performed to assess the quality of AI-generated academic abstracts. The quantitative data generated were analysed using SPSS 27 with non-parametric statistical methods. Key findings revealed: (1) AI-generated abstracts exhibited significantly lower readability across eight metrics, indicating greater complexity and lower readability; (2) Discipline-specific analysis showed five differing metrics in linguistics and eight in computer science; (3) Interdisciplinary comparisons revealed non-significant differences across nine readability metrics, highlighting AI's potential to mimic natural writing. However, it still faces challenges in generating lexically diverse content. These results underscored the current limitations of AI in generating readable and human-like abstracts, especially in technical fields.

References

1. 1.Zou, Y., Kuek, F., Ng, K. H., & Cheng, X. (2024). Comparative analysis of text readability and writing styles in AI-generated vs. Human-written academic abstracts. PLoS One. https://doi.org/10.1016/j.pragma.2024.01.003