Study: Millions of Scientific Papers Have ‘Fingerprints’ of AI in Their Text

Researchers have discovered that the emergence of AI large language models (LLMs) has led to a detectable increase in specific word choices within academic literature, suggesting that AI-generated content is quietly infiltrating peer-reviewed scientific publications.

Phys.org reports that a recent study conducted by a team of U.S. and German researchers has revealed that the rise of AI systems like ChatGPT and Google Gemini has had a significant impact on the language used in academic writing. By analyzing more than 15 million biomedical abstracts on PubMed, the researchers found that since the emergence of LLMs, there has been a corresponding increase in the frequency of certain stylistic word choices within journal articles. The findings, published in the open-access journal Science Advances, suggest that at least 13.5 percent of papers published in 2024 were written with some degree of LLM assistance.

The widespread use of AI-generated content has raised concerns about the accuracy and integrity of research in the academic community. Previous attempts to quantify the rise of AI in academic writing were limited by their reliance on comparing sets of human- and LLM-generated text, which could introduce biases. To avoid these limitations, the authors of this latest study took a different approach, examining changes in the excess use of certain words before and after the public release of ChatGPT to uncover any telltale trends.

The researchers applied a before-and-after approach to analyze patterns of excess word use prior to and following the emergence of LLMs. They discovered a significant shift away from the excess use of “content words” to an excess use of “stylistic and flowery” word choices, such as “showcasing,” “pivotal,” and “grappling,” after the release of LLMs.

By manually assigning parts of speech to each excess word, the authors determined that before 2024, 79.2 percent of excess word choices were nouns. However, during 2024, there was a clearly identifiable shift, with 66 percent of excess word choices being verbs and 14 percent being adjectives. This change in language patterns strongly suggests the influence of AI-generated content in academic writing.

The team also identified notable differences in LLM usage between research fields, countries, and venues. This finding highlights the varying levels of AI adoption and its potential impact on different areas of study and geographic regions.