Most researchers now use generative AI to help write scientific papers and grant proposals, but no one has systematically measured how this is changing the language of science itself. This matters because the same tools that evaluate scientific work—peer review systems, funding decisions, and publication acceptance—are increasingly assisted by AI. Those AI tools were trained on older, human-written text, so they may perform poorly on AI-assisted writing without anyone realising it. The project builds a large dataset of disclosed AI-assisted publications from top STEM journals to track actual language changes. It also tests whether a machine model that predicts a paper’s replicability still works correctly on AI-generated text. If successful, the research will reveal whether AI-assisted writing is altering scientific language in ways that undermine automated evaluation tools. It will also identify why disclosure rates remain low despite journal mandates, and propose strategies to improve transparency. The findings will be shared with journal editors and funders through policy briefs and workshops, helping ensure that the infrastructure of peer review and grant allocation remains reliable as writing practices evolve.
View original technical description
The majority of researchers now report using generative AI (GenAI) to help write scientific texts (e.g., research papers, grant proposals). Yet, little empirical investigation exists on how AI-assisted writing is changing the language of science and how that affects peer review. At the same time, scientific texts are increasingly evaluated not just by human reviewers but also by artificial intelligence (AI). Funders, journals, and conferences are piloting AI-assisted tools to inform decisions on funding allocation or publication acceptance. These tools were trained with older, human-authored text, raising concerns about their performance on GenAI-assisted writing. Through this fellowship, I aim to understand how scientific writing is evolving in the age of GenAI and how those changes may affect AI-assisted peer review. I will do so by building a large dataset of academic publications in the top 100 journals (ranked by h-index and predominantly in STEM fields) where authors have disclosed the use of GenAI tools to help write papers. Analysing these real-world examples will reveal how language has changed (or not) since the rise of GenAI. The project also includes a case study: testing the behaviour of a machine model that predicts a paper’s replicability when applied to GenAI-assisted text. This will serve as a broader example for evaluating AI-assisted peer review tools in this new context. Ethical and policy considerations are central to the project. Despite journals mandating GenAI disclosure, actual reporting remains low. Existing discussions focus on disclosure as a matter of integrity, transparency, and accountability. My work will broaden the conversation by showing that truthful reporting is also critical for collecting reliable data for metascientific research. In response to the low disclosure rate, the project will examine how different top journals enforce these guidelines and identify strategies to increase transparency. Findings will be shared in a policy brief, a webinar for journal editors, and knowledge exchange workshops.
Plain English summaries and category classifications on this site are generated by AI and may not perfectly reflect the original research.
Is something wrong? Let us know