PRIME: Peer Review Improvement for Minimizing Bias in Evaluation

Peer review is at the core of the scientific process, where manuscripts are typically written up and submitted for publication once the results are known and the study completed. Decisions of authors, reviewers, and editors at each stage of the review process have been shown to favour positive results, erroneously conflating positive results with scientific merit. This publication bias is pervasive and corrosive. It skews the evidence base, undermines trust in science, and distorts policy decisions. We propose to address this by evaluating an alternative model of scholarly peer review. We have proposed a two-stage peer review process, results-free peer review (RFPR), where manuscripts are initially reviewed, and a decision to publish is made, based on their introduction and methods without the results. In the second stage, the results and discussion are reviewed for revisions only. This approach aims to reduce bias by focusing decisions of whether to publish on the study's scientific merit de-coupled from the results, incentivising authors to publish high-quality null studies. Working in partnership with diverse publishers (commercial, non-profit, learned society), we aim to test the hypothesis that RFPR will increase methodological quality and null studies in the published literature. A further aim is to investigate the feasibility and acceptance of RFPR among authors, reviewers, editors, and journals, to develop actionable policy recommendations. We will achieve these aims through the following objectives: In partnership with several leading academic journals, we will conduct a randomized controlled trial (RCT) to determine the effectiveness of RFPR as compared to standard review for reducing publication bias. We will also assess how acceptable it is to reviewers, editors, and authors, and how easy/hard it is for journals to implement. In a controlled experiment, we will investigate the cognitive mechanisms underlying biased decision-making during peer review, manipulating the presentation of methods and results and tracking the reviewer’s attention via their eye movements. We will generate actionable policy recommendations and instructional resources engaging with publishers, editors, reviewers, and authors. The PRIME project is international in its ambition; peer review is universal. This work would provide much-needed direct causal evidence of whether RFPR can reduce publication bias in practice. It will directly inform journal publication policies. If our hypotheses are supported, it could pave the way for standard review to be replaced with RFPR review. If not, then we can rule out RFPR as an easy solution to publication bias and use the uniquely rich datasets the PRIME project will provide to unpick why. Basing publication decisions on the quality of the scientific methods aligns the incentives for the researchers’ career advancement with those for rigorous research, getting at the very heart of the reproducibility problem.