Transforming Evidence Synthesis: AI and the (R)evolution of the Evidence Ecosystem

This project will be the first large-scale empirical study of an academic field where the idea of an AI researcher is not a distant possibility but a present reality - namely, evidence synthesis. Evidence synthesis is a process of systematically gathering, evaluating, and integrating information from multiple sources, for example, through systematic reviews. Evidence synthesis is one of the building blocks of our broader scientific system, ensuring the provision of accurate and reliable accounts of existing knowledge. It also epitomises the key challenges of contemporary science – dealing with information overload and the ongoing demand for relevance and impact. AI promises to be a ‘technological fix’ to these challenges – from speeding up the review process by up to 90% to completely automating the synthesis via AI-driven living evidence synthesis. And yet – despite these potentially transformative changes to the science ecosystem - there is virtually no scholarship exploring the epistemic, social and political implications of AI on evidence synthesis. The proposed research will investigate the implications of AI on the practice of evidence synthesis and the broader research ecosystem - including evolving methods and routines, skills and career trajectories, professional norms and standards, and academic institutions and cultures. It will capture this complex and entangled agency through a novel analytical framework exploring the impacts of AI across the three levels: micro-level practices of knowledge production through meso-level organisational standards and careers, all the way to the macro-level of the science system. The key innovation of this project is its comparative design, exploring two types of settings in evidence synthesis: 1.) Settings where practices have been ‘augmented’ by introducing AI tools to enhance some elements of the review process (such as RobotReviewer or DistillerSR); 2.) Settings where human practices are being ‘replaced’ by the automated evidence infrastructure for real-time evidence synthesis. As such, this study will generate novel insights and new theoretical language to capture the profound implications of AI for the entire evidence ecosystem, thus paving the way to transformative shifts in our collective understanding of what constitutes ‘good evidence’. The study will employ a variety of methodological approaches aimed at capturing these complex dynamics, including a mix of 1.) semi-structured interviews with academics, policymakers and practitioners working in the area of evidence synthesis; 2.) wide-ranging document analysis to explore how AI is framed in the academic publications’ methods sections, funding calls, policies, guidelines and strategies (with a particular focus on ethical and responsible uses of AI); 3.) co-creative workshops with academics, practitioners and policymakers to explore emerging AI trends, experiments and challenges in evidence synthesis and develop actionable solutions to its key challenges. The project is strategically positioned to generate a substantial impact on both policy and practice. A window of opportunity is opening for this exploration, marked by growing policy attention and increased investment in automated evidence synthesis. Over the past year, the three key funders - Wellcome Trust, ESRC, and NIHR - committed over £60 million to AI evidence synthesis infrastructure, with organisations like Cochrane Collaboration, the World Bank, and UNDP developing their own AI-driven evidence synthesis tools. The field is undergoing a pivotal moment of collective sense-making, and this study will help ensure the AI (r)evolution is both equitable and responsible.