Every year, UK government departments commission hundreds of quantitative impact evaluations—studies that measure whether a policy actually worked—and the results help decide how to spend billions of pounds of public money. But recent metascience research shows that in academia, the same data and hypothesis can produce wildly different results depending on which analysis choices a researcher makes. No one has yet tested whether the same problem plagues government evaluations. This project will do exactly that. A team of academics and civil servants will recruit dozens of independent research teams to reanalyse existing government evaluations using different but defensible methods. They will also survey a larger pool of researchers about their analysis choices and run a “multiverse” simulation to map every plausible analytical path for a given evaluation. If the results show that government evaluations are as analysis-dependent as academic studies, then policymakers cannot trust any single evaluation’s conclusions. The team will produce concrete guidance for the Magenta Book—the UK government’s official evaluation handbook—to help evaluators design studies whose results hold up regardless of analytical choices. That could directly improve the integrity of spending decisions affecting schools, hospitals, transport, and welfare.
View original technical description
Quantitative impact evaluations – studies which measure the effects of policy – play a central role in government decision-making. In the UK, government departments conduct and commission hundreds of evaluations each year, and impact evaluations are highly integrated into the policymaking process, with many billions of pounds of public funds resting on the results of such studies. It is therefore concerning that, as indicated by recent metascience research, the results of many quantitative analyses in academia appear to be very sensitive to analysis decisions made by researchers. A researcher conducting a quantitative analysis must make a vast array of decisions, such as which data to use, how to measure outcomes, which model to estimate, and so on. As a result, when different academic research teams test the same hypotheses, using the same datasets, they often produce very different results. The potential consequences of this problem for impact evaluations of government policy are profound, as it implies that policymakers cannot be sure that the information taken from any single analysis is correct. However, while this issue has been documented in several academic disciplines, work examining whether the problem of analysis-dependent results affects quantitative evaluations of government interventions is virtually non-existent. In the absence of studies which assess the scale of this problem for policy-relevant research, policymakers cannot know whether the impact evaluations upon which they base key spending decisions are robust. We address this issue by uniting a team of academics with expertise in quantitative impact evaluation methods and UK civil servants who have responsibility for improving the quality of evaluation practice in government. We aim to quantify the degree to which the results of impact evaluations vary when researchers use different, but defensible, analysis choices. Our research is divided in three parts. First, will employ a “many-teams” analysis, in which we will recruit a large number of independent research teams to reanalyse data from existing impact evaluations. Second, we will use a survey-based analysis to solicit analysis choices relevant to existing evaluations from a larger set of independent researchers. Third, we will conduct a “multiverse” analysis to simulate the full universe of possible analysis choices for a given evaluation and investigate how these choices affect estimates of policy impact. Our project will generate significant new evidence on the robustness of quantitative research findings which we will use to directly inform evaluation practice across government. Our research will reveal whether government evaluations, like academic studies, are analysis-dependent, and what the range of possible estimates of policy impact is. We will also investigate whether analysis-dependence is ameliorated by using different research designs, an important factor ignored in previous literature. Together, this research will substantially increase the available evidence on the integrity and rigour of existing quantitative research findings. This research will also enable us to provide tangible recommendations for evaluators in government and the broader evaluation community. We will translate our findings into guidance on how to mitigate analysis-dependence, which will be incorporated into the Magenta Book, the key resource for evaluators across UK government. This guidance will therefore directly contribute to the integrity of research findings used to inform policy and spending decisions.
Plain English summaries and category classifications on this site are generated by AI and may not perfectly reflect the original research.
Is something wrong? Let us know