Comparative judgement works because humans are surprisingly good at picking which of two things has more of a given quality—but the maths behind that reliability is surprisingly weak, and this project aims to fix it. Social scientists use comparative judgement to measure things that resist standard tests, such as a student’s depth of understanding or the prevalence of abuse. Yet the statistical models they rely on are underdeveloped, making results less trustworthy than they could be. At the same time, statisticians who could improve those models often do not know how social scientists actually use them. This project bridges that gap. If successful, the work will produce better statistical tools, an online hub with datasets and code, and a shared research agenda of around 50 questions. The immediate impact is on the quality of social-science research itself—making studies that use comparative judgement more reliable. That matters because those studies inform decisions in education, public policy, and other fields where measurement is difficult. The project is not aimed at a direct consumer product; it is fundamental methodological science that strengthens the foundations of how researchers measure things that cannot be easily counted.
View original technical description
Comparative judgement (CJ) is a powerful method used extensively across the social sciences to tackle difficult measurement problems that evade other approaches. It has been applied to measure wide ranging phenomena, including students’ conceptual understanding, the prevalence of abuse, and mathematical beauty. However, the mathematical foundations of CJ are underdeveloped, and this limits the reliability and trustworthiness of CJ studies. This project will deliver advances in the mathematical models used to analyse CJ data and stimulate cross-disciplinary research collaborations between statisticians who research the mathematics of CJ, and social scientists who use CJ as a research tool. Other measurement methods, such as those based on multiple-choice questions or rubrics, tend to be unreliable because they rely on subjective responses. The CJ method, in which participants are presented with pairs of objects and asked to decide which has more of a given quality, is highly reliable because humans are largely consistent when comparing two things to one another. Social scientists commonly use CJ in a mechanistic way, following instructions to create measurement scales without a grasp of the underlying mathematics. Statisticians commonly develop and evaluate models for analysing CJ datasets without awareness of varied and expanding social science applications. The aim of this project is to develop and test improved statistical models in order to foster collaboration between statisticians and social scientists, leading to a vibrant cross-disciplinary sub-field of CJ research. We will achieve this through three specific objectives. Objective 1 will create an online research hub to support cross-disciplinary CJ research. The hub will include archived CJ datasets, tools for generating datasets, code and packages for analysing datasets, a bibliography of relevant scientific papers, guidance for researchers, and a discussion board. These extensive resources, and channels for ongoing communication, will stimulate and sustain cross-disciplinary research as described below. Objective 2 will create an agenda of large-scale CJ research challenges through collaboration between 25 CJ experts. The agenda will contain around 50 research questions that will provide structure for the online hub to help drive and sustain a vibrant research community. Objective 3 will conduct novel statistical research to address two large-scale research challenges: (i) to enhance our understanding of how fitting different statistical models to CJ data impacts findings across different research contexts; (ii) to address the paucity of methods available to researchers for evaluating how well models perform when used to analyse CJ data. To achieve this, Project Lead Ian Jones, a Professor of Mathematics Education at Loughborough University, and a world-leader in CJ methods, will ‘hop’ to the University of Birmingham’s School of Mathematics and enhance his knowledge of statistical research methodologies. There he will work closely with a group of statisticians who are at the forefront of developing pioneering CJ models. The results of Jones’ research at Birmingham will inform Objectives 1 and 2 and will substantially improve the quality and trustworthiness of CJ-based research. In summary, the project will deliver an online hub, an agenda of grand challenges, and new models and tools, thereby stimulating and sustaining a vibrant cross-disciplinary sub-field. This will improve the reliability and trustworthiness of CJ research across a wide range of academic and real-world applications.
Plain English summaries and category classifications on this site are generated by AI and may not perfectly reflect the original research.
Is something wrong? Let us know