Towards Trustworthy AI: Benchmarking, Validation, and Human Expertise

The aim of my proposed fellowship, as well as my research generally, is to show how social scientists can contribute to the development of trustworthy and hence successful AI technologies, that is systems that actually work for their users. My research will focus on evaluating and extending current AI benchmarking practices and frameworks for assessing the validity, reliability and value of new algorithms in specific contexts of use. Building on insights from my PhD fieldwork, this fellowship will extend my collaboration with the Big Hypotheses project to develop a new framework for AI users—benchmarks for use—that will help them concretely establish when and under what circumstances they can dependably trust these technologies for specific tasks, whether in research, service delivery or other domains. Through a new collaboration with Prof. Jonas Ivarsson in Gothenburg, alongside ongoing work with my mentors Prof. Michael Mair and Prof. Simon Maskell, I will publish in leading journals and present at conferences that bring together AI developers and academics. I will also develop AI training for UK social scientists, using the Fellowship to build connections across user groups and develop impact from my research. Through detailed ethnographic observations in commercial and academic settings, my PhD research provides concrete empirical evidence about the necessity of human interventions and domain expertise in the successful deployment of AI technologies. My findings demonstrate precisely how benchmarks, i.e., locally deployed standards used as points of reference to assess algorithmic performance, function as practical tools for negotiation, clarification, and accountability in day-to-day AI development, directly informing more transparent and realistic assessments of AI systems. Working alongside AI developers as a colleague and actively contributing to their everyday operations during my PhD, I gained a deeper understanding at this practice level. More specifically, my study showed the following: The seemingly "intelligent" appearances of AI systems rely on continuous human interventions, as data scientists develop domain expertise to adapt systems to local circumstances and user needs. Nuanced domain knowledge is central to developing trustworthy AI systems at the technical level. This is because domain knowledge, i.e., knowledge of the practical domain in which the system is to play a role, helps data scientists anticipate pitfalls, address anomalies, and ensure a model’s performance and outputs are reliable before informing decisions. In this process, benchmarks and their metrics serve as accuracy measures and negotiation devices, enabling AI practitioners and stakeholders to collaboratively interpret, question, and compare results so model inferences remain interpretable, accountable, and actionable within local organisational contexts defining the domain. They also enable model refinement for better performance in projected use contexts. Good benchmarking grounds trustworthiness at the system application level. In this way, my research highlights the need for a realistic, empirically informed approach to AI that foregrounds social and technical practices rather than algorithms alone. Algorithms are end products, and we misunderstand them without examining the practices that generate them, the goals they embody, and their limits—well understood within data science but less so outside it. Benchmarking is central to this. It involves standardised tests, datasets, and performance metrics to assess AI models. These shared frameworks help practitioners measure progress, track development, and identify weaknesses. I aim to show how internal benchmarking practices can be expanded to build trust in AI systems among broader user groups and the public.