The UK’s biggest social surveys—like Understanding Society and the British Household Panel Survey—are sitting on decades of detailed questionnaire data that remains difficult for researchers and AI systems to find, combine, or reuse. The problem is metadata: the labels, categories, and descriptions that explain what each survey question means and how the data was collected. Currently, much of this information is locked in inconsistent formats, scattered across institutions, or missing entirely. This project aims to extract that metadata from survey questionnaires, tag it with standard vocabularies used across social science and health research, and store it in a machine-readable knowledge graph. A second workstream will use this enriched metadata to automate decisions about privacy and data disclosure—tasks that today require slow, manual review. If successful, the work could transform how policymakers and analysts combine data from different surveys, across different decades, and even across different languages. It would also make these rich social science resources usable by AI tools, enabling faster, more trustworthy evidence for decisions on health, housing, employment, and public spending. The project builds directly on pilot work funded by UKRI/ESRC over the past year.
View original technical description
The UK’s rich collection of longitudinal population and cross-sectional studies form the backbone of empirical research in the social, economic and behavioural sciences, as well as in epidemiology and health research, providing the basis for evidence-based policy advice. However, the demands of the research community, encapsulated by the FAIR principles, for more timely and higher quality granular metadata is a significant challenge. This project focuses on addressing these barriers to uplift existing metadata resources and lay the basis for future data to be truly FAIR and usable by AI technologies. The project has four main objectives: Extraction of metadata from social survey questionnaires to increase both volume and interoperability of these data sources to enhance discovery. Knowledge enhancement through concept extraction and classification into standard vocabularies which are supported and used by the social science, health and statistical community. Development of pipeline to integrate outputs into a FAIR enabled semantic web (RDF) store / knowledge graph to store and make accessible the metadata generated for use by AI technologies, development of quality assurance tooling. Utilisation of these enhancement metadata to develop new approaches to tackle privacy and disclosure challenges to enable improved decision based and for this to be scaled through automation. These build on the pilot/development work supported by the UKRI/ESRC Future Data Services over the last 12 months. The structured data resources created can be leveraged to create new ways of utilising rich Social Science resources for discovery, harmonisation and comparability, removing previous barriers such as language and non-interoperable formats, providing: Standards-based provenance metadata (how the data was collected) Annotation to standard controlled vocabularies Automation of privacy and access rights How the proposal meets AI for Science Objectives The project would develop metadata extraction and metadata enhancement technologies which would be robust across the known potential data sources and make a significant contribution to the volume and availability of such resources for AI. The development of an RDF metadata store which was curated with community vocabularies would enable new approaches to social science data, not viable with current levels of fragmentation, over and above those already developed in this work. How the proposal meets DSIT Objectives Policy makers are increasingly demanding that analysis is provided in a more timely and comprehensible manner. At the same time, data is coming from more diverse sources, and needs to be combined across disciplines and organisations. We believe that enhanced metadata generated from these work streams, which is structurally and semantically coherent and made available to analysts alongside the data, is a critical part of assuring them that they are looking at data that is trustworthy and can be combined in a meaningful way. The workstream on disclosure is one example of how these metadata can be further utilised to move very challenging manual tasks to (initially semi-) automated decision based processes.
Plain English summaries and category classifications on this site are generated by AI and may not perfectly reflect the original research.
Is something wrong? Let us know