Britain’s major scientific facilities are drowning in data they cannot analyse fast enough. This project builds prototype data pipelines and a repository to make experimental data from X-ray diffraction and electron microscopy “AI ready”—structured, labelled, and trustworthy enough for machine-learning tools to use. The problem is straightforward: instruments at places like Diamond Light Source and the National Electron Diffraction Facility generate exponentially more data than researchers can interpret. Without better data infrastructure, the bottleneck shifts from collecting data to making sense of it. This project tackles that gap by pairing data engineers with domain experts to build pipelines that feed curated datasets into AI services. If successful, the work could accelerate materials discovery and drug development by letting AI screen samples before expensive experiments begin. It also enables autonomous laboratories—machines that steer their own experiments based on real-time analysis. The project tests its approach with a service called CrystaLLM, which predicts crystal structures from composition and diffraction data, a capability that could speed up everything from battery design to pharmaceutical formulation. The core insight—how to generate trusted AI-ready data—applies across all of science.
View original technical description
Although the volume and quality of experimental data from physical sciences is rising exponentially, methods for analysis and interpretation of resulting data are not keeping pace with this growth. At the UK’s major analytical science facilities, the National Research Facilities and for strategic and core equipment, high-throughput synthesis demands timely characterisation and analysis of samples. Particularly important is the ability to rapidly identify those samples that are most likely to give rise to useful data in an experiment. AI can play a key role in such ‘pre-experiment’ screening and subsequent experiment steering. A brief and low-cost initial data collection can be used to plan and steer the more resource consuming workflow of the main experiment that follows. Such high throughput screening can also enable autonomous laboratories by providing the feedback loop that drives an experiment towards a particular goal. This project brings together the Physical Sciences Data Infrastructure (PSDI), UCL, the Ada Lovelace Centre (ALC), central facilities and NRFs. A PSDI goal is to develop and support whole lifecycle data pipelines that are ready for AI, while the ALC is concerned with providing tools and services to enable more efficient use of (experimental) central facilities and their data products. We will leverage the work of both PSDI and ALC by pairing this project with a related project led by ALC in the same call. This PSDI project will construct example data pipelines and a prototype ‘AI ready’ data repository, whereas the ALC project will work on specific applications in central facilities and in home laboratories that use the PSDI developed infrastructure. The approach will be driven and validated across a range of use cases producing well curated datasets with well-defined outcomes from diffraction and microscopy experiments/facilities. The use cases for building pipelines and the repository will be centred on: Serial small molecule crystallography (Diamond Light Source beamline I19) Materials powder diffraction (Diamond Light Source beamline I11) Project M crystallisation and diffraction outcomes dataset (https://www.diamond.ac.uk/ProjectM/) Single crystal electron diffraction (National Electron Diffraction Facility) Autonomous TEM control to enable automated imaging, driving image segmentation and classification (UCL TEM facility) Furthermore, with the view of testing the AI ready nature of the repository contents, a service will be deployed for the AI-driven (CrystaLLM) prediction of crystal structure from composition and experimental X-ray diffraction data (UCL). This project draws together domain experts necessary for the categorisation and collection of highly curated and trustworthy datasets, with data engineers and research software engineers, to build data pipelines that feed the repositories and associated services hosted by PSDI. The project therefore addresses a problem that is common across many sciences: what does it mean to generate trusted ‘AI ready’ data, and how can data be assembled into collections that are useful as training data for AI tools and systems that accelerate how science is done?
Plain English summaries and category classifications on this site are generated by AI and may not perfectly reflect the original research.
Is something wrong? Let us know