Harnessing Synthetic Data Fidelity for Assured Perception of Autonomous Vehicles

As autonomous vehicles (AVs) transition to public roads, ensuring their safety is paramount. Scenario-based testing, gaining recognition for AV safety, employs synthetic data for virtual testing. Significant investment in advanced simulators and Generative AI aims to enhance fidelity, creating realistic virtual environments and sensors mimicking real-world driving conditions. Despite this, synthetic data generation is still incapable of achieving outright fidelity. This leads to a pivot question: What level of fidelity is required for synthetic data to be deemed sufficient for AV safety? This project, driven by the PI's vision of "building trust in an AI-powered world," aligns with the UK's ambition for AV technology and Safe AI. It advances academic knowledge in safety engineering and Safe AI, fostering interdisciplinary collaboration. Economically, it speeds up AV safety solutions, aids in industry product development, and influences AV safety standards/policies. Societally, it ensures safe AV deployment, enhancing public trust. This project focuses on examining instance-level fidelity of a synthetic data-point (e.g., an image). Here, fidelity is defined as the ability to precisely replicate the (unsafe) behaviours of AVs as if they were operating in real-world. Such fidelity should be tailored for the specific System-Under-Testing (SUT); e.g., an emergency braking system needs less detail about obstacles than an object detection system which must classify and locate objects. The proposal hypothesises: fidelity of synthetic data pertains to how the SUT (i.e., Machine Learning (ML) models enabling AV perception) processes the data and for what purposes. A deeper understanding of how the ML-model's behaviours (ranging from its predictions, features learnt, and rationales behind predictions) differ, when exposed to paired real-world and synthetic data, can be measured for quantifying fidelity. Through verification using such fidelity-aware data to establish sufficient confidence in AV safety, we can then answer the fidelity sufficiency question. The aim is to develop a safety framework focused on quantifying synthetic data fidelity and integrating the fidelity information into scenario-based virtual testing. The programme comprises 4 work-packages (WPs) over 2.5 years. WP1 will specify an understanding of the fidelity challenge in virtual testing for AVs, creating a formal specification for synthetic data generation process. A spectrum of fidelity definitions with diverse metrics will be formulated, based on variations in ML behaviours when fed with real-world and synthetic data. WP2 will establish a SUT-specific AI-predictor that quantifies fidelity for a given synthetic data-point. It first creates a training dataset by evaluating fidelity metrics (from WP1) for a set of scenarios with "paired" real-word and synthetic representations; then it employs state-of-the-art Safe AI techniques to train and validate the predictor, ensuring its reliability. WP3 will develop novel ML debugging and acceptance testing methods using fidelity-aware synthetic data (quantified by WP2), considering which and how many scenarios to test. WP4 will integrate, validate, and communicate the methods developed in the preceding WPs as a holistic approach. Both safety argument templates and case studies will be developed, to support requirements derived from WP1 based on evidence from WP3. WMG mentors and partners will advise on thematic aspects and support corresponding WPs, offering feedback in quarterly meetings, workshops, and advisory boards. The team, including two funded PhD students, will have full access to WMG's simulation and computing resources, with WMG engineers' support. The project will utilise £140K+€88K in-kind-contributions from partners including Siemens, Wayve, Denso, Adelard, University-of-Liverpool, City-University-of-London, Technical-University-of-Munich and Fraunhofer-IKS.