Self-driving cars still struggle with rain, fog, and snow because their camera-based vision systems cannot see clearly in bad weather. This project, CORTEX, tackles that bottleneck. Current autonomous vehicles rely heavily on cameras and machine learning, but cameras fail in dirt, rain, ice, and fog. Radar works in all weather but lacks the resolution to classify objects. The researchers will combine advanced radar processing—using beamforming and MIMO techniques to create high-resolution "imaging radar"—with a new class of Transformer algorithms, previously used for speech recognition, to identify and segment objects in radar images. They will test this system on real-world driving data from both on-road and off-road conditions. If successful, the project could remove a major barrier to full autonomy: the inability to drive safely in all weather and on all road surfaces. That would push autonomous vehicles beyond controlled, mapped environments and into everyday use—affecting transport systems, logistics, and personal mobility. The work is applied engineering, not fundamental science, and its value depends on whether the radar-plus-Transformer approach can match or exceed camera-based classification in adverse conditions.
View original technical description
In 2016 many people were predicting fully autonomous cars within 5 years. After initial quick progress, their recent development has slowed significantly. The market is now adjusting to the new, slower rate of development. This has been seen in more measured predictions from manufacturers and experts, with some companies rolling back on autonomy plans and projects. Whilst machine learning and computer vision has enabled vehicles to reach near autonomy in controlled conditions, it is real world driving in all weather conditions that is proving difficult to achieve. Although there have been demonstrations of autonomy using a combination of fully mapped environments and data from large numbers of sensors, the environments in which the demonstration vehicles operate are highly constrained and therefore, do not represent some of the key environmental challenges that will be encountered by these vehicles in real-life driving scenarios. Several challenges have been highlighted in this CCAV call to facilitate full autonomy, however challenges on "autonomous vehicle features that provide real-world benefits to users to work as part of a wider transport system" cannot succeed if the system cannot robustly understand the environment in which the vehicle will drive, i.e. different weather conditions (sunny, dark, raining, foggy or snowing), or different road surfaces across the world. The project addresses the development of key enabling technologies and sensing techniques for autonomous driving in all-road and all-weather driving conditions. Radar systems remain relatively unaffected by adverse environmental conditions (dirt, rain, ice, snow, fog etc…), However, the radar does not have the same level of resolution as a camera. Often is not possible to classify an object based solely on its radar return signature. This is changing as manufacturers add more and more transmit and receive elements to their radar boards. This is improving the accuracy and resolution to create an 'Imaging Radar'. Cortex will use advanced radar processing and beamforming techniques to improve resolution and reduce the known issues e.g. MIMO high side lobes. The objects in the resulting radar images will then be classified and the complete scene will be segmented into different classes. The resulting identification and labeling of objects will not be affected by weather, in the same way a camera would. Autonomous driving based on machine learning, using CNN algorithms from the camera images is common. There is however a new class of 'Transformer' algorithm, previously applied to speech and simple waveform data, that is now being applied to images. This project will explore the Transformer Algorithms and their use in object identification and image segmentation. The project will take real world driving data from on road and off road and compare the classification from radar and camera images.
Plain English summaries and category classifications on this site are generated by AI and may not perfectly reflect the original research.
Is something wrong? Let us know