A research centre in Surrey is teaching machines to see and hear the world at the same time, combining audio and visual data to understand real-world scenes far better than either sense alone could. Most machine perception systems treat sound and vision as separate problems. A camera tracks movement; a microphone picks up noise. But humans navigate the world by fusing both senses—we hear a car approaching before we see it, or locate a speaker in a crowded room by matching voice to face. Current AI largely misses this integration. The Centre for Vision, Speech and Signal Processing (CVSSP) aims to close that gap by building a platform for fundamental research into multi-sensory machine perception. If successful, this work could transform how machines interact with the physical world. In healthcare, intelligent sensing systems could monitor patients at home, detecting falls or distress by combining audio cues with visual data. In entertainment, immersive production could use real-time scene understanding to blend virtual and real elements seamlessly. Retail, sports analysis, and assisted living for older adults are other direct targets. The platform also strengthens international collaborations with leading labs in the US, Asia, and Europe, training the next generation of researchers in specialist facilities that keep UK industry at the forefront of audio-visual technology.
View original technical description
The strategic objective of this platform grant is to underpin Audio-Visual Media Research within the Centre for Vision, Speech and Signal Processing (CVSSP) to pursue fundamental research combining internationally leading expertise in understanding of real-world audio and visual data, and to transfer this capability to impact new application domains. Our goal is to pioneer new technologies which impact directly on industry practice in healthcare, sports, retail, communication, entertainment and training. This builds on CVSSP's unique track-record of world-leading research in both audio and visual machine perception which has enabled ground-breaking technology exploited by UK industry. The strategic contribution and international standing of the centres research in audio and visual media has been recognised by EPSRC through two previous platform grant awards (2003-14) and two programme grant awards in 2013 and 2015. Platform Grant funding is requested to reinforce the critical mass of expertise and knowledge of specialist facilities required to contribute advance in both fundamental understanding and pioneering new technology. In particular this Platform Grant will catalyse advances in multi-sensory machine perception building on the Centre's unique strengths in audio and vision. Key experienced post-doctoral researchers have specialist knowledge and practical know-how, which is an important resource for training new researchers and for maintaining cutting edge research using state-of-the-art facilities. Strategically the Platform Grant will build on recent independent advances in audio and visual scene analysis to lead multi-sensory understanding and modelling of real-world scenes. Research advances will provide the foundation for UK industry to lead the development of technologies ranging from intelligent sensing for healthcare and assisted living to immersive entertainment production. Platform Grant funding will also strengthen CVSSP's international collaboration with leading groups world-wide through extended research secondments US (Washington, USC), Asia (Tsinghua, Tianjin, Kyoto, Tokyo, KAUST) and Europe (INRIA, MPI, Fraunhofer, ETH, EPFL, KTH, CTU, UPF).
Adrian Hilton (Principal Investigator)Jean-Yves Guillemaut (Co-Investigator)John Collomosse (Co-Investigator)Josef Kittler (Co-Investigator)Mark Plumbley (Co-Investigator)Philip J B Jackson (Co-Investigator)Wenwu Wang (Co-Investigator)
Plain English summaries and category classifications on this site are generated by AI and may not perfectly reflect the original research.
Is something wrong? Let us know