Touchscreens and voice commands are just the beginning—this project aims to let people control computers with their whole hand, subtle gestures, or even their presence in a room, using new sensors like programmable electronic skin or radar. The problem is that designing such systems is unreliable. Current AI can reduce human effort but often fails in complex situations, and can control or deskill users. There is no principled way to let people flexibly share control with AI. This project integrates Active Inference—a theory of how brains predict sensory input—into the human-computer loop. It will create software tools that let designers build systems where humans and AI adapt to each other in real time, with users able to dial AI autonomy up or down. If successful, the work could transform how people interact with prosthetics, context-aware environments, and personalised interfaces. It would make systems robust to individual differences, fair for diverse users, and open to creative uses. The tools will be composable and shareable, accelerating research workflows across human-computer interaction. This is applied fundamental science: it builds theory into practical design, with no single immediate product but the potential to reshape how we think about human-machine collaboration.
View original technical description
Problem: Reliable design of interactive systems using advanced sensors and machine learning (ML) is an unsolved problem. New sensors could expand how we interact with computers, but are still hard to design for, without overly constraining user behaviour. AI algorithms can reduce human workload, but can fail in complex contexts and can control, deskill and dis-empower people. We have no principled workflows for designing interaction to allow users to flexibly share autonomy with supporting AI. Objectives: Integrate Active Inference theory into the human-computer interaction loop, linking human behaviour via sensors and ML/inference embeddings with dynamic mediating mechanisms to create end-to-end mutually adaptive loops between humans and systems. Develop novel interaction mechanisms for explicit and implicit control of AI autonomy levels to empower people via shared autonomy, while maintaining their agency. Create systematic, composable software tools for computational interaction design which support prototyping and analysis of ML-infused sensors coupled with humans, which can integrate probabilistic causal models to solve inverse problems with advanced sensors, adapting to closed-loop data. Impact: Using ML to give users freedom to express themselves individually, we can be robust to user heterogeneity, ensure fairness for diverse users and enable creative uses of technologies. Our tools will form the foundation of future usable interfaces with novel sensors and rich data spaces, and advances can be shared and rapidly built on, transforming HCI research workflows. Applications: 1. Whole hand touch interaction via soft, programmable electronic skin, for personaliseable interfaces with novel forms, and prosthetics. 2. Google's Soli radar for 3D gesture, pose and proxemic interaction, 3. Radio Frequency sensing of humans for context-aware interaction.
Plain English summaries and category classifications on this site are generated by AI and may not perfectly reflect the original research.
Is something wrong? Let us know