AI agents powered by large language models are starting to negotiate, trade, and cooperate with each other on behalf of humans—but no one knows how to stop them from turning selfish or malicious when they interact in groups. This matters because these agents will soon manage supply chains, energy grids, financial markets, and healthcare logistics. If one agent learns to manipulate others for its owner’s gain, the entire system could destabilise—causing blackouts, market crashes, or safety failures. Current training methods treat each agent in isolation, ignoring the strategic dynamics that emerge when multiple agents interact. The researcher will visit three leading experts to build a new mathematical framework for multi-agent AI systems. The goal is to design defence mechanisms against manipulation, learning algorithms that maximise collective payoff even when agents are selfish, and control protocols that keep the whole system stable. If successful, this work could prevent the kind of cascading failures that occur when autonomous systems compete rather than cooperate. The research is fundamental—it tackles the core mathematics of multi-agent behaviour—but its findings could directly shape how future AI systems are trained to work together safely in critical infrastructure.
View original technical description
The rapid advancement of large pre-trained AI models such as LLMs and other foundations models has been disruptive to many sectors of our lives, and soon they will act as autonomous agents on our behalf in dealing with complex real-world problems while interacting with other agents and humans. However, it is unclear that when multiple of these large pre-trained AI agents interact with each other, how they would influence each other’s behaviour. This is especially true if they are programmed to be strategic (i.e., selfish, or malicious) on the behalf of their human owners/creators. These strategic behaviours, if not mitigated efficiently, will cause societal, financial, ethical, and safety disasters. Against this background, in this proposal I will aim to build a new research collaboration network with whom I will pursuit research questions addressing the following objectives: (i) To identify novel defence mechanisms that protect these large pre-trained AI agents from being manipulated to misbehave. (ii) To design novel learning algorithms which can help them to efficiently behave (e.g., to maximise total payoff overtime) when interacting with other agents with different strategic/selfish goals. (iii) To develop new protocols to control these agents’ collective behaviour to achieve system stability. (iv) To develop a novel framework to integrate the findings above into the training process of these large AI agents. In particular, I plan to visit the following researchers, each are renown in the respective research areas: Bo An from Nanyang Technological University, Singapore. Expert in designing LLM-based agents Tom Goldstein from University of Maryland, College Park, US. Expert in foundations of LLMs. Rebekka Burkholz from CISPA Helmholtz Center for Information Security, Germany. Expert in sparse neural networks. My aim with these visits is to foster an impactful research collaboration to explore novel and radical research directions to address the objectives mentioned above. Requested cost: Visiting Bo An 3 times :1 week/visit - Total cost: £10050 = 3 x £3350 (£1500 flight tickets + £850 per diem cost for 7 days + £1000 accommodation cost) Visiting Tom Goldstein 3 times: 1 week/visit - Total cost: £8100 = 3 x £2700 (£1000 flight tickets + £700 daily cost for 7 days + £1000 accommodation cost) Visiting Rebekka Burkholz: 3 times: 1 week/visit - Total cost: £6600 = 3 x £2200 (£500 travel cost + £700 per diem cost for 7 days + £1000 accommodation cost) Total travel cost: £24,750.00 GBP. In addition to this, there is a cost to cover 5% of the PI's time. The total cost for this is £4537 directly allocated and £3159 indirect cost. Given this, the total full economic (FEC) cost is: £32,446.00, with the requested cost to be £25,956.80.
Plain English summaries and category classifications on this site are generated by AI and may not perfectly reflect the original research.
Is something wrong? Let us know