HOLO-AIM : Holographic surgical guidance using Artificial Intelligence and Mixed reality for safer and more efficient Surgery

Research question Can artificial intelligence and computer vision be used to project reliable mixed reality visualisation of personalised anatomy onto the patient body during surgery with sufficient accuracy for clinical use? Background Over 310 million surgical procedures occur annually, with intraoperative errors in 15% of cases and 4.2 million postoperative deaths within 30 days. Surgical outcomes vary due to differences in surgeons visuo-spatial abilities, experience, and skill, contributing to approximately 26% of complications. Mixed Reality (MR) may enhance surgical visualisation and reduce variability, but real-time, accurate registration of holograms to patient anatomy remains a challenge. With an NIHR i4i FAST award, we developed and patented an AI-driven solution using depth cameras for real-time 3D scans, enabling automated, markerless registration by aligning point cloud data with preoperative holographic models. We identified sentinel lymph node biopsy (SLNB) for melanoma and breast cancer (70,000 UK cases annually) as an ideal initial application, aiming to improve precision, reduce operative time, enhance safety, and cut costs. Aims and Objectives This project aims to develop an AI-powered mixed reality surgical guidance system for clinical validation (TRL7). Objectives include Creating an AI model for real-time, accurate registration in compliance with medical software regulations. De-risking the technology for feasibility and clinical trials. Engaging patients, the public, and healthcare professionals to support development and adoption. Methods The project follows a structured multi-work package (WP) approach: WP1: Develop the core AI-driven technology for real-time, markerless registration. Tasks include infrastructure setup, development of segmentation and 3D reconstruction algorithms, hologram alignment using transformer networks, and iterative testing on 3D-printed models. WP2: Explore human factors via user research, task analysis, and clinical simulations to ensure usability and integration into surgical workflows. WP3: Embed patient and public involvement and engagement (PPIE), aligning the technology with patient needs and fostering engagement through outreach and dissemination. WP4: Conduct early health economic evaluation, mapping clinical pathways and assessing cost-effectiveness for NHS adoption. WP5: Develop regulatory strategy and quality systems, ensuring compliance for CE/UKCA certification. WP6: Adapt the system for end-user deployment on MR headsets, ensuring system integration, user interface optimisation, and data governance. WP7: Conduct a first in human clinical trial, comparing outcomes to standard care to validate performance and inform broader clinical adoption. Timelines for delivery The project spans 36 months. The first year focuses on programming, coding and technical development, with PPIE involvement throughout. This will be followed by work packages for human factors, regulatory governance, and preliminary health economics. The last year focuses on end user app implementation, clinical trial running to assess feasibility, safety, and finalise health economic evaluation for NHS adoption. Anticipated Impact and Dissemination We believe this technology will enhance surgical accuracy, safety, efficiency, and operative time. it supports precise navigation in complex procedures, lowering risks of injury to critical structures. This could lead to fewer surgical complications, shorter hospital stays, and improved theatre utilization. Initial focus is on breast and melanoma operations, impacting over 70,000 UK patients annually. However, wider application for other surgical indications and clinical fields is anticipated.