Real-time High-Fidelity Augmented Reality in Laparoscopic Liver Resection

Liver disease and liver cancer together cause ~2.5% of deaths annually in England, with a projected rise of liver cancer by 6% from 2025 to 2040 in the UK. Long-term damage to the liver (e.g. Cirrhosis and liver cancer) is treated by surgical removal of parts of the liver or transplant. Laparoscopic (keyhole) and robotic liver resections are pivotal in modern surgery, offering minimally invasive options for removing unhealthy liver parts while preserving function with faster healing. Due to a lack of tactile feedback, liver opacity, and the limitation in the accessibility of interpretable pre-operative 3D MRI/CT scans during the surgery, surgeons need to memorise and interpret the complex liver structures and associated abnormalities throughout, making the surgical procedure longer and technically difficult. I propose developing a real-time augmented reality (AR) system using an advanced data-driven method that will automatically overlap liver segments, functional structures, blood vessels and optimal liver margins from pre-operative scans onto the laparoscopic scene. I will address current unmet clinical needs by enhancing the visualisation, enabling faster and optimal functional organ-preserving liver surgery, leading to improved patient safety and recovery. I will address current gaps in existing algorithms, such as the lack of real-time performance and algorithmic robustness. I have co-designed the project with surgeons, patients, and industry. I will develop a novel framework comprising the following: 1) Extraction of 3D models with eight liver segments and other information such as vessel map and tumour. 2) Intraoperative ultrasound segmentation of major liver landmarks. 3) Mapping of the pre-operative anatomy and key landmarks overlaid onto the liver during surgery. 4) Real-time dynamic navigation using vision and sensor-based system design for improved robustness. I will also develop protocols for achieving standardised multimodal data for robust AR-based systems in liver surgery. Algorithmic validation is a key challenge in current research, so I propose cross-linking the validation designs to infer the accuracy and reliability of each. Building upon my prior work on the segmentation of liver segments, I will develop a novel multi-class 3D segmentation method for tumour, ligament, and optimal resection margins. To fuse the 2D liver image with the extracted 3D maps from MRI/CT scans, I will explore a neural radiance field (NeRF) technique, which aims to reconstruct 3D representations from 2D images. However, I will design NeRF with a displacement estimation algorithm to tackle liver plasticity. After reconstruction, the 3D liver shape will be used to register with the 3D liver model, which will then be projected, and a fast differential rendering technique will be applied to obtain the final 2D-3D fusion. An array of sensors will provide a vision-sensor setup that will be assessed and iterated with the industrial partner on the project. This invention will compensate for failures in vision-based systems and allow faster realignment of pre-operative maps onto the liver during surgery. I will use data from 360 patients over 36 months, overcoming the limited data problem. The primary endpoint is to obtain a reprojection error of less than 1mm (currently over 30mm) and real-time performance. This will be assessed using 20% of the collected data. The secondary endpoint is to determine the usability of the AR technology in the clinic by surgeons, including a scoring system that will evaluate easy-to-use and rendering time for visualisation using the developed framework.