Advancing Social Robots for Personalised Healthcare through Multi-Sensor Fusion and Cognitive Large Language Model

The EU population is ageing at an unprecedented rate, while the working-age population is shrinking. This demographic shift brings significant challenges, including rising demand for long-term care services, a healthcare workforce shortage, and pressure on pension and welfare systems. Recently, social robots have been adopted to provide companionship for human mental healthcare, while the current development is still at the preliminary stage, especially in the healthcare industry. The general limitations of the current social robots are: (a) cannot support human physical healthcare by providing continuous monitoring of vital signs (e.g., respiration rate, heart rate, electrocardiogram), (b) cannot recognise advanced mental health indicators due to the unitary sensors that can only capture visual or acoustic information and (c) cannot provide personalised and empathetic responses for healthcare services.HealRo, as the world’s first multi-sensor healthcare robot solution, is proposed to unleash the potential of social robots for daily healthcare by developing a robust multi-sensor system that can continuously monitor human physical health indicators (e.g., heart rate) for moving subjects over a long distance. In addition, a DL framework for information fusion should be designed to integrate the features extracted from different sensors and enhance affective computing beyond emotion recognition, with advanced indicators (e.g., stress and anxiety) monitored to better evaluate seniors’ mental health conditions. Lastly, a cognitive LLM is designed based on cognitive psychology to guide the LLM through complex reasoning tasks to generate empathetic and unbiased responses to eventually achieve personalised healthcare.