A single researcher is teaching a large language model to read thousands of patient and staff interview transcripts and automatically assess how well mental health services for children and adolescents follow a national care framework. The problem is that qualitative data—rich accounts of what actually happens in clinics and hospitals—is rarely used at scale because analysing it manually takes months of skilled human effort. Current computer methods can count words but miss context, nuance, and implied meaning. This project tests whether generative AI can reliably do what a trained qualitative researcher does: judge whether a service adheres to the THRIVE principles for integrated, person-centred child and adolescent mental health care. If the method works, it could unlock vast archives of existing interview data across health and social care. Services could receive automatic, detailed feedback reports on their performance without waiting for lengthy human analysis. The tools will be open-source and free, avoiding reliance on proprietary models. The researcher is also explicitly testing for AI hallucinations and bias, which are critical risks when using these models for service evaluation.
View original technical description
Background Historically, healthcare quality measurement has leaned heavily on quantitative methods, which use structured, numerical data to generate performance metrics. However, qualitative data offer more nuanced insights (for instance, into the experiences of patients and healthcare providers) and allow for an exploration of the why behind quantitative trends. However, analysing qualitative data is labour-intensive and time-consuming, and lacks the opportunities to scale up that exist for quantitative methods. Traditional natural language processing (NLP) and text mining methods provide good results in structured tasks with predefined rules and features but struggle with contextual and semantic complexity. I propose the integration of Large Language Models (LLMs) at various stages of qualitative data analysis for healthcare quality assessment. LLMs have demonstrated promising abilities in understanding nuanced linguistic relationships and implied meanings; disambiguating words and phrases based on context; and adapting to different domains and conversational contexts. Aim The aim of my PhD is to innovate healthcare quality assessment through Generative AI (GenAI), specifically by supporting qualitative data analysis with LLMs. Data and methods I will use data from the national evaluation of the i-THRIVE programme. The THRIVE Framework, developed in 2014, provides a set of evidence informed principles for creating an integrated and person-centred care system of Child and Adolescent Mental Health Services (CAMHS). i-THRIVE, which stands for implementing-THRIVE, has been developed to support CAMHS sites across the country with implementing THRIVE, drawing on implementation science principles. An evaluation of the i-THRIVE programme was carried out between 2017 and 2020, using qualitative analysis of a large number of interview transcripts, interview notes, and policy documents. This analysis used a detailed rubric (the THRIVE Assessment Tool) to manually assess adherence to THRIVE principles by experienced qualitative researchers. I will start by reviewing existing LLMs in terms of data security, confidentiality, and reproducibility. After selecting an appropriate, non-proprietary LLM for my research, I will develop a strategy to re-analyse the qualitative data from the i-THRIVE evaluation using this LLM, with the aim of automatically assessing adherence of CAMHS services to THRIVE principles. I will exploit state-of-the-art prompt engineering and fine-tuning techniques, and validate the outputs against the existing, manually-derived assessments. I will conduct experiments with different prompting techniques and establish which techniques work best in this context, and explore novel LLM techniques such as Retrieval-Augmented Generation (RAG) and multi-agency to improve the quality of outputs. I will dedicate specific attention to detecting biases and preventing hallucinations. I will develop frameworks for automatic feedback reports generation tailored for CAMHS sites Impact and dissemination Numerous qualitative data sources exist that can provide nuanced insight into the quality of health and social care services, but these sources are currently under-utilised due to the resource-intensive nature of qualitative data analysis. I will develop analytical methods that can be used to accelerate these analyses and reduce the time required from human analysts without compromising the quality of outputs. My methods will be open source and publicly available without any costs, and they will not rely on proprietary models or software.
Plain English summaries and category classifications on this site are generated by AI and may not perfectly reflect the original research.
Is something wrong? Let us know