AI-enabled High Stakes Assessment

Context: Each year, around a million 16 to 18 year olds sit high-stakes exams for university, apprenticeships, or employment. Ensuring swift, accurate, and reliable exam marking is crucial for these life-changing opportunities. Awarding bodies like AQA employ rigorous Quality Assurance (QA) mechanisms to ensure consistent marking and grading quality, yet tasks like standardising human examiners and marking open-ended responses are challenging. Current QA mechanisms cannot eliminate all risks associated with a highly manual and large-scale process. This triggers investigation into automated scoring systems. However, automating the scoring of high-stakes assessment is complex and requires emulation of human behaviour, involving assessing various factors such as content, scientific factuality, coherence, and logical reasoning, to name a few. Recent advancement of large language models (LLMs) has made it possible to explore innovative applications including student assessment. While LLMs offer numerous benefits, they also present new challenges. Challenges: To drive innovation in high-stakes exams, AI systems built on LLMs must address three challenges: Challenge 1 (C1): The AI system needs to be transparent and accountable, able to explain its suggested marks. However, training it to mimic expert human markers is challenging due to limited annotated data (human examiners are rarely required to explicitly explain a mark). Challenge 2 (C2): LLMs ‘hallucinate’, that is, they may produce incorrect or illogical text. This poses a risk when analysing student responses. Tailored data and training of a neural modular framework are needed to address this. Challenge 3 (C3): Human marking cannot be thoroughly objective, so the concept of marking ‘accuracy’ is subject to interpretation. Measuring explanation quality is even less defined. This presents a challenge when training and evaluating AI models. Aims and Objectives: To address the aforementioned challenges we have defined the following objectives: Objective 1 (addressing C1): To develop a hybrid AI approach informed by the cognitive process of human scoring for explainable assessment of open-ended answers in high-stakes educational settings. Objective 2 (addressing C2): To develop a new generation of neural modular paradigm built on LLMs, leveraging a novel neural architecture to improve reasoning and decision-making. Objective 3 (addressing C3): To design a new performance evaluation framework for automated marking systems. Achieving the above objectives requires interdisciplinary research combining the assessment expertise of AQA and the AI expertise of King’s College London (KCL) to develop a new, modular AI system that will improve the quality of secondary school qualifications (e.g. GCSE and A-level), with benefits for students, schools, tertiary education institutions and employers. The system would assist AQA and similar companies with various quality assurance and quality control tasks for delivering exam results and certificates. Applications and Benefits: KCL would benefit from this project by acquiring data, insight and know-how around high-stakes exams. Collaborating with AQA would enable KCL to improve the theoretical foundations of AI (focusing on reasoning, decision making and explainability) and develop a real-world application that addresses practical challenges in education. First-hand involvement of KCL academics will increase the relevance and transformative impact of their work. For AQA, the adoption of an automated marking system will improve the efficiency and robustness of assessment processes, leading to financial, reputational and social benefits. While gaining a competitive advantage in the exams sector, AQA will be able to licence the technology and therefore improve the overall sector’s capabilities.