Brain Commands and Beyond: Decoding Inner Speech for Neural Prosthetics

Millions of people worldwide are deprived of the simple ability to speak because of neurological disorders such as traumatic brain injury, brainstem stroke, or motor neurone disease. In the latter case, the loss of speech is often considered the worst outcome of disease progression. The current state of assistive communication technologies (such as those used by Stephen Hawking) can provide some relief. However, they require residual motor control such as cheek or eye movements. Current technologies also suffer from frustratingly low latencies, with users producing only 20 words per minute. Natural speech, by contrast, is produced at the rate of hundreds of words per minute. For all of these reasons, a new class of speech neuroprosthetic - capable of reading out (or "decoding") intended speech directly from the brain - would provide significant benefits to some of the most isolated people in society. The perfection of speech neuroprosthetics will also represent a scientific milestone in our understanding of how speech and language are represented in the brain. The first speech neuroprosthetic was achieved in a paralysed (anarthric) patient in the summer of 2021. Like much recent work, this landmark study used data from electrodes implanted in the sensorimotor cortex. Although there are advantages to such data, they also have limitations beyond the risk of surgery and installation of electronics into the brain. It is very difficult to obtain large amounts of these surgical data, which limits our ability to leverage the power of deep learning. Another important limitation of surgical data is that speech neuroprosthetics focus on decoding "inner" speech. Unlike overt speech, much less is known about the underlying neurobiology of inner speech. Is it more like imagined articulation ("motor imagery") or imagined audition ("auditory imagery")? Surgical data often targets the sensorimotor cortex, which makes sense for the decoding of overtly articulated speech. But this may be suboptimal for decoding inner speech. Here, we focus on non-invasive inner speech decoding with MRI and magnetoencephalography (MEG). Non-invasive neuroimaging provides, at least, complementary insights to surgical data. The first objective of the project thus seeks to address questions about the nature of inner speech: Where in the brain can we decode it? Does the neural organisation of inner speech differ between individuals? How well can decoders be transferred from one person to another? Answering questions like these will help to design better neuroprosthetics in any imaging modality. Turning to the second objective, there are good reasons to believe that non-invasive methods will produce a viable and less risky speech neuroprosthetic for paralysed patients. MEG-based decoders for speech comprehension (i.e. listening to speech) produce impressive results. Decoding inner speech is harder but - as our pilot data suggests - can be overcome by a combination of big data and deep learning. Thus the project aims to acquire a MEG dataset of sufficient scope (hundreds of hours) within-subject to show that inner speech decoders can, in principle, solve a sequence of tasks from keyword spotting (easier) to large-vocabulary continuous inner speech decoding (harder). The goal is not only to produce state-of-the-art results for each of these tasks, staggered by increasing difficulty and usefulness, but to shape a clear set of objectives for the community to optimise. Thus the MEG data will be released as part of a machine learning competition, inspired by the role that the ImageNet competitions have had in driving the field of computer vision over the past 10 years. We aim to drive similar advances for inner speech decoding.