Brain-to-text: Researchers develop brain-computer interface that turns neural signals into typed text in real time
Thoughts turned into text: Two implanted electrode arrays record the brain activity produced by thinking about writing letters. This information is then collected and processed in real-time by a computer, which converts that data into words on a screen. Courtesy Shenoy lab & Erika Woodrum (artist)
Scientists have developed a brain-computer interface (BCI) which for the first time enables neural signals associated with handwriting to be decoded and turned into text in real time.
This breakthrough in BCI technology has the potential to enable paralysed or paraplegic individuals – unable to write or speak – to communicate in writing, at speed, in real time.
The study was published in Nature, May 12.
The researchers developed an intracortical BCI that decodes attempted handwriting movements from neural activity in the motor cortex and translates it to text in real time, using a recurrent neural network decoding system.
They tested the device in a participant with paralysis which in essence enabled him to type without using his hands. By thinking about handwriting letters, the researchers were able to decode the neural signals with an algorithm which turned the letters into typed text on a screen in real time.
The study participant typed 90 characters per minute – more than double the previous record for typing with such a BCI. However, previous studies had not used decoded neural handwriting signals. They had instead used thought-controlled cursor movements on a virtual keyboard.
Neural signals associated with handwriting
Study coauthor Krishna Shenoy, a Howard Hughes Medical Institute (HHMI) investigator at Stanford University and his team have in recent years been studying neural activity associated with speech in an effort to reproduce it. However, they had not considered trying to decode neural signals associated with handwriting, says Frank Willett, an HHMI research specialist and neuroscientist in Shenoy’s group.
First, the participant was asked to copy letters that were displayed on the screen, which included the 26 lower-case letters along with some punctuation: “>” which was used as a space and “~” which was used as a “full stop.” At the same time, implanted electrodes recorded the brain activity from approximately 200 individual neurons associated with handwriting that responded differently while he mentally “wrote” each individual character. After a series of training sessions, the BCI’s computer algorithms learned how to recognize neural patterns corresponding to individual letters, allowing the participant to “write” new sentences that hadn’t been printed out before, with the computer displaying the letters in real time.
“This method is a marked improvement over existing communication BCIs that rely on using the brain to move a cursor to ‘type’ words on a screen,” said Willett, the study’s lead author. “Attempting to write each letter produces a unique pattern of activity in the brain, making it easier for the computer to identify what is being written with much greater accuracy and speed.”
Level of flexibility
This system also provides a level of flexibility that is crucial to restoring communication. Some studies have gone as far as attempting direct thought-to-speech BCIs that, while promising, are currently limited by what is possible through recordings from the surface of the brain which averages responses across thousands of neurons.
“Right now, other investigators can achieve about a 50-word dictionary using machine learning methods when decoding speech,” said Dr. Shenoy. “By using handwriting to record from hundreds of individual neurons, we can write any letter and thus any word which provides a truly ‘open vocabulary’ that can be used in almost any life situation.”
Although it is still relatively early days for this technology, it offers the potential to help those who have completely lost the ability to write and speak.
In the future, Dr. Shenoy’s team intends to test the system on a patient who has lost the ability to speak, such as someone with advanced ALS. In addition, they are looking to increase the number of characters available to the participants (such as capital letters and numbers).
The participant is part of a clinical trial, called BrainGate2, which is being conducted by a collaboration of internationally recognized laboratories, universities, and hospitals working to advance brain-computer interface technologies.
Reference
Willett, F.R., Avansino, D.T., Hochberg, L.R. et al. High-performance brain-to-text communication via handwriting. Nature 593, 249–254 (2021). https://doi.org/10.1038/s41586-021-03506-2
