Researchers at the University of California, San Francisco reported this week that they have built a brain-computer interface able to decode a paralyzed person's intended speech and upper-body gestures at the same time, driving a digital avatar that talks and moves within seconds of the person's intent. The findings, led by neurosurgeon Edward Chang, were published Sept. 14 in Nature Neuroscience, and the team describes it as the first system to translate both channels of communication together rather than one at a time.
Three participants living with severe paralysis caused by amyotrophic lateral sclerosis or a brainstem stroke had a thin grid of electrodes, called an electrocorticography array, placed over the motor cortex. Machine-learning software trained on brain activity recorded while the participants attempted to speak and gesture together — saying "hello" while waving, or "yes" while nodding — decoded intent more accurately than software trained on speech or gesture data alone. Two of the three participants went on to control a personalized, full-body avatar in real time, with one reaching close to 100 percent decoding accuracy for both speech and gesture across a series of test conversations.
A missing layer of conversation
Earlier speech neuroprostheses, including previous devices from Chang's own lab that let a paralyzed woman generate synthetic speech and a talking digital avatar, have converted brain signals into text or spoken words but largely left out gesture — the nods, waves and shrugs that carry much of the meaning in ordinary conversation. The new work, developed with support from the National Institutes of Health, builds on more than a decade of research into reading intended movement and speech directly from the brain's sensorimotor cortex, and adds evidence that the two functions are more intertwined in that brain region than previously modeled.
Conversation is about much more than the words being spoken. It's a multilayered, dynamic process involving the whole motor cortex.
Edward Chang, UCSF
The study remains an early proof of concept. Only three people were implanted, one did not achieve reliable joint control of the avatar, and the system currently depends on a wired connection between the implanted electrode array and external processing hardware, which confines its use to a lab setting. Researchers say they next plan to test a fully implantable, wireless version meant for everyday use — a step that would require additional trials, regulatory review and years of engineering work before any path toward wider clinical availability for people with ALS, brainstem stroke or other causes of severe paralysis.
