Scientists from the Graphenex-UTS Center for Artificial Intelligence at the Sydney Technological University (UTS) have developed a portable, non-invasive system that can decode thoughts and convert them to the text. Technology can help people who have lost the ability to speak due to illness or injury, including stroke or paralysis. It can also facilitate the interaction of a person with machines, for example, control of a bio -onic hand or robot.
This research was allocated as a key at the conference neurips in the New Orleans, demonstrating leading achievements in the field of artificial intelligence and machine learning. The work was led by Professor CT LIN and graduate students of YICUN DUAN and JINZHOU ZHOU from the Faculty of Engineering and Information Technologies UTS.
The participants in the study read texts, while there were special hats on their heads that write down the electrical activity of the brain through the scalp using an electroencephalogram (EEG). The artificial intelligence of Dewave, developed by researchers, analyzed the EEG-signals, converting them to words and sentences.
The UTS study was carried out with 29 participants, which makes it more reliable and adaptive compared to previous technologies, testing only one or two people. Despite the noise signals received through a hat instead of implanted electrodes, the study showed advanced results translated by EEG.
The model copes better with verbs than with nouns, often offering synonymous pairs of words instead of accurate translations. However, despite difficulties, the model demonstrates significant results, building keywords and creating similar sentences.
The accuracy of the translation at the moment is about 40% on the BLEU-1 scale, which measures the similarity of machine translation with high-quality reference translations. Researchers hope to bring this indicator to the level of traditional programs for translation of language or speech recognition, which is close to 90%.
Experts emphasized that the study is the first to translate the raw EEG waves directly into the language, which is a significant breakthrough. It is noted that this is the first case of using discrete coding methods in the process of converting the brain into text. The role of integration with large language models is also emphasized.
Previously, to transfer brain signals into the tongue, either surgical intervention was required for implantation of electrodes into the brain, as in the Neuralink project, Ilona Mask, or MRI scan, which was expensive and inconvenient for everyday use. The new technology can be used both with the eye tracking system, and without it.