Decoding imaginary handwriting trajectories of multi-stroke characters for universal brain-to-text translation

The potential to decode handwriting trajectories from brain signals has yet to be fully explored in clinical brain-computer interfaces (BCIs). One of challenges remains that the clinical BCIs mostly rely on imaginary movement due to motor deficit of the subject, which often leads to misalignment with neural activity and impedes accurate decoding. Here, we recorded intracortical neural signals from a paralyzed patient during imaginary handwriting of Chinese characters, from which the trajectories of handwriting were reconstructed and translated into texts using machine learning approach. We introduced an innovated decoding framework that incorporates a novel loss function, DILATE, to accommodate both shape and temporal distortions between movement and neural activity to account for the misalignment issue. Our method reconstructed closely resembled and human-recognizable handwriting trajectories, outperforming the conventional mean square error loss by 10% of recognition rate. Moreover, the new decoding framework enabled effective multi-day data fusion, resulting in further 15% enhancement. With a dynamic time warping approach, the recognition rate achieved up to 91.1% within a 1000-character database. Additionally, we applied our method to a previous subject who imaged handwriting of English letters, showcasing its capability for single-trail trajectory reconstruction and 13.5% higher recognition outcomes. Altogether, these findings demonstrated a new decoding scheme for BCIs that could accurately reconstruct the imaginary handwriting trajectory. This advancement paves the way for a universal brain-to-text communication system that is applicable to any written language, marking a significant leap forward in the field of neural decoding and BCI technology.