Early-stage fusion of EEG and fNIRS improves classification of motor imagery - Li, Yang; Zhang, Xin; Ming, Dong (Frontiers in Neuroscience, 2023)
Many research papers have reported successful implementation of hybrid brain-computer interfaces by complementarily combining EEG and fNIRS, to improve classification performance. However, modality or feature fusion of EEG and fNIRS was usually designed for specific user cases, which were generally customized and hard to be generalized. How to effectively utilize information from the two modalities was still unclear. In this paper, we conducted a study to investigate the stage of bi-modal fusion based on EEG and fNIRS. A Y-shaped neural network was proposed and evaluated on an open dataset, which fuses the bi-modal information in different stages. The results suggests that the early-stage fusion of EEG and fNIRS have significantly higher performance compared to middle-stage and late-stage fusion network configuration (N=58, P <0.05). With the complete proposed framework, the average accuracy of 29 participants reaches 76.21% in the left-or-right hand motor imagery task in leave-one-cross validation, using bi-modal data as network inputs respectively, which is an excellent result and is basically same as the state-of-the-art hybrid BCI methods based on EEG and fNIRS data.