EEG-fNIRS-based hybrid image construction and classification using CNN-LSTM - Mughal, Nabeeha Ehsan; Khan, Muhammad Jawad; Khalil, Khurram; Javed, Kashif; Sajid, Hasan; Naseer, Noman; Ghafoor, Usman; Hong, Keum-Shik (Frontiers in Neurorobotics, 2022)
The constantly evolving human–machine interaction and advancement in socio-technical systems have made it essential to analyze vital human factors like mental workload, vigilance, fatigue, and stress by monitoring brain states for optimum performance and human safety. Similarly, brain signals have become paramount for rehabilitation and assistive purposes in fields such as brain–computer interface (BCI) and closed-loop neuromodulation for neurological disorders and motor disabilities. The complex, non-stationary, and low signal-to-noise ratio of brain signals poses a significant challenge for researchers to design robust and reliable BCI systems to accurately detect the meaningful changes in brain states outside the laboratory environment. Different neuroimaging modalities are used in hybrid settings to enhance the accuracy, increase control commands, and to decrease time for brain activity detection. Functional near-infrared spectroscopy (fNIRS) and electroencephalography (EEG) measure the hemodynamic and electrical activity of brain with good spatial and temporal resolution respectively. But in hybrid settings, where both modalities enhance the output performance of BCI , their data compatibility due to huge discrepancy between their sampling rate and number of channels still remains a challenge for real-time BCI applications. Traditional methods like down-sampling and channel selection, result in important information loss while making both modalities compatible. In this study, we present a novel recurrence plot (RP)-based time- distributed convolutional neural network and long short-term memory (CNN-LSTM) algorithm for integrated classification of fNIRS EEG for hybrid BCI applications. The acquired brain signals are first projected into a nonlinear dimension with RPs and fed into the CNN to extract essential features, without performing any down sampling. Then, LSTM is used to learn the chronological features and time-dependence relation to detect brain activity. The average accuracies achieved with the proposed model were 78.44%% for fNIRS, 86.24% for EEG, and 88.41% for hybrid EEG-fNIRS BCI. Moreover, the maximum accuracies achieved were 85.9%, 88.1%, and 92.4%, respectively. The results confirm the viability of the RP-based deep-learning algorithm for successful BCI systems.