MindLink-Eumpy: An Open-Source Python Toolbox for Multimodal Emotion Recognition - Li, Ruixin; Liang, Yan; Liu, Xiaojian; Wang, Bingbing; Huang, Wenxin; Cai, Zhaoxin; Ye, Yaoguang; Qiu, Lina; Pan, Jiahui (Frontiers in Human Neuroscience, 2021)
Emotion recognition plays an important role in intelligent human computer interaction, but the related research still faces low accuracy and subject-dependent limitation. In this paper, an open-source software toolbox called MindLink-Eumpy is developed to recognize emotions by integrating electroencephalogram (EEG) and facial expression information. MindLink-Eumpy applies a series of tools to automatically obtain physiological data from subjects. In the detection of facial expressions, the algorithm used by MindLink-Eumpy is a multi-task convolutional neural network (CNN) based on transfer learning technique. In the detection of EEG, MindLink-Eumpy provides two algorithms, including a subject-dependent model based on support vector machine (SVM) and a subject-independent model based on long short-term memory network (LSTM). In the decision-level fusion, the weight enumerator and AdaBoost technique are applied to combine the predictions of SVM and CNN. Two offline experiments were conducted on the DEAP dataset and MAHNOB-HCI dataset respectively and an online experiment was conducted on fifteen healthy subjects. The results show that multimodal methods outperform single-modal methods in both offline and online experiments. In the subject-dependent condition, the multimodal method achieved an accuracy of 71.00% for the valence dimension, and an accuracy of 72.14% for the arousal dimension. In the subject-independent condition, the LSTM-based method achieved an accuracy of 78.56% for the valence dimension, and an accuracy of 77.22% for the arousal dimension. The feasibility and efficiency of MindLink-Eumpy for emotion recognition is thus demonstrated.