Emotion mainly refers to a kind of inner emotional response produced by the psychological needs of people. It is a series of subjective psychological experiences formed by integrating the feelings, perceptions, and behaviors of people (Li, 2017). In the daily life and work of people, emotion is everywhere. In medical care, different nursing measures can be provided according to the emotions of patients to improve the quality of care. When treating psychological and physical diseases, an emotional state can be used as an auxiliary diagnostic basis. In the human-computer interaction, by identifying emotional states, the interaction is more friendly and natural. Therefore, the qualitative recognition and classification of emotional states of people have a high research value.

Psychological and emotional states can be transmitted in a variety of signal forms, such as via facial expressions, voice intonation, text, body posture, and other directly observable emotional signals. An emotional state of a person can also be learned through physiological signals, such as electrical brain signals, skin temperature, and breathing signals. Compared with observable emotions, electroencephalogram (EEG) signals are obtained by directly reading brain activity. EEG signals are more real and difficult to disguise. Therefore, the results obtained by using EEG signals to classify emotions are more authentic and reliable (Ullsperger et al., 2014; Naseer and Hong, 2015; Chen et al., 2020). As a result, this article mainly discusses studying emotion recognition methods based on EEG signals.

In EEG emotion recognition research, the first type of method is based on traditional machine learning technology. This kind of method first expands the emotional feature extraction from the EEG signal and then expands the corresponding emotional pattern learning and classification of the feature data. Emotional feature extraction of EEG signals can be divided into four categories, including time-domain features, frequency-domain features, time-frequency features, and EEG signal non-linear dynamic features. For example, Atkinson et al. proposed an EEG emotion recognition method based on feature selection and a kernel classifier (Atkinson and Campos, 2016). In this study, the statistical characteristics of the EEG signal, frequency band power of different frequencies, Hjorth parameter, and fractal dimension are used. The statistical features include median, SD, and kurtosis coefficient. The frequency bands of EEG signals use θ (4–8 Hz), low α (8–10 Hz), α (8–12 Hz), β (12–30 Hz), and γ (30–45 Hz). Then, a feature selection method based on mutual information and a kernel classifier are used to optimize the performance of the emotion classification task. Mohammadi et al. decomposed the EEG signal using a wavelet transform, while the entropy feature and the energy feature were extracted from each sub-band signal. Then, the support vector machine (SVM) and k-nearest neighbor (KNN) pattern classifiers were used for training and testing (Mohammadi et al., 2017). The results showed that the accuracy of arousal was 86.75%, and the accuracy of valence was 84.05%. Fernández-Varela et al. evaluated the emotion recognition performance of six machine learning models in detail as follows: Fisher’s linear discriminant, SVM, artificial neural network, classification tree, KNN, and naive Bayes (Isaac et al., 2017). Li et al. (2018) studied the effectiveness and performance of 9 time-frequency domain features and 9 non-linear dynamic features through the combination of different feature selection and SVMs. Abhishek Tiwari et al. proposed motif series and graph-theoretical features, before analyzing three feature selection methods [ANOVA-based feature ranking and selection, minimum redundancy maximum relevance (mRMR) feature selection, and recursive feature elimination (RFE)] and three fusion strategies (feature fusion, score-level fusion, and output-associated fusion). They finally concluded that motif series features are better than spectral features and that feature-level fusion strategies can improve recognition accuracy. Score-level fusion can improve arousal prediction (Tiwari and Falk, 2019). Fu Yang et al. constructed a multidimensional high-dimensional feature set by extracting eight linear features, such as the Hjorth parameter, power spectral density, and SD, and two non-linear features, such as wavelet entropy and sample entropy. Then, the ST-SBSSVM method was proposed by combining the significance test, sequential backward selection, and the SVM (Yang et al., 2019). The accuracy of the ST-SBSSVM was 72% (DEAP) and 98% (SEED).

The second type of method is based on deep learning technology. Deep learning has powerful feature extraction and classification capabilities. The main deep learning models used in the field of EEG signal classification include convolutional neural networks (CNNs), recurrent neural networks (RNNs), and generative adversarial networks (GANs). For example, Wang and Shang (2013) used a deep belief network (DBN) to extract features from the original physiological data and constructed three classifiers to estimate the arousal, valence, and liking of emotions. Yang et al. (2018) and Banee et al. (2019) put 32 electrodes into a 9 × 9 matrix according to the position of the international 10–20 system. The former used a time sliding window to divide the continuous EEG signal and calculated the average value of the corresponding electrode in each window. The latter directly inputs the original signal value into the matrix. The final input of the CNN network is a two-dimensional matrix containing spatial location information. Li et al. (2017) used a sparse wavelet transform to transform the EEG signals of each channel into a two-dimensional spectrum and then input them into a CNN-RNN hybrid network. Hwang et al. (2020) generated an egg image with topological information based on differential entropy (DE) features and then used a deep network structure combining a CNN and an LSTM. Shawky et al. (2018) converted EEG signals into a three-dimensional continuous frame input depth network, which compared EEG signals with images while ignoring the spatial position information of electrodes in EEG signals. Ant topic et al. proposed two feature mapping methods, called topography (TOPO-FM) and holography (HOLO-FM). Then, the deep learning technique was employed for feature learning from TOPO-FM and HOLO-FM, and finally, SVM was used to classify emotions (Topic and Russo, 2021). In the above research, the transformation of the EEG feature form imitated the image learning mode of deep networks, including a CNN, an RNN, and a DBN as much as possible. In addition, there are studies on EEG signal analysis by using brain function networks, such as energy landscape (Krzemiński et al., 2020; Klepl et al., 2021), which provides a new idea for the study of EEG signals.

Electroencephalogram signals are non-stationary, non-linear, and complex physiological signals with multicomponent mechanisms (Golyandina et al., 2001). In the machine learning method, the time domain, frequency domain, and time-frequency domain characteristics ignore the fact that the EEG signal is a non-linear signal, and non-linear dynamic features need to extract sufficient structural information of EEG components. In addition, the traditional machine learning method extracts features from multiple channels of EEG signals and then connects them to a single feature vector without considering the temporal dynamics. In the method based on deep learning, it is difficult to train a high-quality model due to the small number of samples and the low resolution of EEG images. In addition, the feature analysis and extraction of EEG signals are easily affected by noise, which impacts the effect of the deep learning model. Therefore, this study discusses combining the features extracted by manual calculation with a neural network to solve two problems. The first problem is simply and effectively representing multichannel EEG signals before establishing the emotion recognition model. The second problem is constructing effective features to learn temporal dynamics. We regarded brain emotion processing as an overall coordinated physiological response. To save artificial features while retaining the original information as much as possible, we proposed emotion recognition based on dynamic energy features using a Bi-LSTM network, and the main contributions are as follows:

To verify the feasibility and effectiveness of the proposed method, we performed four groups of experiments: comparison with traditional methods, feature type selection, individual emotion recognition, and comparison with other methods.

The experiment is carried out by using the feature extraction method proposed in this study. The contrast method is the traditional method of extracting features from a single-channel signal. The methods are as follows: extracting features from signals in each channel, then using PCA for dimension reduction, and finally using SVM for recognition. To input the features into the SVM model, the feature vectors are composed of the mean value of the corresponding eigenvalues in the time window. In this way, the time-domain dynamics are considered to a certain extent. The relevant parameters of SVM are as follows: the type is C-SVC, the kernel function is RBF, and the setting method of penalty coefficient is Scan the parameter space 2^[–3:8] in step 1 to search for the best value.

The experimental dataset is SEED, and positive and negative emotions are identified. The results are shown in Figure 4.

In this study, we first obtained the energy sequence of each channel and then extracted features from the energy sequence. Therefore, the feature dimension of the energy sequence is 1/62 of that of the single channel. As seen in Figure 4, the recognition rate of the proposed method is 72.5% and that of the traditional method is 72.2%, which shows that the proposed dynamic energy sequence can retain the characteristics of each channel of the EEG signal and effectively reduce the feature dimension.

In this study, the data of 6 subjects in the seed dataset are randomly selected. According to the feature extraction method in the “Dynamic energy feature set” section, 49 types of features are extracted. Then, the MIPCA method is used to analyze the feature set F^270×49. The cumulative contribution rate of the principal components is shown in Figure 5. In Figure 5, the x-axis represents the principal component serial number, and the y-axis represents the cumulative contribution rate. The y-axis red mark indicates that the cumulative contribution rate has reached 94.65, and the x-axis red mark indicates that the number of principal components when the cumulative contribution rate is 94.65. In our experiment, the principal components whose cumulative contribution rate is before 95% are selected to determine the final feature set. Considering that different subjects may have different suitable features, we conducted three experiments at random and conducted statistical analysis on the results. The final features are shown in Table 4.

It can be seen from Table 4 that among the selected features, there are 8 time-domain features with the largest number. This shows that the time-domain features have a strong resolution. In the frequency domain, the features in beta, gamma, and all bands are better than those in other bands. This is consistent with the conclusion in the study by Li et al. (2018). For the DEAP dataset, in addition to the features in Table 4, we added the wavelet entropy (gamma) and the power spectrum sum (all).

In this study, we adopted two verification strategies, namely, leave one subject out (LOSO) and 10-fold cross validation (CV). For SEED, we classified two-dimensional emotions and three-dimensional emotions, and for DEAP, we identified the accuracy of arousal and valence. The results of SEED are shown in Figure 6. Figure 6A shows the recognition accuracy of two-dimensional emotion under the LOSO strategy. The average value of accuracy is 83.87%. Figure 6B shows the recognition accuracy of three-dimensional emotion under the LOSO strategy. The average value of accuracy is 69.76%. Figure 6C shows the recognition accuracy of two-dimensional emotion under the 10-fold CV strategy. Figure 6D shows the recognition accuracy of three-dimensional emotion under the 10-fold CV strategy.

The DEAP results are shown in Figure 7. Figure 7A shows the accuracy of arousal and valence under the LOSO strategy. The average value of arousal was 63.35% and the valence was 61.39%. Figure 7B shows that the accuracy of arousal and valence is under the 10-fold CV strategy. The average value of arousal is 77.3%, and the valence is 75.7%. The results of Figures 6, 7 verify the effectiveness of our method.

We compared the results obtained by the proposed method with other methods that adopt the same dataset and adopt the same verification strategy. The comparison results are shown in Table 5.

In this study, we proposed the concept of an energy sequence to reduce the influence of noise caused by feature extraction and then construct a dynamic energy feature set including time-domain features, frequency domain features, and non-linear features. The network model adopts fully connected layers and LSTM networks, which are suitable for small datasets. In Figure 1, for the dynamic energy sequence feature set + SVM, the recognition rate is 72.5%. For the traditional feature set + SVM, the recognition rate is 72.2%, which shows that the dynamic energy sequence can preserve the features of the EEG signal and effectively reduce the feature dimension. To further extract effective features, this study comprehensively considers the effectiveness of a single feature for labeling and the information redundancy between multiple features. Through the feature type selection experiment, 17 kinds of feature vectors were selected from the SEED dataset, and 19 kinds of feature vectors were selected from the DEAP dataset. Finally, to verify the feasibility of our method and to facilitate the comparison with advanced methods, this study selects two verification strategies, namely, LOSO and 10-fold CV. For the SEED dataset, with 2-classification with LOSO, in the study by Li et al. (2018), the accuracy is 83.33%; in the study by Yang et al. (2019), it is 89%; and in our method, it is 83.87%. For the 3-classification problem, in the study by Zheng et al. (2019), the accuracy is 60.93%, and in our method, it is 69.76%. For the 2-classification with a 10-fold CV, in the study by Topic and Russo (2021), the accuracy is 88.45%, and in our method, it is 89.42%. For the DEAP dataset, with 2-classification with LOSO, in the study by Li et al. (2018), the average accuracy is 59.06%; for our method V, the average accuracy is 61.39%, and for A, it is 63.35%. With a 10-fold CV, the result of our method (V: 75.78%, A: 77.34%) is better than the result (V: 71.09%, A: 72.58%) in the study by Siddharth et al. (2019) and close to the result (V: 76.61%, A: 77.72%) in the study by Topic and Russo (2021). Therefore, our method is effective. Emotion is a physiological and psychological activity state induced by internal or external stimuli. Although different scalp regions are affected by emotion, when emotion changes, EEG signals will change harmoniously as a whole. Therefore, this study analyses the synchronization of signals in different regions as a whole, which is consistent with the phenomenon that emotional changes will lead to the overall change of EEG signals. The limitation of this study is that taking all channel signals as a whole cannot verify the relationship between different regions and different emotions. In the future study, we will continue to study the noise problem in EEG signal analysis and processing, as well as the relationship between different regions and emotions.

In this study, the concept of an energy sequence is proposed, and it is verified through theory and experiment that an emotional recognition method based on an energy sequence can reduce the influence of noise and improve accuracy. In addition, due to the small size of the EEG dataset, this study uses the combination of traditional features and a Bi-LSTM deep network. Compared with other traditional methods and CNN deep neural models, this method can achieve higher average accuracy.

Publicly available datasets were analyzed in this study. This data can be found here: https://bcmi.sjtu.edu.cn/home/seed/downloads.html and http://www.eecs.qmul.ac.uk/mmv/datasets/deap/readme.html.

MZ developed the method based on dynamic energy features, performed all the data analysis, and wrote the manuscript. QW advised data analysis and edited the manuscript. JL supervised and led the planning and implementation of research activities and revised the manuscript. All authors contributed to the article and approved the submitted version.

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.