In this part, we use AOMEMD and HT to construct Hilbert spectrum. For a single-channel EEG signal x(t), the AOMEMD is first used to decompose x(t) into a finite number of IMFs and a residue. Therefore, x(t) can be represented as Equation 2:

where c_k(t) (k = 1, 2,…, n_imf) is the kth IMF and r(t) represents the residue. The frequency of the n_imf IMFs decreases from the first to the n_imfth in order. In this work, we use the AOMEMD without the optimization strategy, which can save computation time while maintaining performance (Sun et al., 2024). The AOMEMD obtains IMFs through the following sifting process and the details of EMD are referred to the work of Huang et al. (1998).

Step 1: Input the signal x(t). Initialize k = 1 and r_k–1(t) = x(t). The number of phases is n_p.

Step 2: Determine the amplitude a¯k and frequency f¯k of the kth group masking signal v_k(t) with resulted IMFs by applying EMD to r_k–1(t).

Step 3: Construct the kth group masking signal vk⁢j⁢(t)=a¯k⁢c⁢o⁢s⁢[2⁢π⁢f¯k⁢t+2⁢π⁢(j-1)/np], (j = 1, 2,…,n_p). Obtain the kth IMF ck⁢(t)=[∑j=1npEMD1⁢(rk-1⁢(t)+vk⁢j⁢(t))]/np, where EMD₁ (⋅) represents to obtain the first IMF using EMD.

Step 4: Update r_k(t) = r_k–1(t)- c_k(t) and k = k+1. If r_k–1(t) fulfils termination criterion, r(t) = r_k–1(t); otherwise, go to step 2 and execute the loop.

For the obtained c_k(t) (k = 1, 2,…, n_imf) by AOMEMD, we use the HT to obtain the instantaneous frequency f_k(t) and instantaneous amplitude a_k(t) of c_k(t). The formula for y_k(t) obtained by applying the HT to c_k(t) is shown in Equation 3 (Huang et al., 1998):

where p.v. is the cauchy principal value. Then, f_k(t) and a_k(t) are solved as shown in Equations 4, 5:

Then the amplitude distribution of x(t) with frequency and time is the Hilbert spectrum (HS), denoted as HS(f, t), expressed as follows Equation 6:

Where Re represents the real part and i is the imaginary unit. HS(f, t) is a two-dimensional matrix with a time resolution equal to the sampling period (Molla and Hirose, 2007). Examples of HS are shown in Figure 4. As shown in Figure 4, the time-frequency distribution of samples from set EC and set ES is quite different.

For a single-channel EEG signal x(t) of length T_EEG, the RP is computed as the following. First, according to Takens’ embedding theory (Takens, 1985), a phase space is reconstructed for x(t), and a phase point in this space is denoted as s_n and n = 1, 2, …, T_EEG – T_ps (m–1), where T_ps is the time delay and m is the embedding dimension. T_ps and m can be selected using mutual information (MI) and false nearest neighbor (FNN) methods, respectively (He et al., 2023). Second, the RP is defined according to Equation 7 below:

where ε is the distance threshold and || ⋅ || is the Euclidean norm. By assigning a black dot to the RP element (n, j) of RP(n, j) = 1 and a white dot to the RP element (n, j) of RP(n, j) = 0, a binary square image of an RP can be obtained, as shown in Figures 5A,C.

Binary square images constructed using the threshold method lose a lot of information, so we convert the RP to a grayscale intensity image (named grayscale RP, GRP). The examples of GRP are shown in Figures 5B,D. The GRP is defined according to Equation 8 below (Chen and Shi, 2019):

In this section, the network structure in each module of the HG-SANet is described in detail.

The first HS-PCNet module inputs the HS built in section 2.2.1 into a parallel two-channel CNN network containing different convolutional kernels. CNN overcomes the limitation of insufficient feature extraction ability of machine learning methods through simultaneous shift calculation of convolutional kernel in the time and frequency dimensions of feature maps (Zhang et al., 2015). It has been used in time-frequency feature extraction of EEG signals (Sui et al., 2021). Therefore, in this section, we use CNN to further extract the high-level time-frequency features of HS. For HS, we design a parallel two-channel CNN network containing different types of convolutional kernels for feature extraction. Two types of convolution kernels are set as [N_kernel, 1] and [N_kernel, N_kernel]. As EEG signals comprise time-series data, we construct a convolution kernel of size [N_kernel, 1] to make feature extraction pay more attention to changes in the time domain. The convolution kernel of size [N_kernel, N_kernel] slides synchronously in the time domain and frequency domain dimensions of the HS to retain its original time-frequency characteristics. The time-frequency features are fully extracted by complementing the high-dimensional features of the two branches. The structure and details of the HS-PCNet module are shown in Figure 6A. The structure and parameter settings in each CNN block of the HS-PCNet module are shown in Table 2. In Table 2, the serial number corresponds to the serial number in Figure 6A. Each CNN block has a batch normalization layer and a ReLU activation layer between the 2D convolution (Conv 2D) and max pooling layers, which are omitted to save space. For the HS-PCNet module, the batch normalization layer normalizes the input data in small batches to speed up the training of the HS-PCNet and reduce the sensitivity to the network initialization. The max pooling layer performs downsampling by dividing the feature map into rectangular pooling regions and calculating the maximum value for each region, which helps reduce overfitting. The dropout layer makes the activation value of a certain neuron stop working with a certain probability, helping to prevent the HS-PCNet from overfitting (Krizhevsky et al., 2017).

A large number of studies have proved the advantage of residual networks in the field of image recognition (He et al., 2015). Therefore, the second GRP-ResNet module inputs the GRP in section 2.2.2 into a CNN with residual connections to fully learn the nonlinear dynamic features in the GRP. The convolutional module in the GRP-ResNet can use the receptive field of neurons to extract high-level local feature representation of the GRP, and the residual module allows cross-layer propagation, which can avoid overfitting caused by too many layers in the network, and will not lose important information in the feature (He et al., 2015). The overall structure of the GRP-ResNet module is shown in Figure 6B. The structure and parameter settings in each residual block and CNN block are shown in Table 3, and the serial number corresponds to the serial number in Figure 6B. In each block, there is a batch normalization layer after the 2D convolution (Conv 2D) layers, which is omitted to save space.

Research shows that the self-attention mechanism (Vaswani et al., 2017) can help the network select important features and assign higher weights to these important features to improve the performance of downstream tasks (Lin et al., 2022; Yang Q. et al., 2023). Therefore, in the third MF-SANet, a feature fusion module based on a multi-head self-attention mechanism is proposed to assign optimal weights to different types of features obtained by the HS-PCNet module and GRP-ResNet module to enhance the information extraction capability of HG-SANet further. The feature fusion formulas are calculated as follows. First, the features extracted from the HS-PCNet module and GRP-ResNet module are concatenated and the concatenated features are denoted as Feature_initial. In the self-attention mechanism, there are three kinds of important input queries, keys and values, denoted as QUE, KEY, and VAL, respectively. They are calculated as Equations 9–11 (Lin et al., 2022):

where j = 1, 2, …, N_head and N_head is the number of attention heads. WjQUE, WjKEY, and WjVALare the parameter matrices. Then, the features of the final output are calculated as Equation 12:

where W^o is a parameter matric and Concat(⋅) is the concatenating operation. The HEAD_j is calculated as Equation 13

where d_KEY is the dimension of keys.

In the classification layer based on the full connection layer, the activation function after the full connection layer is the softmax function.

In this paper, epileptic EEG recognition is evaluated using precision (P), recall (R), accuracy (Acc), and specificity (SP) (Sriraam and Raghu, 2017; Gao et al., 2018). The sensitivity and recall are calculated using the same formula, so we no longer calculate sensitivity separately. Precision focuses on evaluating the percentage of true positive samples in all predicted positive samples. Recall focuses on the percentage of all positive samples that are successfully predicted to be positive. Accuracy is the proportion of correctly classified samples in total samples. The specificity is the proportion of all negative samples predicted correctly to all actual negative samples. These metrics are calculated as shown in Equations 14–17.

where N_TP is the number of true positive (TP) samples, N_TN is the number of true negative (TN) samples, N_FP is the number of false positive (FP) samples, and N_FN is the number of false negative (FN) samples.

In the decision stage, the HG-SANet gives prediction labels for all short segments of each sample. Finally, based on the prediction labels of short segments, the majority voting method is used to make the final prediction for the category of each test sample.

Parameters of the HG-SANet in the training process are set as follows. Adaptive moment estimation (Adam) optimizer is used to train the HG-SANet. The epoch used for training is 30, and the mini-batch size used for each training iteration is 32. The learning rate is 0.001. The cross-entropy loss function is used as the loss function. We reduce the overfitting of the HG-SANet by adding the regularization term of the weight to the loss function. The number of heads in the attention module is set to 2. In the testing process, the testing sample is input into the proposed system trained by the training set as shown in Figure 3 to obtain the final recognition result. The ten-fold cross-validation is used to obtain an unbiased evaluation of classification performance.