As for RNNs, the classical LSTM cell is proposed to deal with the problem of “long-term dependencies” by introducing a “gate” into the cell to improve the remembering capacity of the standard recurrent cell. { f t = σ ( W f h h t − 1 + W f x x t + b f ) i t = σ ( W i h h t − 1 + W i x x t + b i ) c ˜ t = tanh ( W c ˜ h h t − 1 + W c ˜ x x t + b c ˜ ) c t = f t ⋅ c t − 1 + i t ⋅ c ˜ t o t = σ ( W o h h t − 1 + W o x x t + b o ) h t = o t ⋅ tanh ( c t ) (1) where W f h , W f x , W i h , W i x , W c ˜ h , W c ˜ x , W o h , and W o x are weight matrices and b f , b i , b c ˜ , and  b o are biases of LSTM to be learned during training. The above variables can parameterize the transformations of the input gate i t , forget gate f t , and output gate o t , respectively. σ in Eq. 1 is the sigmoid function and ⋅ stands for element-wise multiplication. c t denotes the cell state of LSTM. x t includes the inputs of LSTM cell unit, and h t is the hidden layer (Wang et al., 2016; Kong et al., 2017; Yu et al., 2019). One can find the mathematical models of the RNN and GRU in the Supplementary Material.

Based on the LSTM model, we propose our SS-RNN model, which better utilizes historical information and could enhance the long-term memory of the model. The architecture of the SS-RNN model is shown in Figure 1. It consists of a feature extractor and a three-layer strengthened skip LSTM (SS-LSTM) network (Figure 1A). The feature extractor is added here to process the datasets with multiple features (not time series data) like the Diabetes data and Breast cancer data used in this paper. It extracts the features of multiple feature data. Then, the output of the feature extractor is reshaped to a matrix of 32*4 for further input into the SS-LSTM network (refer to Supplementary Figure S55 and Supplementary Material). For standard time series datasets, such as Arrhythmia dataset, Epilepsy dataset 1, and Epilepsy dataset 2 used in this paper, we input them to SS-LSTM directly for training. Figure 1B shows the structure of a neuron in the second layer SS-LSTM, and the information at moment of t-skip (skip is positive integer) is used to strengthen the memory of the moment t.

In comparison with the LSTM model, by adding the information from time t − 1, the information from the time of t-skip is also involved in the input at current time t (i.e., x t ). So, the SS-RNN mathematical model can be written as follows: { f t = σ ( W f s k i p h t − s k i p + W f h h t − 1 + W f x x t + b f ) i t = σ ( W i s k i p h t − s k i p + W i h h t − 1 + W i x x t + b i ) c ˜ t = tanh ( W c ˜ s k i p h t − s k i p + W c ˜ h h t − 1 + W c ˜ x x t + b c ˜ ) c t = f t ⋅ c t − 1 + i t ⋅ c ˜ t o t = σ ( W o s k i p h t − s k i p + W o h h t − 1 + W o x x t + b o ) h t = o t ⋅ tanh ( c t ) (2) where W f s k i p , W i s k i p , W c ˜ s k i p , and W o s k i p are weight matrices for the corresponding inputs of the network activation functions, and h t − s k i p is the output of the moment t-skip.

Obviously, from the above model, there are two important issues to address: 1) information of which historical moments should be involved into the current moment? 2) how should the past information be involved into the current moment? To answer these two questions, we enumerated all the methods to add the historical information to the current recurrent unit. These methods can be divided into continuous addition and discontinuous addition. The last information input consists of adding directly and weight weighting and function mapping for calculation. There are in total six models (Models A–F used in this work) for the addition of historical information, shown in Figure 2, and the detailed descriptions can be seen below (also, refer to the Supplementary Material).

Model A The information of historical moments (t-skip) is directly added to the current moment (t) and the method is discontinuous (Figure 2A). The mathematical expressions of the LSTM cell can be written as follows: { f t = σ ( W f h h t − 1 + W f x x t + b f ) i t = σ ( W i h h t − 1 + W i x x t + b i ) c ˜ t = tanh ( W c ˜ h h t − 1 + W c ˜ x x t + b c ˜ ) c t = f t ⋅ c t − 1 + i t ⋅ c ˜ t o t = σ ( W o h h t − 1 + W o x x t + b o ) N = o t ⋅ tanh ( c t ) h t = { N + h t − skip , if   t = 1 + i × skip N , if   t ≠ 1 + i × skip (3) where skip is the order and i∈N+ (N+ is the set of positive integers); the part marked in bold indicates that the original formula has been changed. The order of Model A in Figure 2A is 3. For example, as shown in Figure 2A, when t = 4 with skip = 3, the input of recurrent unit h ₄ comes from h ₁ , h ₃ and x ₄ , and h ₁ is directly added to the original output of h ₄ to form a new output of h ₄ . Every three moments, additional historical information is added to the current moment.

Model B Similar to Model A, but the past information is added to the current moment after the transformation of the activation function by a weight of W n (Figure 2B). The corresponding mathematical expressions can be rewritten as: { M = W f h h t − 1 + W fskip ⋅ h t-skip f t = { σ ( W f x x t + b f + M ) , σ ( W f h h t − 1 + W f x x + b f ) , if   t = 1 + i × skip if   t ≠ 1 + i × skip N = W i h h t − 1 + W iskip ⋅ h t-skip i t = { σ ( W i x x t + b i + N ) , σ ( W i h h t − 1 + W i x x t + b i ) , if   t = 1 + i × skip if   t ≠ 1 + i × skip Q = W c ˜ h h t − 1 + W c ˜ skip ⋅ h t-skip c ˜ t = { tanh ( W c ˜ x x t + b c ˜ + Q ) , tanh ( W c ˜ h h t − 1 + W c ˜ x x t + b c ˜ ) , c t = f t ⋅ c t − 1 + i t ⋅ c ˜ t if   t = 1 + i × skip if   t ≠ 1 + i × skip R = W o h h t − 1 + W oskip ⋅ h t-skip o t = { σ ( W o x x t + b o + R ) , σ ( W o h h t − 1 + W o x x t + b o ) , h t = o t ⋅ tanh ( c t ) if   t = 1 + i × skip if   t ≠ 1 + i × skip (4)

When t = 4, the input of loop unit h ₄ comes from h ₁ , h _3, and x ₄ . After h ₁ is weighted, the function is transformed to add it to the current moment and form the output of new h ₄ .

Model C It continuously adds additional historical information to the current moment in a direct addition (Figure 2C). The corresponding mathematical expressions can be rewritten as: h t = o t ⋅ tanh ( c t ) + h t − skip (5)

The parts in bold represent changes to the original formula. The other part is the basic formula of LSTM. For example, when t = 4, the input of loop unit h₄ comes from h ₁ , h ₃ , and x ₄ , and h ₁ is directly added to the current moment to form the output of new h₄. Model C can be regarded as the general form of Model A. In both models, the additional historical information is calculated in the same way. Model A adds historical information intermittently, and Model C adds historical information continuously where every current moment adds the historical information of the moment of t-skip, and which leads to a greater computational complexity for the model.

Model D It continuously adds historical information to the current moment in the form of weight weighting and function mapping (Figure 2D). The corresponding mathematical expressions can be rewritten as: { f t = σ ( W f h h t − 1 + W f x x t + W fskip ⋅ h t − skip + b f ) i t = σ ( W i h h t − 1 + W i x x t + W iskip ⋅ h t − skip + b i ) c ˜ t = tanh ( W c ˜ h h t − 1 + W c ˜ x x t + W c ˜ skip ⋅ h t − skip + b c ˜ ) c t = f t ⋅ c t − 1 + i t ⋅ c ˜ t o t = σ ( W o h h t − 1 + W o x x t + W oskip ⋅ h t − skip + b o ) h t = o t ⋅ tanh ( c t ) (6)

When t = 4, the input of loop unit h ₄ comes from h ₁ , h ₃ , and x ₄ , and h ₁ is directly added to the current moment to form the output of new h ₄ . Model D can be regarded as the general form of Model B. In Model B and Model D, additional historical information is calculated in the same way, Model B adds historical information intermittently, and Model D adds historical information continuously.

Model E It intermittently adds additional historical information to the current moment in the form of weight weighting and function mapping (Figure 2E). The corresponding mathematical expressions can be rewritten as: { M = W f 1 h t − 1 + W f 2 h t − 2 + W f 3 h t − 3 f t = σ ( W f x x t + b f + M ) N = W i 1 h t − 1 + W i 2 h t − 2 + W i 3 h t − 3 i t = σ ( W i x x t + b i + N ) Q = W c ˜ 1 h t − 1 + W c ˜ 2 h t − 2 + W c ˜ 3 h t − 3 c ˜ t = tanh ( W c ˜ x x t + b c ˜ + Q ) c t = f t ⋅ c t − 1 + i t ⋅ c ˜ t R = W o 1 h t − 1 + W o 2 h t − 2 + W o 3 h t − 3 o t = σ ( W o x x t + b o + R ) h t = o t ⋅ tanh ( c t ) (7)

When t = 4, the input of loop unit h ₄ comes from h ₁ , h ₂ , h ₃ , and x ₄ , and h ₁ and h ₂ are added to the current moment through weight weighting and function mapping and constitutes the output of new h ₄ .

Model F It intermittently adds historical information to the current moment, and the historical information directly adds to the current moment (Figure 2F). The corresponding mathematical expressions after the improvement of LSTM can be rewritten as: { N = o t ⋅ tanh ( c t ) h t = { N + ∑ s = 2 skip h t − s , if   t = 1 + i × skip N , i f   t ≠ 1 + i × skip (8)

To test the effect of long-term memory introduced in this work on data classification, we first conduct experiments on three time series datasets (i.e., Arrhythmia dataset, Epilepsy dataset 1, and Epilepsy dataset 2). In addition, due to the potential correlations between the characteristics in some non-time-series biomedical data, we also perform experiments on two disease datasets: Diabetes dataset and Breast cancer dataset, to validate the ability of the model on non-time series data classification. Each dataset was split into training and testing set using the standard split. Table 1 summarizes the details of the five datasets.

Arrhythmia dataset It contains 109,338 recordings of 48 half-hour excerpts of two-channel ambulatory ECG, and which have been divided into five classes based on the heart rate: one normal and four abnormal.

Epilepsy datasets There are two well-known Epilepsy datasets used in this work. One is from the Department of Epilepsy at the University of Bonn, Germany, and which contains five categories (A–E) of 100 single-channel 23.6-s segments of electroencephalogram (EEG) signals (11,500 in total). The other is from Children’s Hospital Boston including 361,377 EEG recordings from 22 epileptic patients and these recordings have been grouped into two classes.

Diabetes dataset It contains 16 features, such as age, sex, and polyuria, and the source is from the University of California at Irvine Machine Learning Repository. This has been collected using direct questionnaires from the patients at Sylhet Diabetes Hospital in Sylhet, Bangladesh, and approved by a medical doctor.

Breast cancer dataset It contains nine features from UC Irvine Machine Learning Repository (see Supplementary Material).

The original five datasets are available through the websites listed in Table 1, and we also rearranged them for the convenience of use, and which can be found in the Supplementary Material.

For the classification task, the models are evaluated by the classification accuracy, precision, recall, and F1-score, which are defined by the confusion matrix. It is one of the most intuitive metrics used for evaluating the performance and accuracy of the model in machine learning, especially used for classification problems. The terms associated with confusion matrix can be defined as follows: True positives (TP), when the actual class of the data point is 1 and the predicted outcome is also 1. True negatives (TN) are the cases when the actual class of the given data point is 0 and the predicted result is also 0. False positives (FP) are the cases when the actual class of the data point is 0 and the predicted outcome is 1, which can be assumed that the model predicts incorrectly as the actual class is positive. False negatives (FN) are the cases when the actual class should be 1 and the predicted outcome is 0, where the model predicts incorrectly as negative. The forms are expressed as follows: { A c c u r a c y = T P + T N T P + F P + F N + T N P r e c i s i o n = T P T P + F P R e c a l l = T P T P + F N F 1 − s c o r e = 2 × P r e c i s i o n × R e c a l l P r e c i s i o n + R e c a l l (9)

In SS-RNN, the information of historical moments (e.g., t-skip) can be added to the current moment (i.e., t) to accurately classify sequential data with long-term dependences. To determine the best methods of the past information addition and verify the effectiveness of the SS-RNN model, we did six groups of comparison experiments on five datasets, respectively. The six different models (Models A–F) and five datasets are shown in Theoretical Model Analysis and Data Collection. For each experiment, there are three steps: data preprocessing, training, and test.

Data preprocessing Outliers and missing values often appear in the dataset, whereas the network model cannot process those data samples. We first fill the missing values with the mean of the variable and delete the samples with outliers, which can be judged from the method of Anomaly Detection. The pre-processed time series datasets (e.g., Arrhythmia and Epilepsy datasets in Table 1) can be directly input into the SS-LSTM model. However, for the non-time series data with multiple features and different dimensions (e.g., Diabetes and Breast cancer datasets used in this work), after the above preprocessing, it needs to be fed into the feature extractor to obtain a new set of data and their characters, which can be further transformed to a matrix of 32*4 as input into the SS-LSTM. Taking the Diabetes dataset as an example, we also give detailed descriptions in the Supplementary Material.

Training For each dataset (Table 1), the training set is used to train the model. The optimized parameters of the network are as follows: dimensions of the network are 128, 64, 32, and 16, respectively (Figure 1A). For each dataset, the configuration of the SS-LSTM model is implemented in Pytorch using Eqs 3–8, and the dimensions for the three layers of the SS-LSTM model are 18, 8, and 5, respectively. The activation function is tanh, and the training algorithm is stochastic gradient descent with a learning rate of 0.01 and a training epoch of 50. Here, we used the cross-entropy loss as the objective function for training the network: Loss = − ∑ i = 1 n y i ⁡ log ( y i ^ ) (10) where y i is the true value, and y i ^ is corresponding predicted value. The batch size of each dataset after fine-tuning is shown in the Supplementary Material.

Test For each dataset, 25 different comparative experiments were performed using different structures of LSTM. One of the experiments adopted the ordinary LSTM, while the others used SS-LSTM with different models (i.e., Models A–F in Figure 2). For each model, the values of skip were set as 2, 3, 4 and 5, respectively. Furthermore, we also used the other classical models (LSTM, GRU and Bi-LSTM) to create the classification set for three of the datasets (i.e., Arrhythmia, Epilepsy 1 and Diabetes), and made a comparison with our SS-RNN model.

To test the effect of the addition of past information on the data classification, we used our network with six different SS-LSTM models (Models A–F; Figure 2) to classify the data for five datasets (Table 1), respectively. For each SS-LSTM model, different values of skip (e.g., skip = 2, 3, 4, 5) were used. As shown in Supplementary Figures S1–S10, S12–S17, S19–S24, S26–S31 in the Supplementary Material, the loss functions calculated by Eq. 10 for the experiments in this work always converged before 50 steps, indicating that 50 steps are sufficient for the training and test processes.

Furthermore, we also made classifications for three of the datasets (Arrhythmia dataset, Epilepsy dataset 1, and Diabetes dataset) by using the classical networks such as LSTM, GRU, and Bi-LSTM with default parameters of the torch.nn module, and compared the results with that from the SS-RNN with Model A of skip = 3 (Figure 6; Tables 2–4). We also show the simulation results of other indexes with Model A of skip = 3 and Model C of skip = 5 in the Supplementary Material.

From Figure 6, it shows that our SS-RNN method can improve the classification accuracy as compared to the classical methods. Also, from Tables 2–4, it can be found that the other main indexes are almost improved. At the same time, we compared our method with the latest methods RNN, RNN+GRU, RNN+LSTM, and MCNN (Zhang et al., 2017; Singh et al., 2018) with the same Arrhythmia dataset. The result is shown in Table 5, which also indicates that our SS-RNN method can improve the classification accuracy.

In fact, as a variant of LSTM, GRU reduces the forget gate and input gate, and adds the update gate. GRU has simpler internal structure and less parameters than LSTM, which reduces the risk of overfitting. Although LSTM and GRU partially solve the problem of the vanishing gradient of the RNN, the information loss is still very severe in the propagation of a very long distance. Bi-LSTM, namely, bi-directional LSTM, does not change any internal structure of LSTM itself. LSTM is applied twice in different directions, and then the LSTM results obtained twice are spliced as the final output. For datasets with both forward and backward dependencies, this method can enhance the correlation between data and improve the efficiency of the model. It is often used to capture some specific pre or post features of language and syntax in natural language processing. However, in biological datasets like ECG and EEG, the progression and onset of diseases are irreversible, so the relationship between data in the reverse time direction is not of practical significance for disease classification. In addition, excessive number of parameters may lead to overfitting in network training, so the Bi-LSTM model is not suitable here.

The long-term memory ability of LSTM and GRU is weak, and with the increase of the time step, the farther away the memory, the more information the model forgot and the less it remembered. Our model has enhanced the information in distant moments, which makes up for the defect of long-term dependence in RNNs. Therefore, our SS-RNN method can improve the precision, recall, F1-score, and finally improve the classification accuracy of sequential data compared with other models.