To fill the research gap mentioned above, in this study, a convenient and efficient end-to-end BCI framework for MWL estimation was proposed. The contributions of the article can be summarized as follows:

First, our proposed end-to-end BCI framework for workers' MWL estimation, which decodes mental workload related relevant information directly from raw EEG, is able to avoid the time consumption associated with complex feature extraction and thus meet the hardware requirements of brain-computer interface systems.

Second, this method uses a combination of filters, ASR, and ICA with ADJUST to obtain relatively pure EEG signals without manual involvement. Additionally, the MWL related neural information is decoded smoothly from the original EEG by the designed time fixed 3-D-CNN layers while the temporal dimension is unchanged. Then the following bi-LSTM layers can be used to extract temporal features.

Third, according to two types of comparison experiments and ablation studies, the estimation effectiveness of EEG-TNet can be proved.

In this study, convolutional layers include four normal convolutional layers, a depthwise convolutional, and a pointwise convolutional. For the normal convolutional, the input of the convolutional layer is X_in(C_in, D_in, H_in, W_in), and the output y_out(C_out, D_out, H_out, W_out). The formula of the convolutional layer can be described as follows:

Where the ⋆ is the valid 3D cross-correlation operation. The shape of y_out(C_out, D_out, H_out, W_out) can be calculated according to the kernel size (K_D, K_H, K_W) and the kernel number C_out. Specifically, the depthwise separable convolution (Chollet, 2017) which consists of depthwise convolutional and pointwise convolutional was used in our research to extract spatial information from EEG with a lower number of convolutional parameters (Chollet, 2017).

By designing time-fixed 3D convolutional layers, we retain the EEG information in each time step and further analyze the temporal information using LSTM networks (Hochreiter and Schmidhuber, 1997). Recurrent neural networks (RNN) have an excellent memory capability owing to their distinctive self-connected structure, which has an absolute advantage in processing temporal data (Mikolov et al., 2010). The LSTM network is a popular expansion of RNN to address the gradient disappearance problem while RNN only processes long-term data. The LSTM introduces a gating mechanism to control the rate of accumulation of information, including adding new information and forgetting previous information by using the gates. There are three gates including input gate i_t, forget gate f_t, cell gate g_t, and output gate o_t, respectively. Specifically, the forget gate f_t controls the rates of previous information required to be forgotten about the internal state c_t−1 at the last moment.

The input gate i_t determines the rates of new information which is allowed to be added to the current c_t. Two steps are required to achieve this. First, calculate the input gate i_t and cell gate g_t are calculated. Second, update memory cells C_t by combining forgetting gates f_t and input gates i_t.

Ultimately, we need to determine the output, which is based on the state of our memory cells C_t. First, a sigmoid layer is used to determine which parts of the memory cell state will be output. Second, the memory cell is processed through tanh and multiplied by o_t.

The fully connected layer serves as an “estimator” in the entire neural network structure. The procedures such as convolutional layers, pooling, LSTM, and activation function translate the original data to the hidden feature space. The fully connected layer transfers them to the sample labeling space. As Equation (8) shows, the fully connected layer multiplies the weight matrix with the input vector and then adds the bias.

where A^T is the learnable parameter and b is the bias. In addition, a softmax activation function may be used to calculate the likely distribution of the output classes. In the final FC layer, the softmax function is utilized, which is defined as follows:

where i is the input vector, the output S_i is between o to 1, and ∑iSi=1

The database used in this study is STEW (Lim et al., 2018), which contains EEG data of 48 subjects under different MWL levels. Specifically, the subjects performed the Simultaneous Capacity (SIMKAP) test to induce MWL. After the test, all the subjects were required to finish the subjective questionnaire to report their MWL, which is a 9-point rating scale. During the whole experiments, the EEG signals were recorded using an Emotiv EPOC EEG headset with 14 electrodes (AF3, F7, F3, FC5, T7, P7, O1, O2, P8, T8, FC6, F4, F8, AF4) and two reference channels (CMS, DRL). The sampling frequency was 128 Hz and the resolution was 16-bit A/D. In this study, the classifier was proposed to finish two tasks, the first one is classified “No Task” vs. “SIMKAP Task”, which was a binary classification. The second task was classifying Low vs. Moderate vs. High MWL, which was divided by a rating scale. A detailed definition of the label can be found in the article (Lim et al., 2018).

To meet the requirements of the automation process, we eliminated parts of the preprocessing process that require manual intervention, such as manual artifact removal and manual judgment of ICA components to remove artifacts, especially eye movement artifacts (Fan et al., 2021; Peng et al., 2021). This undoubtedly reduces the quality of the data and, therefore, the accuracy of the recognition, but it makes sense for real-world applications (Rosanne et al., 2021). Table 1 shows the comparison of traditional preprocessing steps and ours. The whole preprocessing steps are

The architecture of our EEG-TNet framework is inspired by the network architecture EEGNet of Lawhern et al. (2018), which is a widely used end-to-end EEG BCI framework. The detailed framework of our proposed EEG-TNet model can be summarized in three steps, which are shown in Figure 2. Step 1 is to segment the raw EEG to the required size and expand a new dimension for the model need, which is used to keep the temporal dimension stable. Step 2 is to extract the temporal and spatial information from each EEG fragment without between-fragments temporal information loss according to the designed temporal fixed 3-D-CNN layers. Step 3 is to extract the temporal information between each EEG fragment by using the LSTM layer. The output of the last time step in the last layer is used to compute the final status according to the fully connected layer and the softmax function.

In the subject-dependent experiment, we adopt a similar experimental protocol as that of Chakladar et al. (2020); Kingphai and Moshfeghi (2021); Zhu et al. (2021). The five-fold cross-validation method was applied to evaluate the performance of the framework. As shown in Figure 2, specifically, the first 80% of all samples were selected as the training set and the remaining 20% of the samples were kept aside as the test dataset. Then the other 20% of the samples were selected as the second test set, the last 80% of samples were the second training set. Divide all samples like this five times and the average accuracy of the five experiments was taken as the final result. In addition, five-fold cross-validation was also applied to find optimal hyperparameters.

The proposed EEG-TNet framework was compared with the other four baseline methods for MWL estimation on the STEW dataset under the subject-dependent experiments setting, as shown in Table 2. The results showed that the proposed EEG-TNet framework achieved higher estimation accuracies than the other four methods. Although most of the recent studies use many kinds of features like frequency features [PSD (Chakladar et al., 2020; Kingphai and Moshfeghi, 2021)], non-linear features [Approximate Entropy (ApEn) (Chakladar et al., 2020; Kingphai and Moshfeghi, 2021)], linear features [autoregressive coefficient (AR) (Chakladar et al., 2020)], and the graph-based features (clustering coefficient, mean degree) (Zhu et al., 2021). However, traditional machine learning models (SVM and random forest cannot learn the full EEG information. Moreover, some combined deep neural networks (CNN-LSTM, BLSTM-LSTM) show better performance than machine learning models, it is still well below our proposed model.

The leave-subject-out (LSO) cross-validation method was used to evaluate the performance of the proposed framework in the subject-independent experiments. As shown in Figure 3, in the LSO cross-validation experimental protocol, the EEG samples of 36 subjects (80% of a total of 45 subjects) were selected for training the model and the last EEG samples of 8 subjects (20% of a total of 45 subjects) were used to test the model performance. The whole process was repeated five times so that all the subjects' samples were taken as the test set. The average accuracy of the five experiments was taken as the final result. Similarly, the hyperparameters were found according to the five-fold cross-validation in the training set.

Compared with the subject-dependent experiments, there are fewer studies that perform the subject-independent experiments because of their difficulty. The proposed EEG-TNet framework was compared with the other three baseline methods under the subject-independent experiments setting, as shown in Table 3. It is worth noting that no studies were conducted with either dual-task estimation (No Task vs. Task) or triple task estimation (Lo vs. Mi vs. Hi) subject-independent experiments simultaneously until now (Lim et al., 2018; Pandey et al., 2020).

Table 3 summarized the comparative results regarding the average estimation accuracy under the subject-independent experiments setting. Pandey et al. (2020) realized that handcrafted features would slow down the speed of computing and significantly increased the evaluation time, making it challenging to apply them in practical scenarios. Therefore, he used an end-to-end structure similar to ours to finish the dual-task estimation(No Task vs. Task). However, the results were unsatisfactory, with his best assessment only reaching 61.08%, which is only a tiny improvement over the random classification (50%).

For practical reasons, we need to focus more on triple-task estimation (Lo vs. Mi vs. Hi) than on the dual-task estimation (No Task vs. Task). Lim et al. (2018) contributed the STEW dataset, where he extracted the PSD of different bans as features and used SVM as a classifier. Although their recognition results were slightly higher than our results, comparing the confusion matrices shows that our estimation results are more balanced and valid. As Figure 4 shows, the accuracy of their results was 99.54% for low MWL, down to 46.15% for medium workloads and only 31.07% for high MWL, which was even lower than the random results (33.33%). In contrast, our estimation accuracies range from 52.00 to 74.72%. On the most difficult estimation task with high MWL, our results were nearly twice as good as theirs.

The main objective of this study is to propose a practical and effective MWL estimation method for workers of special occupational groups, which can be used to ensure safety during the course of their work. EEG signals are regarded as the gold standard and BCI systems based on EEG signals have natural advantages. However, most BCI systems cannot meet the requirements of online evaluation due to the high manual involvement in signal noise reduction, complex and time-consuming feature extraction, and other disadvantages. In order to solve the disadvantage of manual involvement in signal noise reduction, this system uses a combination of filters, ASR, and ICA with ADJUST to obtain relatively pure EEG signals without manual involvement.

The classification model is at the heart of the BCI system, because of the system's computing time, deployment method, and evaluation accuracy depends on it. In most related BCI systems, the handcrafted features were used to estimate the MWL, which may be computationally demanding and not suitable for a real-time system. This study exploited deep neural networks' powerful feature extraction and classification capabilities to design the EEG-TNet network as the computational core of an end-to-end BCI framework. This study improves the traditional neural network model named EEG-Net and extracts features through the processes of data segmentation, dimension expansion, time fixed 3-D-CNN layers, Bi-LSTM layer, fully connected layer, and softmax operation. The time-fixed method was designed to ensure the temporal segment order, and a Bi-LSTM layer was added at the end for temporal information analysis. Moreover, the total time cost of this model is only 386.74 ms in our machine [System: Ubuntu 20.04, CPU: Intel(R) Core(TM) i7-8700K CPU @ 3.70GHz, Memory: 32 GB, GPU: GeForce RTX 2080Ti]. The low time cost proves that the proposed EEG-TNet can meet the requirements of real-time application.

The most important metric for evaluating a model is estimation accuracy. Unlike other research areas, both subject-dependent experiments and subject-independent experiments need to be considered in the field of human factors engineering. In most cases, subject-dependent experiments are more accurate because such experimental methods allow the model to obtain EEG data for each individual during the training phase. Information on individual differences can be extracted. As shown in Table 2, most studies have achieved more than 80% or even more than 90% recognition accuracy, while our recognition accuracy is close to 100%.

However, the subject-dependent experimental approach is often not applicable to practical scenarios. Training a unique classification model for each worker would be time-consuming and costly, so subject-independent experiments use unseen subjects' data as the test set, satisfying the need for “Plug-and-Play”. However, head shape, scalp impedance, and psychological state can all affect the EEG data, resulting in large variations in EEG data among subjects. The accuracy of the method is poor. Therefore, as shown in Table 3, almost all of the previous methods do not apply to estimating the MWL of workers. In the dual-task estimation, the method proposed by Pandey et al. (2020) was only marginally more accurate than random. In contrast, our method was able to achieve 82.78%, which is sufficient for use in realistic scenarios.

The triple task estimation(Lo vs. Mi vs. Hi) is the most urgent from the point of view of ensuring safety in different fields like transportation and construction. Accurate assessment of the high or moderate MWL of workers helps managers to allocate tasks rationally and to avoid overloading workers with work that could lead to human factor accidents. Although the estimation accuracy of the method proposed by Lim et al. (2018) appears to be slightly higher than our proposed EEG-TNet method. By analyzing and comparing the confusion matrices of the two methods in Figure 4, the method of Lim et al. (2018) may not apply to worker workload estimation. According to their confusion matrix, we can find that the low MWL was assessed at 99.54%, which means that almost all low load situations were successfully identified. However, the estimation accuracy for moderate MWL dropped to 46.15%, and only 31.07% of the samples with high MWL were correctly estimated, with an accuracy rate even lower than the random results (33.33%). More notably, 37.57% of the medium workload samples and 38.40% of the high MWL samples were misclassified as low MWL. In many occupational fields, underestimating workers' workload by managers can lead to their scheduling of excessive workloads, leading to workers being overloaded, making the chances of unsafe behavior much higher. In our assessment results, just 5.25% of the moderately loaded and 19.78% of the high loaded sample were incorrectly underestimated as low workloads. Compared to Lim et al. (2018), the likelihood of underestimation was 7 times and 2 times lower, respectively.

To verify the effectiveness of our model, ablation experiments are performed on the STEW database. There are two kinds of ablation experiments: (1) Ablation experiments on the effectiveness of designed automated preprocessing method (2) Ablation experiments on the effectiveness of Bi-LSTM. As Figure 5 shows, estimation accuracy significantly decreased when the data pre-processing process or LSTM layer was removed, not only in subject-dependent experiments but also in subject-independent experiments.

However, there were some limitations to this study. First, this study used the multi-channel EEG to build the EEG-TNet model. However, collecting multi-channel EEG needs gel-based EEG caps or clumsy dry electrode EEG caps, which is too troublesome to fill the practical usage. Additionally, the increase in the number of channels also brings a significant increase in computational complexity. Furthermore, although the estimation accuracy for high MWL is much larger than the previous study, it is still not enough for application scenarios. Finally, the STEW database only contains 48 students' EEG data under different levels of MWL, which were not selected to be representative. It is important to analyze the EEG signals of different occupational groups, ages, and work experiences.