There exists a rich body of research pertaining to NIRS-based brain activity and emotional state classification. Naito et al. (2007) present communication means for patients struggling with amyotrophic lateral sclerosis (ALS) using quadratic discriminant analysis (QDA). Their model utilizes maximum amplitude and phase change as features to achieve an average accuracy of 80% on binary “yes/no” answers of forty male and female patients. Tai and Chau (2009) compares the performance of linear discriminant analysis (LDA) and support vector machine (SVM) on NIRS signals associated with the single-trail classification of the positively and negatively induced emotional tasks at individual level. Their results suggest that classification accuracy of these models vary with the length of the input signals. Luu and Chau (2009) apply linear discriminant analysis (LDA) on mean signal amplitude of NIRS data of nine human subjects to achieve an average accuracy of 80% on evaluating the choice of drinks among two available options in a single-trial scenario. Cui et al. (2010c) apply linear SVM on NIRS-related finger tapping task performed by six participants. Furthermore, they present an insightful investigation of the effect of the different feature spaces on classification accuracy. Their results suggest that features that provide the best classification for one dataset may not be optimal for all NIRS data, thereby suggesting their further optimization for individual participants. Holper and Wolf (2011) apply Fisher's linear discriminant analysis (FLDA) on motor imagery tasks of simple and sequential finger-tapping to report an average classification accuracy of 81.0% that is computed based on the classification performance of FLDA on NIRS data of the participants at the individual level. Hu et al. (2012) utilize contrast-to-noise ratio (CNR) as feature to decode deception on eight male subjects. They report classification accuracies of 83.44 and 81.14% using RBF and linear support vector machines (SVM), respectively. Furthermore, the accuracy of their model increases to 87.5% when applying their approach on an inter-subject setting (seven out of eight subjects). Naseer and Hong (2013a) use LDA on mean and slope of NIRS data as features to perform a left- and right-motor imagery by ten participants. Their approach achieves 73.35 and 83.0% accuracies on right- and left-wrist imagery tasks, respectively. Furthermore, they report an improvement in accuracy of their model by focusing on 2–7 s out of entire 10 s trials while extracting features, achieving average accuracies of 77.56 and 87.28% for right and left wrists, respectively. Herff et al. (2013a) apply LDA for binary discrimination between relax state and three different tasks (i.e., mental arithmetic, mental rotation, and word generation). They obtain 71% accuracy on mental arithmetic, 62% accuracy on mental rotation task, and 70% accuracy on word generation with respect to relax state on ten subjects. Nguyen et al. (2013) compare the performance of SVM in contrast with one-hidden-layer artificial neural network (ANN) for two hands tapping tasks performed by three human subjects. They use polynomial regression coefficients as features to report best average accuracy of 82.5% on right and left hands tapping of these subjects, using SVM. Furthermore, they obtain 85% on right and 75% on left hands tapping, using ANN. Herff et al. (2014) use fNIRS data along with LDA to classify between n-back (n ∈ {1, 2, 3}) and resting state to achieve up to 78% accuracy for single-trail discrimination. Naseer et al. (2014) compare the performance of LDA and SVM on online binary classification of mental yes/no answers (i.e., performing mental arithmetic vs. relax state in response to given questions) to report average classification accuracies of 74.28 and 82.14%, given the performance of these classifiers at the individual level. Xu et al. (2014) adopt χ² statistic for feature extraction through discretization of NIRS data and apply linear SVM to achieve classification accuracy of 69–81% on right hand clench force motor imagery and clench speed motor imagery on six subjects. This article presents a useful literature review on the topic as well. Naseer and Hong (2015a) apply multi-class LDA for classification of the motor imagery based responses to four-choice questions (e.g., left-hand motor imagery to indicate option A) to report an accuracy of 73.3%, averaged on performance of their classifier at the individual level. Hong et al. (2015) use mean and slope of NIRS signal and multi-class LDA to classify between mental arithmetic, left hand motor imagery, and right hand motor imagery. They report an average accuracy of 75.6% on ten participants. Naseer et al. (2016) study the choice of optimal feature selection for binary classification of mental arithmetic and relax states, using LDA. Their results indicate that combination of the mean and the peak values of the signals associated with the individuals result in a significant improvement of the accuracy of their classifier. Naseer and Hong (2015b) present a comprehensive review of this topic.

Despite impressive and promising results on classification of different brain activities using NIRS and fNIRS time series, all aforementioned approaches unanimously focus on improvement of the performance of different classification approaches at the individual (i.e., intra-subject) level, reporting their results that are averaged on the performance of these classifiers on single-participant basis. The major drawback of such an evaluation paradigm is the strong dependency of the accuracy of the adapted model on the performance of the individuals, thereby exhibiting high variation/bias. More specifically, there is a paucity of research on modeling and study of classification approaches that aim for generalization and scalability. Our approach addresses this issue via training on combined data of all participants (i.e., inter-subject level), thereby narrowing the gap between intra- and inter-subject brain activity prediction. It is apparent that such an approach facilitates the deployment and integration of these models in real-time systems since their learning mechanism is independent of the individual that they interact with.

Kamran and Hong (2014) argue that the NIRS time series data is a linear combination of various components, ranging from dynamical characteristics of the oxy-(HbO) and deoxy-hemoglobin (HbR) changes in a specific brain region and the influence from previous stimuli, to the physiological signals that prevail such time series data, and the baseline effect. This claim is further supported by Cui et al. (2010c) whose comparative analysis suggest that the slope (i.e., a linear correlate) of the NIRS data forms an important and highly informative feature in comparison to various feature spaces. These results explain the emergence of linear classifiers as dominant approaches to brain activity detection based on NIRS time series as presented in Section 1.1.

We take this observations and results into consideration while formulating our novel approach to brain activity prediction. In cognitive psychology, cognitive load refers to the total amount of mental effort utilized by the working memory while conducting a mental activity (Sweller, 1988). As such, the mental workload classification refers to the ability to distinguish between various level of brain activity that are pertinent to the same family of working memory through mathematical modeling of their corresponding time series data. In particular, we address the prediction of n-back task (Kirchner, 1958) as a proxy measure of mental workload. The n-back task is a continuous performance assessment, frequently used in cognitive neuroscience, to measure the working memory capacity (Gazzaniga et al., 2014). In this setting, the human participant is presented with a sequence of stimuli and the task consists of indicating when the current stimulus matches the one from n steps earlier in the sequence. The simple operational principles of such tasks provide opportunity to model changes in mental workload of human subjects through analysis of the effect of their level of difficulty on NIRS-related patterns of brain activity. Our study and its subsequent results focus on training a model on labeled data of human participants performing 1- and 2-back tasks, thereby distinguishing between these tasks during their prediction to infer the level of task difficulty based on its effect on mental workload. Our contributions are as follows:

The remainder of this article is organized as follows. We elaborate on formulation of our approach in Section 2. Section 3 provides details on data acquisition and experimental setup along with the data preprocessing and feature extraction steps. Results and comparative study of our model in contrast with state-of-the-art techniques in NIRS literature are presented in Section 4. Section 6 presents conclusion and some insight on the future direction of this research.

Let X represent the input feature matrix that corresponds to the combined data of task spaces 𝕋₁ and 𝕋₂. Furthermore, let y be the row vector associated with X whose ith row entry represent the label of the ith feature vector in X. The weight vector that maps X onto y using the normal equation is (Cormen et al., 2001):

Let x→1∈𝕋1 and x→2∈𝕋2 be the geometric medians of task spaces 𝕋₁ and 𝕋₂, respectively. We compute the midpoint of these task spaces as a mean of their corresponding geometric medians with respect to their coordinates (i.e., their respective feature vectors):

with χx→i(j) being the jth coordinate (i.e., feature) of geometric median of the ith task space, i.e., x→i∈𝕋i, i=1,2 and |.| returns the cardinality of its argument. Given x→ and the weights W in Equation (3), the new Sigmoidal boundary condition for 𝕋_i, i = 1, 2 is:

i.e., new boundary condition, β, is obtained through application of Sigmoid activation function on inner product of weight vector W and midpoint x→. We utilize β to predict the labels of new data as:

with p→ being the new feature vector associated with the recently generated input NIRS data.

Claim 2.3. The midpoint of the geometric medians of the two task spaces 𝕋₁ and 𝕋₂, defines the most linearly optimal boundary between them [please refer to Appendix 5 (Supplementary Material) for the proof].

Twenty-eight healthy right-handed volunteers (11 male and 17 female, M = 30.96 years, SD = 10.84) participated in the experiment. Prior to data collection, we received approval from the ethical committee at Advanced Telecommunications Research Institute International (Approval Code: 16-601-1), along with the informed consent from all participants. The data is acquired with a wearable optical topography system “HOT-1000,” developed by Hitachi High-Technologies Corporation (please refer to Figure 1). It is wore on the forehead of participants and collects data through four channels (i.e., Left₁, Left₃, Right₁, and Right₃, as shown in Figure 1). Furthermore, it allows for recording of the measurement of brain activity through detection of total blood flow via emitting a wavelength laser light (810 nm) at the 10 Hz sampling rate. The participants were requested to sit in front of a screen where the on-screen 1- and 2-back instructions (please refer to Section 1.2 for details) are presented. We use the FLANDERS (Nicholls et al., 2013) handedness questionnaire to measure the skilled hand preference of the participants. After a resting period of 1 min, they were instructed to focus on the voice, listening to a sequence of numbers in two separate tasks, clicking on the left mouse bottom if they recognize a repeated number meeting the 1- or 2-back repetition in the first and the second tasks, respectively.

First, we normalize the data corresponding to the four NIRS channels via subtracting the mean of the 1 min resting period as a baseline from this data. Next, we apply a 5-degree polynomial butter worth filter on each channel with 0.01 and 0.6 Hz for low and high bandpass, respectively. This is followed by linear detrending of the time series signals associated with each of these four channels. Lastly, we apply a 2-degree polynomial non-linear detrending.

It is customary in NIRS data preprocessing to apply segmentation on the original data of participants, thereby increasing the size of samples that are, in most cases, small. However, we strongly believe that such segmentations degrade the performance of any supervised classifier, preventing its true accuracy to be estimated. Figure 2 shows the Euclidean norm distribution of NIRS data associated with 1-back (red-colored circles) and 2-back (circles in blue) tasks of seven randomly selected female participants in our study. In this figure, there are a number of participants whose data do not follow the general trend, namely, having their 2-back Euclidean norms above those associated with 1-back task. Although such misleading data are customary in many applications, segmentation of such cases introduces a rather redundant source of misclassification by prediction models. In fact, the negative effect of segmentation on estimation of true accuracy of any supervised classifier is significant as shown below.

Theorem 3.1. Segmentation reduces the accuracy of any supervised classifier by a factor that is exponential to the depth of segmentation [please refer to Appendix 5 (Supplementary Material) for the proof].

Corollary 3.1.1. Segmentation reduces the accuracy of any supervised classifier by 12×s(d-1) in worst case scenario [please refer to Appendix 5 (Supplementary Material) for the proof].

Features can be directly extracted from raw NIRS data (Power et al., 2010). Alternatively, they are extracted from data after its transformation into hemoglobin concentration using Beer-Lamberts law (Hong et al., 2015). Luu and Chau (2009) show that effect of these two feature extraction strategies on prediction accuracy is insignificant. Moreover, Power et al. (2010) argues that the use of raw data for extracting features facilitates the integration of models into real world setting due to its less computational intensity. We adapt the same perspective for feature extraction in this article.

We compute separate sets of identical features for each of the four channels of our NIRS data. More specifically, we extract mean and slope of the signal (i.e., SM and SS, respectively) (Hong et al., 2015), contrast-to-noise-ratio (CNR) (Hu et al., 2012), and the moving average (Luu and Chau, 2009). In addition, we calculate the differential entropy (DE) of the data associated with these channels [please refer to Appnedix 6 (Supplementary Material)]. Although, DE is used as a feature in classification of brain activity and emotional state estimation based on electroencephalogram (EEG) data (Herff et al., 2013b; Shi et al., 2013; Kumaran et al., 2016a), this is the first time, to the best of our knowledge, that it is utilized for NIRS-based brain activity prediction. While calculating features, we divide the stream of NIRS data that correspond to each channel into four equal length sub-streams. Next, we compute the respective features for each of these sub-streams. This result in a four-dimensional feature vector in case of CNR, and DE for every channel. It is apparent that it is an eight-dimensional vector in case of SM and SS.

We compare the performance of our approach in contrast with the prominent state-of-the-art techniques in NIRS-based classification. An overview of the literature pertinent to NIRS-based classification reveals that linear discriminant analysis (LDA) (Herff et al., 2013a; Naseer and Hong, 2013b; Hong et al., 2015), linear support vector machine (SVM) (Cui et al., 2010c; Hu et al., 2012; Hai et al., 2013), and quadratic discriminant analysis (QDA) (Naito et al., 2007) are dominant approaches that are adopted by the research community in this field. This is mainly due to the underlying linear trends of various components (e.g., oxy-(HbO) and deoxy-hemoglobin (HbR) changes in a specific brain region, etc.) that form the NIRS data time series (Cui et al., 2010c; Kamran and Hong, 2014). However, in addition to these methodologies, we include the comparative analysis of our approach in contrast with logistic regression (Freedman, 2009), RBF SVM (Chang et al., 2010), k-nearest-neighbor (KNN) (Fix and Hodges, 1951), decision tree (Breiman et al., 1984), random forest (Shi et al., 1995), and Naive Bayes (Stuart and Norvig, 2003) algorithms to ensure a thorough analysis of the performance of our model. We use Python scikit-learn¹ package for this purpose. It is worth noting that the best setting of the parameters of these models are K = 3, d = 3, n = 10, c = 1e5 for number of neighbors in KNN, depth in decision tree, number of estimators in random forest, and penalty term in logistic regression, respectively. Furthermore, the penalty terms for linear and RBF SVM are c = 0.025 and c = 1.0, respectively.

Algorithm 1 summarizes the procedure for acquiring the average prediction accuracy of a given classifier during the experiment. More specifically, we adopt a percentage-wise, N-Fold cross-validation strategy, starting with assigning 90% of total number of participants in random and without replacement (indicated by split variable) for training and finishing with splitting the data into half between train and test sets in a 5% countdown steps (line 14) which results in nine times of splitting in total. While assigning subjects for training and testing, we ensure that all data corresponding to a given participant is entirely assigned to one of these two sets, thereby preventing any potential similarity and/or shared representation of the individual information affecting/biasing the prediction accuracy of the classifiers. For each of these splits, we perform the prediction by a given classifier, ℂ, and collect its estimate, for a total number of 20 rounds (i.e., lines 6 through 9). We follow these procedure for every calculated feature (please refer to Section 3.3) and on every four NIRS channels. Finally, we report the best average prediction accuracy of the classifier, along with the type of feature and the channel leading to this result.

It is worth noting that we include an additional step in case of our model to compute the best number of polynomial features to our model (i.e., line 9 in this algorithm). More specifically, we add a brute-force step in Algorithm 1 to add a polynomial feature to the input feature matrix X in Equation (3). The degree of this polynomial features is selected from the range [0, 12] with 0, indicating the original feature matrix X and without addition of any polynomial feature. We can afford this extra polynomial degree evaluation on our model due to its overall low-cost computational complexity, as outlined in Appendix 8 (Supplementary Material). Considering this procedure, there are 4(channels)×4(features)×9(random splits)×20(repetition of random splits) = 2880 steps involved to obtain the accuracy of each of the comparative classifiers. These steps increase to 2880 × 12(polynomial feature selection) = 34, 560 in case of our approach [indicated as SNC i.e., Sigmoid-Normal form Classifier due to the normal form regression in Equation (3)].

This section provides details on performance results of our proposed approach in comparison with the selected classification strategies, outlined in Section 4.1. We present the average prediction accuracy of these techniques that are acquired through the steps described in Algorithm 1, along with the precision, the recall, and the F1-score of these classifiers. In addition, we outline the channel type and the type of feature, leading to their best average performances, respectively. Furthermore, we apply statistical analysis on these results to realize the degree of statistical significance in their performance differences. While conducting these comparative analyses, we consider three different settings of data, thereby empirically investigating the effect of gender difference on verbal working memory task (Li et al., 2010). More specifically, we consider the data associated with female only, data associated with male only, and the combined data of male and female participants.

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.