Sustained attention is an umbrella term used in the field of cognitive psychology generally referring to a subject's readiness to detect unpredictably occurring signals over prolonged periods (Sarter et al., 2001). Studies dating back from over a century have addressed the difficulties of remaining vigilant when performing trivial or repetitive tasks (Bills, 1931a,b; Thorndike, 1912). Since then, several authors have explored the pitfalls of human inattentiveness in a variety of contexts, from driving accidents (National Highway Traffic Safety Administration, 2020) to in-class pedagogical activities (Bligh, 2000; Wang et al., 2021). This remains an active research topic in experimental psychology, with ongoing efforts to identify effective strategies to mitigate decrements in vigilance and task engagement (Massar et al., 2016; Hopstaken et al., 2015a,b; Esterman et al., 2016).

In a recent study, Robison et al. (2021) examined the specific effects of goal-setting, feedback, and incentives to counter attention deficits in a psychomotor vigilance task. Although some of the experimental manipulations were effective in mitigating slower Reaction Times (RTs) at later trials, none eliminated it. Specifically, both providing participants with a specific goal to strive toward and giving them performance feedback mitigated the a worsening of task performance across time. Further, participants in these conditions self-reported feeling more motivated and exhibited greater evoked pupillary responses. Traditionally, pupil diameter has been used as a measure of attentional effort (Beatty, 1982). In recent years, pupillometry has become an increasingly common psychophysiological tool, as we learn more about its neurobiological derivations and the psychological processes it accompanies. For example, it has been demonstrated that the pupil is sensitive to cognitive conflict (van der Wel and van Steenbergen, 2018), momentary lapses of attention (Unsworth and Robison, 2016, 2018), successful/unsuccessful memory retrieval (Papesh et al., 2012; Goldinger and Papesh, 2012), and can even be used to determine whether or not an individual is mind-wandering (Franklin et al., 2013; Mittner et al., 2016). Further, eye-gaze based detection of mind-wandering has been successfully employed while people read (Foulsham et al., 2013), watch films and video lectures (Mills et al., 2016; Hutt et al., 2017) and engage with online learning systems (Hutt et al., 2019). Therefore, we had reason to believe we could leverage state-of-the-art machine learning tools to predict an individual's psychological state, momentary level of attentiveness, and behavior via pupillometry and gaze positions during a simple sustained attention task.

The objective of this work is to revisit the extensive pupillometry and reaction time (RT) data collected in Robison et al. (2021) and create new predictive models of attentiveness using a modern machine learning (ML) framework. In ML, a typical model (estimator) starts with a set of preprocessed features (explanatory variables) and is trained to predict a target variable from these features. If the target variable is continuous, the task is considered a regression problem. If the target variable is categorical, the task is a classification problem. The richness of the original study allows us to propose multiple ML problems of interest by varying the corresponding target variable and the composition of the feature set. We will solve:

RT predictions similar to the first family of problems were attempted in Robison et al. (2021). However, here we explore various initial feature lists and regressors available in scikit-learn. The last two families of ML problems have not been explored yet. The outline of the paper is as follows: First, we review the experimental setup of the psychomotor vigilance task from Robison et al. (2021), describing how eye measurements were collected throughout the trials and detailing the various experimental manipulations that subjects underwent, which could impact their motivation and, consequently, their RTs. Next, we explain our pipeline for pre-processing the features that will be used in the subsequent models and carefully define the three families of machine learning problems investigated in our study. Third, we describe the feature matrices and pipelines used in each case. In the results section, we summarize the balanced accuracy scores and R-squared values for all classification and regression problems, along with a ranking of features according to their importance. Finally, we discuss and interpret our results in light of modern psychological theories.

Figure 1 illustrates the psychomotor vigilance task described in Robison et al. (2021). Briefly, subjects (N = 353) in a dark room stare at a blank screen that is first replaced by a fixation screen and then by a screen that displays zeroes for a random interval between 2–10 s. After this delay, the zeroes begin counting forward like a stopwatch. Subjects were instructed to press the space bar as soon as the stopwatch starts (timer phase). Once pressed, the counting stops, the screen freezes for 1 s (in some cases displaying a feedback screen), the RT is recorded, the screen resets, and a new trial begins. The subject's eye movements (diameter of left/right pupils and their gaze positions) are recorded throughout the process (140 trials).

As listed in Table 1 (top left), the experimental conditions varied among subjects in terms of the type of goal (easy, hard, or none), feedback (yes/no), and reward conditions (time incentive, cash incentive, or none). Subjects in group G₀ were asked to respond to the cue as quickly as possible. For subjects in group G₁, the average reaction time (RT) was provided every 28 trials. Subjects in group G₂ were asked to keep their RT below 800 ms (easy goal). Subjects in groups G₃ and G₄ were asked to keep their RT below 300 ms (hard goal), with G₄ subjects receiving the same feedback as those in G₁. Subjects in group G₅ were told they could leave early if they met the goal, while group G₆ subjects were promised $10 upon reaching the goal. In reality, all groups completed the same number of trials, regardless of performance. Table 1 (right) lists all measurements collected during the experiment and the derived features used in our study.

In our study, we solve three families of machine learning problems: (i) four RT-regression problems at the population level, (ii) three RT-classification problems at both individual and population levels, and (iii) six Experimental Condition (EC) classification problems at group levels. See Table 2 for details.

Figure 2 depicts the seven key steps in the general pipeline used in the regression and classification problems: data curation, feature engineering, data imputation, data standardization, feature selection, model selection, and hyper parameter tuning. See the paragraphs below for some details.

Table 3 summarizes the results for the three families of machine learning problems that we solve: (i) four RT-regression problems at the population level RRTpop[1,…,4], (ii) six RT-classification problems CRTpop[1,2,3] and CRTind[1,2,3], and (iii) six experimental Condition (EC) classification problems at group levels CECgroup[1,…,6]. We list the number of features before and after feature selection, the best classifier or regressor for each problem, and the corresponding balanced accuracy or R-squared score. We favor balanced accuracy scores over (simple) accuracy for all classification problems to account for differences in group sizes for the different labels. Note that the scores for all classification problems surpass random guessing by a significant margin.

The box-plots in Figure 3 depicts R-squared values over 400 cross-validations for problems RRTpop[1,…,4] using our top model, the Multi-layer Perceptron regressor (MLP regressor). Problems RRTpop[1,2,3] include 348 subject IDs in the initial feature list, and their mean R-squared are 0.25 ± 0.03, 0.25 ± 0.03, 0.24 ± 0.03 respectively. Contrasting, the mean R-squared for problem RRTpop[4] (that does not include subject IDs) is 0.04 ± 0.01, demonstrating the importance of IDs as RT predictors. Interestingly, the addition of categorical variables encoding experimental conditions (problem RRTpop[2] vs. RRTpop[3]) did not lead to any improvement in the predictions.

Table 4 lists the selected features for problems CRTpop[1,2,3]. Note that the trial number and the categorical variable encoding the presence of a goal were selected by all problems. All problems select at least one of the pupillometry features and an experimental condition, with the Nearest centroid classifier being the best method for problems CRTpop[1,2]. Many classifiers provide similar balanced accuracy scores for CRTpop[3] (adaboost, label propagation, linear discriminant, LGBM, random forest, SVC), so we chose linear discriminant as it is simpler to hyper-tune. We run five cross-validations with the wrapper (testing all methods) and 20 cross-validations for the best one.

We solve the RT-classification problems also at the individual level, i.e., trying to classify RT responses as attentive, semi-attentive, and inattentive for each subject in turn. For these problems (CRTind[1,2,3]), the experimental conditions are not included in the feature list. Table 3 shows the mean of the balanced accuracy scores (using the best method) over all subjects. We run five cross-validations with the wrapper (testing all methods) and rank the methods according to the median of the balanced accuracy scores. Figure 4 shows histograms for the frequency of selected features for problem CRTind[1,2,3]. Similar to the results obtained at the population level, the trial number was also the most frequently selected feature for most subjects.

The last family of machine learning problems examined in this study is significantly different from the other two; while RRTpop[1,…,4] and CRTpop[1,2,3] try to predict or categorize RT responses, problems CECgroup[1,…,6] try to disambiguate subjects exposed to different experimental conditions. As shown in Table 1, subjects were binned into groups G₀, …, G₆ depending on the presented type of goal (easy, hard, or none), feedback (yes/no), and reward (time incentive, cash incentive, or none). For these problems, experimental conditions are no longer features used to predict RTs. In fact, RTs are now themselves features used in the six n-way classification tasks (see label type in Table 2). Also, the eye-derived features are no longer collected solely at the fixation phase of the experiment, but instead, throughout the entire trial.

With this different setup, we can now investigate if differences in experimental conditions imprint noticeable changes in the collected physiological data. In problem CECgroup[1], for example, we combine subjects who received either easy or hard goals into a single group [G₂+G₃] and compare them with those who received none [G₀]. This leads to a binary classification problem, [G₀] × [G₂+G₃]. In problem CECgroup[4], we keep G₂ and G₃ separate, which lead to a three-way classification problem instead, [G₀] × [G₂] × [G₃]. The setup for the other CECgroup problems follows the same rationale, and our best classification results are listed in the bottom part of Table 3.

Problems CECgroup[1,2,3] consist of binary classification tasks for one type of experimental condition (vs. control group). Top balanced accuracy scores for the presence of goal, feedback, and reward are 0.69, 0.70, and 0.64 respectively. Problems CECgroup[4,5] consist of three-way classification tasks for goal and reward with top scores of 0.51 and 0.48. Finally, CECgroup[6] is a seven-way classification task for all groups, for which we achieve a 0.24 score. All scores surpass random guessing by a significant margin, and in all problems except CECgroup[4], the most accurate model was also the most informative (using AIC).

Figure 5 depicts the selected features for problems CECgroup[1,…,6]. Since their initial feature list is very large (with 980 features), it is expected that in some cases, the pipeline's original feature-selection step might provide sub-optimal solutions. In an effort to improve our results, we empirically complemented the list of selected features of some problems with features selected by others. Specifically, CECgroup[4] improved when we added features from CECgroup[1], CECgroup[5] improved when we added features from CECgroup[2], and CECgroup[6] improved when we added features from CECgroup[2,5]. As expected, the number of features is proportional to the complexity of the task, with significant overlap between the corresponding binary and three-way tasks (CECgroup[1,4] and CECgroup[3,5]).

The initial feature list for problems CECgroup[1,…,6] is large because we collect six pupillometry features along with the RT per trial (7 × 140 = 980). In what follows, we investigate how well can we classify subjects using coarser measurements. Instead of collecting features at every trial, we average them over 10, 20, and 35 trials, thus obtaining physiological data with decreasing resolution. We use the same feature sorting and selection methods used in the original CECgroup[1,…,6] problems.

Figure 6 displays the mean of the balanced accuracy scores over 400 cross-validations (repeated stratified k fold with 20 splits and 20 repeats) for the best method found by the wrapper. In general, scores tend to slowly decrease as we average features over more trials. Problems CECgroup[1,…,5] have similar decay rates in balanced accuracy (≈−0.005), while CECgroup[6] is slightly more robust (≈−0.002). We report the number of selected features, the exact scores, the standard deviations, the best classifiers, and the AIC values for all problems (at different resolutions) in the Supplementary material.

Figure 7 shows the important features for problems CECgroup[1,…,6], with green/blue blocks representing features selected once/twice under any resolution level. Moreover, if a selected feature is averaged over k trials, we color all corresponding k trials in the matrix plot. Note that there are no overlaps for more than two resolution levels, indicating that predictive features at the highest resolution level are not optimal at the lower ones.

The application of ML to analyze sustained attention data presents significant theoretical, analytical, and practical benefits. Theoretically, ML allows us to identify complex patterns and relationships within the data, enhancing our understanding of cognitive processes in sustained attention. This analytical approach is particularly valuable in behavioral neuroscience and psychology, providing deeper insights into complex behaviors.

Analytically, ML offers robust and flexible tools to handle high-dimensional data, enabling the development of predictive models with high accuracy and generalizability. Our study demonstrates the potential of these models to classify attentional states and predict RTs. The experimental setup led to multiple ML problems, highlighting the importance of carefully defining problems, as the initial feature list and target variable may change. For the same setup, we proposed three families of ML problems. However, predicting human responses remains challenging, even in controlled setups, illustrating the necessity of sophisticated ML pipelines.

Practically, predicting and understanding attentiveness through ML has far-reaching implications. In education, predictive models can inform personalized learning strategies to maintain engagement. In occupational health, these models can monitor and enhance productivity and safety by identifying periods of low attentiveness and implementing timely interventions. Additionally, adaptive user interfaces and assistive technologies can respond to users' attentional states, improving overall user experience and performance.

The present results are a first step toward leveraging pupil diameter and gaze position data to predict moments of inattentiveness. Recent work has pursued similar goals with more dynamic environments and additional data (e.g., body movement, blinking) (D'Mello et al., 2022; Bosch and D'Mello, 2022). In future work, we aim to design tasks that produce robust indices of inattentiveness using eye data and additional sensors for multidimensional data collection. Ultimately, our goal is to build a reproducible and generalizable model that detects attentional lapses in real time, mitigating failures of sustained attention. Portable and accurate wearable eye-tracking technology could be especially useful in air traffic control, baggage screening, life-guarding, and others.