Cognitive deficits are associated with unemployment, fewer social contacts as well as problems with household activities, and thus have an essential impact on the daily lives of multiple sclerosis (MS) patients (1–5). However, it can be rather challenging to understand how poor performance on a neuropsychological test translates into impaired functioning in everyday life if only conventional laboratory measurements with limited relevance to real-life experiences of MS patients are used. Here, we employed eye-tracking analysis during an ecologically valid visual search task to investigate how cognitively impaired MS patients as defined by the diagnostic standard, the Symbol Digit Modalities Test (SDMT) (3, 6), differ from those with a preserved information processing ability.

Eye-tracking has the potential to measure cognition in neurodegenerative diseases (7) and requires only a limited amount of resources. It will become more widespread in the foreseeable future, as the quality of cameras in laptops and tablets and the necessary software has developed to such an extent making it possible to employ them for cognitive experiments, e.g., via a web browser (8). Studies using eye-tracking while performing the SDMT have already shown that MS patients differ from healthy individuals, among other measures, with an increased total number of fixations in the test area (9). This suggests that MS patients may have uncertainties during their visual search behavior, possibly reflected by the fact that the target areas are checked multiple times for safety (9). This might be a direct consequence of their cognitive impairment. Similarly, patients with mild cognitive impairment (MCI), often an early stage of the Alzheimer's disease continuum, have been shown to perform significantly worse than healthy controls in a visual search task, which was associated with more pronounced cognitive impairment (10, 11).

However, a typical experimental task using simple visual stimuli (e.g., arrows, dots, etc.) does not reflect a situation from the patients' everyday life. A visual search task with pictures showing everyday situations would possess a much higher ecologically validity. Therefore, we explored eye-tracking during a visual search task with pictures showing everyday scenarios. The decision where and when to move the point of fixation is a key aspect of eye-movement control (12). This decision is driven by visuospatial attention and modulates the speed of visual search (13). In general, attentional top-down control is one of the key aspects in visual search (14) and thus eye-tracking can reflect cognitive processes (7, 9–12).

We hypothesized that MS patients with low SDMT scores would need more time to fixate the location of the target object, as this reflects processing speed and the integrity of top-down processes such as expectations and prior knowledge that determine where to search for a target object. In addition, we expected the less efficient search behavior of cognitively impaired patients to be associated also with a larger amount of fixations of non-target areas prior to the first fixation of the target and a shorter total fixation duration compared to cognitively preserved patients.

When using z < −1 as a cut-off for impaired performance in SDMT, impaired and unimpaired MS patients did not differ regarding age, gender, visual acuity and disease duration (all p > 0.05, Table 1). The same was found when using z < −1.5 as a threshold (all p > 0.05, Table 1). However, in this analysis impaired patients exhibited a longer disease duration (p = 0.037, Table 1). For both definitions of below-average information processing speed, impaired patients had a higher EDSS value (p = 0.002) and 9-hole-peg-test score (p < 0.001; Table 1).

Comparing the impaired patients (SDMT z < −1.5) with cognitively unimpaired patients we observed significant differences on TFF (p = 0.001, q = 0.006), TFD (p = 0.014, q = 0.042), and ACC (p = 0.033, q = 0.0495). Patients with good SDMT performance were faster in fixating the target for the first time (1.55s ± 0.39s vs. 2.06s ± 0.24s), fixated the target longer (2.38s ± 1.19s vs. 1.28s ± 0.75s) and were more accurate (72 ± 11% vs. 62 ± 10%) in detecting the target (Figure 1, Table 2).

In the explorative analyses of the visual search behavior of MS patients, we found significant effects of their SDMT performance (SDMT z > −1 vs. SDMT z < −1) on TFF (p = 0.006), TFD (p = 0.005), and FC (p = 0.033). Patients with average or better SDMT performance were faster in fixating the target for the first time (1.55s ± 0.40s vs. 1.96s ± 0.33s) and they had more (6.08 ± 1.48 vs. 4.71 ± 2.05) and longer (2.47s ± 1.19s vs. 1.29s ± 0.71s) fixations on the target (Table 3).

Univariate analyses of variance with “reaction time” as dependent variable, “SDMT performance” as independent variable and the 9HPT as a covariate showed that the main effect of SDMT performance was significant with slower reaction times for the impaired patients (SDMT z < −1.5) (p = 0.027, q = 0.0495, 3.437s ± 0.566s vs. 3.782s ± 0.681s). The 9HPT as a covariate was not significant (p = 0.381). Using the explorative cut-off of SDMT z < −1 did not show any significant effect on reaction time and the 9HPT performance (p = 0.110 and p = 0.510).

Including the within-subjects factor “expectedness” as another exploratory analysis revealed an interaction between the expectedness of the location of the target and the cognitive status only for accuracy with unimpaired patients being more accurate than impaired patients for the expected condition, while this difference was even more pronounced for the unexpected condition (p = 0.037, Table 4, Figure 2). Similar to our primary analysis, we found for TFF, TFD and reaction time but not for FB and FC significant group differences (see Table 4). Not surprisingly, the main effect of expectedness was significant for almost all eye tracking performance and behavioral measures with stimuli in expected locations resulting in better accuracy, shorter reaction times, less TFF and FB as well as more FC (Table 4). We found also significant correlations between SDMT performance and TFF expected (r = −0.498, p = 0.003, Figure 3) but not TFF unexpected (r = −0.212, p = 0.229), as well as between SDMT and TFD expected (r = 0.441, p = 0.009) and TFD unexpected (r = 0.484, p = 0.004), (see Figure 3).

Our findings demonstrate how impaired cognitive processing speed in MS is translated directly into dysfunctional visual search in naturalistic scenes measured with eye-tracking. Impaired patients take longer to find and fixate the target object, and although they take longer, they are less accurate.

They fixated the target object for a shorter time, probably because they first needed more time to locate it in the scene and invested thus more time in exploring the periphery to gain confidence about their decision. The SDMT is diagnostically helpful, but it does not tell us anything about the real world. Our results offer an explanation how impaired cognitive processing limits the patients' daily functioning.

Impaired patients (SDMT z < −1.5) indicated with less accuracy whether the searched object was present or not and needed more time to react. This difference was even more pronounced, as demonstrated by the significant GLM interaction for accuracy, when the target was located in an unexpected position (Figure 2). This suggests that visual search in naturalistic scenes is strongly driven by top-down cognitive processes. Our patients were instructed before each trial which object they had to search for. An intact information processing ability would allow the subject to optimize his or her searching strategy (e.g., “milk carton is usually found in the fridge”) and fixate the target more quickly, if it is located in its expected position, as demonstrated by the significant correlation between SDMT performance and time to first fixation for the expected condition (Figure 3). This, however, seems not to be the case, if the target is in an unexpected position (e.g., on the floor)—in this case optimizing the search strategy is not possible by employing top-down cognitive processes, as evident by the absence of correlation between SDMT performance and time to first fixation for the unexpected condition (Figure 3).

Interestingly, when using a more permissive threshold for defining cognitive impairment (SDMT z < −1), group differences in visual search behavior remained significant while behavioral measures (i.e., accuracy and reaction time) were no longer significant. This suggests that eye-tracking could be useful in detecting subtle cognitive impairment even before neuropsychological and behavioral performance declines.

Using saccade analysis, previous work has shown that MS patients have a higher error rate (e.g., looking to the wrong side) and a longer latency compared to healthy controls (23). In particular, SDMT and the “Paced Auditory Serial Addition Test” performance correlate with error rate and latency (24). Our study relates to these findings, even though we did not explicitly study saccades. Saccades and fixations are the basis of eye movements (12) and TFF reflects the combined time from all saccades and fixations before the first fixation on the target and thus provides at least an indirectly comparable measure. Thus, our results complement the current literature by employing an ecologically valid paradigm (naturalistic visual scenes) and experimental instructions that are closer to the real challenges MS patients face in their daily lives. Since saccade generation is essential for an optimal performance in a visual search task, the increased error rate found in previous studies using saccade measurements might be linked to our results showing lower accuracy of MS patients. Furthermore, longer latencies in saccade generation might be associated with longer search times (25), which are increased in MS, as we show here using TFF.

It is not surprising that there is a correlation between the SDMT and eye-tracking, as the SDMT assesses visual scanning, memory and perceptual speed, among other functions (26). Pavisian et al. demonstrated that the eye-tracking behavior of MS patients differs from that of healthy individuals during the performance of the SDMT (9). We extend these findings to a more naturalistic task that is directly linked to the patients' daily lives and stress the importance of intact information processing speed for visual search in the context of real-life situations.

With increasing availability of eye-tracking, e.g., based on web browsers, visual screening tests could be performed, for example before an appointment at the doctor's office or even at home. They could be used to identify patients who should undergo a more intensive neuropsychological examination. Furthermore, understanding better how daily functions are affected by impaired information processing could facilitate the development of better cognitive monitoring and rehabilitation strategies. Our results could be explained on the one hand by slowed bottom-up information processing and on the other hand by impaired top-down executive control. Since both perceptual processes and executive control are impaired in MS (27–29), it remains unclear to what extent deficits of bottom-up or top-down processing or their combination are causal for the cognitive deficits of MS patients.

In MS, but also in other neurodegenerative diseases, eye tracking has limitations in cases of severely reduced visual acuity or oculomotor dysfunction (involvement of cranial nerves). The significance of this problem is less essential for patients at the early stages of the disease but might become more serious and hamper implementing eye-tracking in cognitive monitoring or rehabilitation of patients with higher EDSS scores or in advanced progressive stages of the disease.

One further possible limitation is the variable delay between the eye tracking experiment and SDMT testing (up to 6 weeks). In order to prevent fatigue, the two measurements were not performed on the same day, although we knew that this could affect our results. However, since the sequence (SDMT first, eye tracking second and vice versa) of the two measurements was pseudorandomized across all participants, we believe that the influence of any possible systematic bias on our findings has been minimized. Nevertheless, future studies should employ, if possible, a standardized measurement schedule.

Among the limitations of the current study are the small to moderate number of patients included, the absence of progressive MS patients and the cross-sectional nature of our investigation. Longitudinal studies with different disease phenotypes would extend our findings and test the notion that eye-tracking changes predict later clinical and/or cognitive decline. Interestingly, a longitudinal study over 2 years studying antisaccades, has shown that error rate, latency and accuracy worsened in MS patients, whereas EDSS scores remained largely unchanged, which might suggest that eye-tracking could be sensitive to subclinical disease activity and more subtle neurodegenerative processes (30).

Naturalistic cognitive assessments which reflect everyday difficulties that MS patients encounter in their daily routine might be challenging. Visual search tools employing ecologically valid paradigms would be one way to address this problem, but data supporting their validation in MS is limited. On the other hand, there is an increasing amount of data using e.g., cell phone applications or wearables (31, 32) to detect disease progression parameters. After appropriate validation, eye-tracking employed by such applications might prove useful in identifying progression in cognitive impairment.

The raw data supporting the conclusions of this article will be made available by the authors, without undue reservation.

The studies involving human participants were reviewed and approved by Ethikkommission des Fachbereichs Medizin, Faculty of Medicine, Goethe-University Frankfurt. The patients/participants provided their written informed consent to participate in this study.

JG: designed and conceptualized study, analyzed and interpreted the data, and drafted and revised the manuscript for intellectual content. HB and LF: analyzed the data, interpreted the data, and major role in the acquisition of data. DS and CB: analyzed the data and major role in the acquisition of data. MN, JK, SF, and MB: designed and conceptualized study, interpreted the data, and revised the manuscript for intellectual content. CF: designed and conceptualized study and revised the manuscript for intellectual content. YY: design and conceptualized study, analyzed and interpreted the data, and drafted and revised the manuscript for intellectual content. All authors contributed to the article and approved the submitted version.

YY has been supported by travel grants from Novartis and Sanofi Genzyme, has received an honorarium for active participation in an advisory board by Sanofi Genzyme as well as speaking honoraria by Roche and Sanofi Genzyme. CF reports speaker honoraria and honoraria for participating in advisory boards from Alexion, Bristol Myers Sqibb, Novartis, Teva, Merck, Sanofi-Genzyme, and Roche. CF received research support from Novartis and Sanofi-Genzyme. The remaining authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.