A total of 62 paid (approximately 8.4 USD/hour) undergraduate or graduate students participated in one of three experiments. A total of 20 (nine females, aged 20–26 years), 20 (nine females, 19–32 years), and 22 (11 females, 19–29 years) students participated in Experiments 1, 2, and 3, respectively. Two participants in Experiment 3 were excluded from further analyses because they fell asleep or hummed songs during the experiment. All participants were naïve as to the purpose of the experiments. They had normal or corrected normal vision and participated in only one of the three experiments. The study was approved by the Seoul National University Institutional Review Board, and informed written consent was obtained from each participant before the experiments.

Visual stimuli were 24 face images from eight people (four women and four men), expressing “happy,” “fearful,” and “neutral” facial emotions. These images were selected from a facial expression database, the Pictures of Facial Affect (Ekman and Friesen, 1976), and permission to use them was obtained from the Paul Ekman Group, LLC¹. Participants viewed a display, where a central target image was flanked vertically and horizontally by four distractor images, and classified the target face as either “male” or “female” (Figures 1A–C). The target and the flankers always differed in facial identity, while the four flankers were always identical to one another in terms of both identity and facial expression. All images were adjusted to match in size (a rectangle of 1.84° × 1.58° of visual angle), in mean luminance (44.18 cd/m²), and in image contrast (15.7% root mean square contrast). The target and flanker images were separated from one another by a gap of 0.05°. The images were presented in monotone color scale (gray scale for Experiments 1 and 3; green, red, and gray scale for Experiment 2) against a dark (0.32 cd/m²) background. Visual stimuli were generated by running the Psychophysics Toolbox extensions (Brainard, 1997; Pelli, 1997; Kleiner et al., 2007) in conjunction with MATLAB 2014b (The MathWorks Inc.) on an iMac computer and displayed on a separate 19-inch LCD monitor (DELL 1905 FP) with a spatiotemporal resolution of 1280 pixels × 1024 pixels and 60 Hz in a dark room. The monitor was 65 cm away from participants’ eyes.

In each trial, participants gazed at the center of the display, where a small (0.75°) fixation cross appeared at the trial onset and stayed for 0.7–1.0 s until presentation of the target image. To see whether and how the emotional influence on cognitive control depends on the relative timing differences between cognitive control and emotional processing, we varied the onset of flanker images to be either earlier or equal to that of the target image, resulting in three conditions of SOA (0-s, 0.1-s, and 0.3-s SOAs; Figure 1D). Participants were instructed to judge the gender of the target image as accurately and quickly as possible and to respond by pressing one of two keys on a keyboard. The target and flanker images both remained displayed until participants made a response or 1.5 s from the target onset. Thus, the flanker images never disappeared before the onset of the target image, accompanying the target image always until the moment of button press, regardless of the SOA conditions. After the offset of the stimuli, the entire screen went blank for variable durations (0.3–0.7 s) before the next trial began.

In all experiments, the task-relevant information was conjunctively defined in two domains. In the space domain, the task-relevant information was provided at the central position. In the feature domain, the task-relevant information was the ensemble of facial features contributing to gender discrimination. Thus, for fast yet accurate gender discrimination of the central target, which was flanked by the peripheral distractors, participants had to integrate the task-relevant information by deploying their selective attention to the gender-discriminating features of a facial image presented at the target location. As in previous studies, the efficiency of selective information processing was estimated by measuring how much gender discrimination of the target was delayed when the distractors carried conflicting gender information compared to when they did not, ‘congruency effect.’ In the current study, we further distracted selective information processing by additionally manipulating the task-irrelevant features of facial images in various ways. In Experiments 1 and 2, we examined how the spatial location of task-irrelevant features affects the processing of the task-relevant features by manipulating the task-irrelevant feature either only in the target image or only in the flanker images. The task-irrelevant feature was “emotion expression” in Experiment 1 (Figure 1A) and “color tone” in Experiment 2 (Figure 1B). In Experiment 3, we manipulated the task-irrelevant emotion expression both in the target and flanker locations simultaneously (Figure 1C).

All experiments included the same three SOAs (Figure 1D). We opted to use this particular set of SOAs because the congruency effect is expected to vary dynamically over this range, reaching its peak at around a 0.1-s SOA (Taylor, 1977; Flowers and Wilcox, 1982; Flowers, 1990; Schmidt and Schmidt, 2013). Thus, this range of SOAs was expected to sufficiently detect any potential effects of task-irrelevant emotion expression. In addition, using three SOAs allowed us to explore the time-dependent changes of any interactions between task-relevant cognitive control and task-irrelevant emotion reactivity (and its regulation). Below, we described the detailed procedures specific to each experiment.

Across all experiments, participants rarely made incorrect gender judgments [mean error rates for Experiment 1 (Figure 2A) = 2.63%; Experiment 2 = 2.07%; Experiment 3 = 2.01%]. Thus, we analyzed response time (RT) data acquired from the trials in which participants made correct judgments (Figure 2B). Data from trials in which RTs were extremely fast or slow were discarded from further analysis; valid RT measures were defined as those falling within the ±3 standard deviations from the RT averaged across all correct trials for each participant and all experiments. This resulted in an exclusion of 1.75%, 1.79%, and 1.73% of correct trials for Experiments 1, 2, and 3, respectively.

To evaluate the contributions of the task-irrelevant features and the SOAs to the congruency effects in RT measures, we conducted statistical analyses as follows. First, we ran repeated-measures analyses of variance (ANOVAs) to test whether significant RT differences existed between the factors of interest. Results of ANOVAs were reported with F-values, Sphericity Assumed p-values, and the effect size (ηp2; Cohen, 1973).

If this ANOVA returned a significant result, the Bonferroni test was used to perform post hoc evaluations of pairwise differences between the factors to compare the congruency effects between the conditions of task-irrelevant feature manipulation at each of the SOA conditions. Bonferroni-adjusted p-values were used to judge the significance of post hoc comparisons, unless stated otherwise.

Across the participants, the mean overall RT was slower when the gender was incongruent (532 ms ± 11.2 ms) than when it was congruent (519 ms ± 12.1 ms) [F_congruency (1,19) = 26.492, p < 0.001, ηp2 = 0.853]. Having confirmed the overall gender congruency effect (i.e., the delayed RT to the gender-incongruent faces; Figures 2C,D), we examined whether the degree of effect was modulated depending on the other factors.

First, the size of congruency effect varied substantially depending on the SOAs’ length [F_{Congruency × SOA} (2,38) = 17.840, p < 0.001, ηp2 = 0.484]. The congruency effect was almost negligible when the target and flankers were presented simultaneously (0-s SOA; 2.61 ms ± 1.44 ms), reached its maximum when the flankers advanced the target by 0.1 s (0.1-s SOA; 20.31 ms ± 3.48 ms), and then diminished when the flankers advanced the target by 0.3 s (0.3-s SOA; 12.77 ms ± 3.27 ms).

In addition, we also assessed whether this modulation was affected by the location of task-irrelevant emotion expression. The pattern of SOA-dependent congruency effects indeed differed depending on whether the task-irrelevant emotion expression was embedded in the central face (the ‘emotion in target’ condition; Figures 3A,C) or in the peripheral faces (the ‘emotion in flankers’ condition; Figures 3B,D) [F_{Congruency × SOA × Location} (2,38) = 3.691; p = 0.034; ηp2 = 0.163]. Post hoc tests showed that the congruency effect was significantly greater in the ‘emotion in flankers’ condition than in the ‘emotion in target’ condition at the SOAs of 0 and 0.3 s (p = 0.034; ηp2 = 0.216 and p = 0.012; ηp2 = 0.291, respectively; Figures 3C,D). This indicates that gender judgments in the gender-incongruent trials were slower when the task-irrelevant emotion was expressed in the flanker faces than when it was expressed in the target face. Nevertheless, this three-way interaction effect between ‘congruency in facial gender,’ ‘target-flanker SOA,’ and ‘location of emotion expression’ factors was not modulated by the valence of emotion expression [“happy,” “fearful,” “neutral”; F_{Congruency × SOA × Location × V alence} (4,76) = 1.064; p = 0.380; ηp2 = 0.053].

There was a block-dependent congruency effect in the “neutral” trials, in which the facial expression was neutral both in the target and the flankers. Although these trials were physically identical, the congruency effect at 0.3-s SOA was significant in the ‘emotion in flankers’ blocks, but non-significant in the ‘emotion in target’ blocks.

As in Experiment 1, the gender congruency effect was significant overall [F_Congruency (1,19) = 14.840; p = 0.001; ηp2 = 0.439], and its size varied substantially as a function of SOA [F_{Congruency × SOA} (2,38) = 10.654; p < 0.001; ηp2 = 0.359]. Unlike Experiment 1, however, the pattern of SOA-dependent congruency effects remained the same, regardless of whether the task-irrelevant color features accompanied the central face (‘color in target’ condition; Figure 4A) or the peripheral faces (‘color in flankers’ condition; Figure 4B) [F_{Congruency × SOA × Location} (2,38) = 1.353; p = 0.271; ηp2 = 0.066]. The congruency effect was significant only at the SOA of 0.1 s both in the ‘color in target’ condition and in the ‘color in flankers’ condition (p < 0.001; ηp2 = 0.590 and p < 0.00; ηp2 = 0.639,respectively).

The results of Experiment 2 indicated that the congruency effects at the SOA of 0.3 s found in Experiment 1 could not be simply attributed to saliency of task-irrelevant features. Instead, our results suggest that the influence of emotion expression on cognitive control goes beyond what can be inflicted by low-level, salient visual features, such as transient, bottom-up distraction of attention.

We confirmed the overall congruency effect [F_Congruency (1,19) = 9.673; p = 0.006; ηp2 = 0.337] and its SOA-dependent changes [F_{Congruency × SOA} (2,38) = 6.687; p = 0.003; ηp2 = 0.260], which were found in Experiments 1 and 2. More importantly, the gender congruency effect differed between the ‘congruent in emotion’ and the ‘incongruent in emotion’ conditions over different ‘target-flanker SOA’ conditions (Figure 5), as indicated by the marginal interaction effect between gender congruency, SOA, and emotion congruency [F_{Congruency × SOA × EC} (2,38) = 2.996; p = 0.062; ηp2 = 0.136]. Specifically, when the flankers advanced the target by 0.3 s, the congruency effect was significant in the congruent emotion condition, but not significant in the incongruent emotion condition (p = 0.001; ηp2 = 0.446 and p = 0.255; ηp2 = 0.068, respectively).

When the flankers appeared ahead of the target by 0.1 s, the gender congruency effect was observed consistently in all viewing conditions examined in our study. When the flankers further advanced the target by 0.3 s, however, the congruency effect disappeared in most of the conditions but remained significant only in the following two conditions (solid symbols in Figure 6): (i) when emotion expression was embedded only in the flankers [F(1,19) = 18.240; p < 0.001; ηp2 = 0.490; ‘emotion in flankers’ condition of Experiment 1; solid triangles in Figure 6], and (ii) when emotion expression of the flankers matched that of the target [F(1,19) = 15.302; p = 0.001; ηp2 = 0.446; ‘congruent in emotion’ condition of Experiment 3; solid circles in Figure 6]. The prolonged congruency effects in these two conditions were invariant to the gender and valence of facial images. As a control, we confirmed that the congruency effect found at the SOA of 0.3 s in the ‘emotion in flankers’ condition did not hold when non-emotional but highly salient low-level features, i.e., monotonic tones of primary colors, were used as task-irrelevant features (‘color in flankers’ condition of Experiment 2; black empty downward triangles in Figure 6). This indicates that the sustained congruency effect in the ‘emotion in flankers’ condition cannot be simply explained by the low-level saliency effect known to capture bottom-up attention (Treisman and Gelade, 1980).

Experiment 1 found that the gender congruency effect was more prolonged when task-irrelevant emotional features were embedded in the flanking faces. Why was the behavioral cost of conflict processing higher in these conditions? One may refer to the well-known findings that facial images with emotional expressions are salient enough to capture bottom-up attention, thus evoking strong neural responses (Vuilleumier et al., 2001). However, this does not fully explain why the congruency effect at the SOA of 0.3 s was not observed when the primary colors, i.e., salient, but not emotional features, were applied to the face images in the flanking regions in Experiment 2. If the capture of bottom-up attention by perceptually salient distractors were alone responsible, then no such difference in the congruency effect would have been observed. Thus, “perceptual saliency” is not sufficient to explain these findings, and “emotional features” seem necessary for the prolonged congruency effect by the flankers. This implies that emotionally salient stimuli, compared to non-emotionally salient stimuli, are more difficult to be controlled by top-down selective attention. This implication seems consistent with previous studies on brain-damaged patients, which suggested that facial or bodily emotion expressions are processed pre-attentively or automatically (Vuilleumier and Schwartz, 2001; Vuilleumier et al., 2002; Anders et al., 2009; Tamietto et al., 2015). For instance, the amygdala and orbitofrontal cortex—areas known for representing emotional reactivity—were readily activated by emotional faces even in patients with hemispatial neglect and visual extinction due to damaged right inferior parietal cortex (Vuilleumier and Schwartz, 2001). In addition, GY, a patient with damaged primary visual cortex, could discriminate facial expressions presented in his blinded visual field above a chance level (De Gelder et al., 1999), a phenomenon called “affective blindsight” [see Celeghin et al. (2015) for a review].

As mentioned earlier, the non-specific influence of emotion on cognitive control might either be always deteriorative (Verbruggen and De Houwer, 2007; Dennis et al., 2008; Hart et al., 2010; Padmala et al., 2011; Kalanthroff et al., 2013) or always facilitatory (Kanske and Kotz, 2010, 2011; Birk et al., 2011; Melcher et al., 2012; Zinchenko et al., 2015). However, this conjecture is at odds with our finding of location-specific modulation of the prolonged congruency effect at the SOA of 0.3 s (as visualized by the contrast between the empty and solid triangles in Figure 6). Instead, we prefer to interpret this prolonged gender congruency effect found in the ‘emotion in flankers’ condition as a counteraction of bottom-up emotion-induced imbalance in spatial attention against goal-directed execution of top-down spatial selective attention. Facial expressions with strong emotional valence, such as those used in the current study, are likely to promote substantial emotional reactions and subsequent unfolding emotional processes not only for those embedded in the cognitively attended, target face, but also for those in cognitively unattended, peripheral faces.

As mentioned in section “Experiment 1: Emotional Saliency in Flanking Faces Prolongs Congruency Effects”, in trials in which the target and the flankers were all neutral faces, the congruency effect at the SOA of 0.3 s was observed when those trials were intermixed with the trials with emotional flankers, but not when intermixed with those with an emotional target. What could have resulted in these “within-block” congruency effects for the neutral faces? We considered two possible candidates: “emotion unfolding effects” and “emotion aftereffects.” Emotion unfolding effects refer to the possibility that the emotional reactions are not triggered on and off swiftly in a moment-to-moment fashion, but instead unfold gradually, thus sustaining over an extended period of time. In other words, emotional reactions induced by emotional facial expressions might have lingered over several trials, and may have therefore been capable of counteracting the cognitive control of selective attention. This interpretation is only speculative, but certainly not implausible given the iterative and accumulative nature of emotion regulation (Gross, 2015). Alternatively, the emotion aftereffects (Webster et al., 2004), i.e., the tendency to perceive an emotion expression (e.g., “happy”) that is the opposite of that of an adapted emotion (e.g., “fearful”), suggests that visualization of neutral faces after the visualization of an emotional face in a previous trial can lead to them being perceived as emotional faces. In other words, neutral flankers could have acted as pseudo fearful flankers after happy flankers appeared in previous trials, or as pseudo happy flankers after fearful flankers. Although the emotion unfolding effects and the emotion aftereffects differ in time scale, the former lingering over several trials and the latter occurring on a trial-to-trial basis, both explanations are equally plausible, and indistinguishable in the current experimental design. To resolve this issue, further experiments would be required, in which the two factors, carry-over and adaptation, are orthogonally manipulated.

Cognitive control can be regulated in a flexible manner at any moments of time until goal-directed actions are completed by evaluating the future benefits and costs of those actions (Mirabella, 2014; Botvinick and Braver, 2015) or by monitoring conflicts of diverse origins, response errors, or novel events (Dixon and Christoff, 2012; Cavanagh and Frank, 2014). When interpreted in the emotion regulation framework, previous neuroimaging results from tasks involving emotion conflict appear to offer an interesting account for our findings of the prolonged gender congruency effect for the target and flankers whose task-irrelevant facial features were matched (congruent) in emotional valence. Using a variant of the Stoop paradigm, Egner et al. (2008) showed that the lateral prefrontal cortex controls non-emotional distractors by enhancing the task-relevant responses to targets, whereas the rostral anterior cingulate cortex controls emotional distractors by decreasing task-irrelevant responses to emotional distractors. Using a variant of the Flanker paradigm, Ochsner et al. (2009) also reported that the emotional (categorization of words into positive vs. negative valences) and non-emotional (categorization of words into metal vs. fruit categories) flanker tasks recruit different cortical areas; the rostral cingulate cortex/ventromedial prefrontal cortex and ventrolateral prefrontal cortex (vlPFC), respectively. Furthermore, Dolcos and McCarthy (2006) demonstrated the interference of facial memory task performance by task-irrelevant emotional scenes, and this interference was accompanied by the activation of a cortical area involved in inhibiting emotion (vlPFC) and the deactivation of a cortical area involved in memory performance (dorsolateral prefrontal cortex). Put together, these neuroimaging studies suggest that the human brain is equipped with two separate mechanisms for controlling emotional and non-emotional stimuli, and that the two mechanisms can interfere with each other when arranged such that one is recruited in task-relevant information processing while the other in task-irrelevant information processing. In this sense, one possible interpretation of the prolonged and augmented gender congruency effects for the emotionally non-conflicting target and flanker faces in Experiment 3 might be that the cortical mechanism for detecting the need for inhibitory control, the presence of sensory conflicts in our case, is hindered by the cortical mechanism for emotional regulation that detected the opposite situation, the absence of conflicts. This is likely to lead the integration of congruent emotional reactions between the target and the flankers. Our interpretation seems consistent with a recent event-related-potential study, which suggests that processing of emotion conflict is modulated by top-down attention in a manner similar to cognitive control (Zhou et al., 2016).

The temporal dynamics of the congruency effects was quite robust, exhibiting an inverted “v”-shape in most of the viewing conditions: very weak, maximal, and then diminishing at the 0-s, 0.1-s, and 0.3-s SOAs, respectively. This confirms and extends the findings by previous studies on the flanker paradigm (Flowers and Wilcox, 1982; Flowers, 1990; Schmidt and Schmidt, 2013). For example, our results replicate those of Flowers and Wilcox (1982), who reported that the congruency effect increased when letter flankers preceded a target letter, and then decreased after reaching the maximum point at around 0.1 s. Schmidt and Schmidt (2013) also demonstrated the inverted v-shape pattern in a flanker experiment using schematic faces, showing that the congruency effect increased when the flanker faces preceded the target face, but decreased when the target preceded the flankers or when the target and flankers appeared simultaneously. The “priming effect” has been considered to account for the inverted v-shape of the congruency effect (Flowers and Wilcox, 1982; Denton and Shiffrin, 2012) because priming is short-lived and thus creates a narrow time window during which the flankers can interfere with the processing of the target. Again, from the perspective of emotion regulation, the prolonged congruency effect in our study suggests that the emotional reaction, presumably promoted in the subcortical and cortical network of emotional reactivity, can be a cascade of unfolding processes unless regulated by the cortical network of emotion regulation (Etkin et al., 2015).

The results reported here should be interpreted with some cautionary notes. First, our participants, who were all Asian, performed the task on the images of Caucasian faces. As shown by previous studies, Asians may perceive Caucasian facial expressions, particularly negative emotions, differently from Caucasians (Elfenbein and Ambady, 2003; Zebrowitz et al., 2010). This could explain why no valence effects were found, which have otherwise been consistently reported in previous studies (Fenske and Eastwood, 2003; Holthausen et al., 2016). Thus, the effect of emotional valence on the congruency effects must be addressed properly in future studies by using the images of faces that have the same ethnic identity as that of the participants. Second, the temporal dynamics of congruency effects, i.e., the inverted v-shape, was quite robust throughout all the experiments, and the time points were carefully chosen based on previous studies (Flowers and Wilcox, 1982; Schmidt and Schmidt, 2013). However, we need to confirm our findings, either over finer time scales or for wider SOA ranges, in future studies, because the time course of congruency effects augmented by task-irrelevant emotional features might turn out to be more complicated or even longer than the longest SOA (0.3 s) currently explored.

We are quick and ready to infer emotional states from the subtle facial features of our neighbors, presumably because this ability helps us to react efficiently to important physical or social events, thus providing us with evolutionary advantages. This implies that even when we are engaged in a demanding cognitive task, emotional expressions, irrespective of their relevance to the task, are likely to trigger emotional reactions. The current study, guided by the recent advance in understanding how the brain processes and regulates such emotional reactions, demonstrated that task-irrelevant emotional expressions can interfere with the cognition of human faces over an extended period of time. We suggest that the spatial imbalance in emotional saliency counteracts cognitive control of selective attention in one route whereas the congruency in emotional valence hinders the execution of inhibitory control in the other route. These two suggested routes, and the temporal dynamics of emotion-cognition interplay occurring there, certainly merit further investigation, particularly with respect to the identification of their neural substrates via neuroimaging or electrophysiological experiments.