1. Introduction
In a natural environment, the visual system receives a constant stream of dynamically changing and high-dimensional information which must be processed efficiently in order to create a coherent picture of our world. Throughout the past century several heuristics and mechanisms have been proposed to explain how the visual system undertakes such a complex computational task (see e.g., Heider, 1970; Gilbert and Li, 2013 for an overview). Gestalt psychologists were pioneers in describing the ways in which the visual system may group, distinguish and/or segregate different objects in a visual scene (Koffka, 1935). One of their proposed heuristics was the “Law of Good Continuation” (Wertheimer, 1923; Coren and Girgus, 1980), stating that visual elements following smooth global trajectories are perceptually grouped. Contour integration takes this notion and adds the requirement of local oriented elements being aligned in a collinear or cocircular fashion in order to be perceived as a global “contour.” Humans are remarkably efficient at performing the integration of local elements into global contours (Field et al., 1993; Kovacs, 1996), even when there is jitter in the alignment of the individual contour elements (Field et al., 1993), or if a section of a contour is occluded (Yin et al., 1997) as in the Kanizsa triangle (Wang et al., 2012). Functionally, contour integration allows for the identification of boundaries and paves the way for segmenting a visual scene into objects and background.
Several psychophysical (Kellman and Shipley, 1991; Mullen et al., 2000; Li and Gilbert, 2002), neurophysiological (Bosking et al., 1997; Gilbert et al., 2000) and computational studies (Li, 1998; Ernst et al., 2012; Singh et al., 2014; Sarti and Citti, 2015) have helped to create a comprehensive picture of contour integration encompassing different levels of understanding. From these studies we have learned much about which physical properties elements in a scene must have, and how they must be positioned in relation to each other, in order to induce the perception of a contour. For example, contour perception deteriorates with increasing degree of separation between elements (Mandon and Kreiter, 2005; Strother and Kubovy, 2006; May and Hess, 2008; Ernst et al., 2012) and with increasing deviation of element orientation from perfect alignment to the contour path (Bex et al., 2001). In addition, contours have been found to have a higher salience if element properties have a greater degree of similarity, e.g., if their spatial frequencies (Dakin and Hess, 1998; Persike and Meinhardt, 2015) or phases (Hansen and Hess, 2006) are identical. Taken together, these studies revealed a great robustness of contour integration against variations in a multitude of stimulus parameters. On a neural level, this finding is supported by strong correlates of contour integration in electrophysiological recordings, which emerge independently of the behavioral task (Bauer and Heinze, 2002).
Moreover, contour integration seems to be a fast process. Before being masked, stimuli need to be presented for only a few tenths milliseconds for a contour to be perceived. In particular, a stimulus presentation time of only 30 ms suffices for human observers (Ernst et al., 2012) to perceive contours with performances over 75% correct (2-AFC task), and for macaque monkeys (Mandon and Kreiter, 2005) to correctly discriminate contours with a performance of about 66% (in a task with a 25% chance level).
The picture emerging from these studies is that as soon as single contour elements match the criteria for good continuation and contour integration, contour perception appears to have a pop-out nature—suggesting neural mechanisms dominated by feedforward or recurrent integration of visual information which are barely influenced by cortical feedback or the actual cognitive state.
However, this picture changes when the psychophysics of contour perception is related to neural dynamics and anatomical structures in the brain. Both electrophysiological and imaging studies have successfully identified neural signatures of contour integration in human subjects (Altmann et al., 2003; Mathes et al., 2006; Mathes and Fahle, 2007; Mijovic et al., 2014; Volberg and Greenlee, 2014), in cats (Gilbert and Wiesel, 1990; Samonds et al., 2006) and in macaque monkeys (Li et al., 2006, 2008; Gilad et al., 2013). While association fields measured as firing rate facilitation and suppression in dependence on the spatial configuration of two line segments have essentially confirmed the law of good continuation (Kapadia et al., 2000), investigating the dynamics of contour integration has turned up some interesting findings. In early visual areas such as V1, responses to individual Gabor elements comprising a stimulus have been observed as early as 40–140 ms after stimulus onset (Gilad et al., 2013), whether these Gabor elements are part of a contour or not. However, firing rate increments in V1 induced by the alignment of individual Gabor elements comprising a contour have been observed almost exclusively at a later stage, approximately 150–250 ms after stimulus onset (Gilad et al., 2013). This delay suggests the involvement of higher visual areas in the process of CI since V1 circuits would respond much faster. Chen et al. (2014) showed that neural signatures of contour integration indeed appear earlier in area V4 than in V1, and were also accompanied by larger firing rate modulations in V4 than in V1. Similarly, Lee and Nguyen (2001) found earlier and stronger activity patterns in V2 than in V1 when rhesus monkeys were presented with Kanizsa figures arranged to form illusory contours. These observations of the temporal dynamics during CI present further evidence for the idea that higher visual areas are more strongly involved in CI than V1.
These observations have two important consequences: First, the notion of contour integration as a process not involving any feedback interactions has been challenged. Second, if integration is predominantly performed in areas such as V2 or V4, which are known to be strongly influenced by attention, then contour detection might more heavily depend on the current cognitive state than previously thought, especially in situations where contour integration is based on very weak or noisy sensory signals.
Taking together both physiological and psychophysical work, in terms of spatial aspects of various stimulus configurations we obtain a coherent account of contour integration. However, in the majority of these studies, stimuli containing contours were static, and often viewing time was limited by showing targets only briefly. Therefore, it is presently unknown how efficiently contour integration is performed in dynamic stimuli where contours can appear and disappear over an extended observation time. This question becomes even more relevant when considering that dynamic stimuli are a default situation, since in natural environments scenes are observed over an extended period of time, and image content often changes continuously.
In this study, we investigated a prototypical situation in which subjects have to observe a dynamic scene for an extended period of time. The task for our subjects was to detect the appearance of a contour which was embedded into an ensemble of oriented and slowly rotating Gabor patches. The contour could form at a random time in a random place in the stimulus display. Specifically, we contrasted situations in which the target was presented from a blank slate and the stimulus was presented for only 235 ms, with situations in which the stimulus had to be viewed for an extended period of time and the target formed after viewing the stimulus for a random period which could last up to 4 s.
First, we investigated how efficiently contour detection is performed under these two scenarios. Considering the reported “pop-out” nature of contour integration, one would expect very similar performance in brief and extended dynamic displays. However, on a perceptual level extended viewing is different from brief presentation, and hence extended viewing might be accompanied by changes in cognitive state with competing expectations forming and being evaluated against the dynamic visual evidence. Given that neurophysiological evidence locates contour integration in visual areas such as V2 and V4 (Anzai et al., 2007; Chen et al., 2014), one might expect that these changes in cognitive state interact with contour integration, thus potentially leading to very different performances in these two scenarios.
Indeed we found performance to be significantly different between a brief dynamic presentation condition and an extended dynamic presentation condition. In the extended condition contour detection performance was approximately 20% lower than it was in the brief presentation condition, and reaction times were approximately 270 ms longer. We addressed this observation in the second part of our experiment by cueing a contour position and/or a contour shape in order to investigate to what extent directing observer's focus to a position in space, and/or to a particular shape, can support contour integration in extended vs. brief presentation. In situations where detection performance is low, we quantified how much the different cues can improve contour detection, and thus, to what extent the different cues can restore contour detection performance. We expected to find a differential effect in brief and extended presentations, hypothesizing that the visual system may be able to better capitalize on the information provided by the cues in extended presentations as this case would allow for top-down effects to take place. Finally, our paradigm allowed us to combine spatial and shape cues in order to investigate whether cue combinations yield higher perceptual gains than single cues. Our results show that cues indeed increase perceptual performance, and they suggest differential effects of the single cues or cue combinations when comparing brief and extended presentations.
4. Discussion
In summary, we have found that extended presentations of dynamic stimuli affect the process of contour integration in an unexpected manner. Prolonged presentations lasting from 1.9 to 3.8 s (Long timing condition) lead to large reductions in detection performance when compared to presentations lasting about 235 ms (Peak timing condition). In both conditions the final 235 ms period in stimulus presentation containing the target contour was identical, hence the drop in performance was clearly a consequence of the dynamic stimulus shown prior to the moment of perfect alignment, and not a consequence of the rotation per se. Furthermore, we could show that cueing/attending a particular contour shape or position in the visual field can partially compensate for this effect and enhance contour detection in dynamic stimuli. In order to also quantify the effects of cueing in short stimulus presentations we first decreased SOA until contour detection performance matched baseline performance in extended presentations, thus creating the Short timing condition. Subsequently, we found similar improvements in performance when providing cues in the Short condition, indicating that independently of stimulus presentation time, attention can support contour integration to a similar extent. In addition, in the Long timing condition it appears that the visual system could make better use of cue combinations. Finally, we observed perceptual learning effects from testing block to testing block, which lasted weeks2, suggesting that functional changes must have taken place in the visual system leading to an improved contour detection performance. This improvement was clearly stronger for brief stimulus presentations. In the following, we will discuss our findings in more detail.
The stark reduction in performance for extended stimulus presentations was unexpected since contour integration has been shown to increase monotonically with increasing SOAs and to require only 60 ms to be reliably performed in macaque monkeys (Mandon and Kreiter, 2005) and only about 100 ms to achieve the same performance in human observers (Ernst et al., 2012). In both, Peak and Long timing conditions, there was a relatively ample window of time when the contour was almost perfectly aligned. In our stimulus the maximal rotation speed of elements at more than 5° of visual angle eccentricity was 80°/s, thus the maximal misalignment of the contour segments was ±10° during the final 235 ms period in both conditions. A jitter of ±10° is known to barely affect performance (Field et al., 1993), and indeed performance in the Peak condition was relatively high at about 85% correct responses. In the Long condition, shortly before the appearance of the target contour, the visual system must therefore have been in a state which severely impairs our ability to detect contours. In dynamic scenes this finding implies that contour integration might have a much smaller importance for visual perception and might require a much stronger support by other feature integration processes than previously thought. Furthermore, performance in the Long condition was initially low (about 65%), thus some observers did not have a conscious percept of the contours—this effect might have implications for other cognitive processes which we will discuss later (e.g., perceptual learning).
There might be several possible causes for the stark drop in performance, mainly: (a) neural adaptation processes, (b) perceptual hysteresis, (c) an increased attentional load or attention fatigue in the Long timing condition, (d) the effects of transiently enhanced neural activity immediately after stimulus onset, and/or (e) temporal and recurrent processes in visual perception. To elaborate:
Adaptation is a phenomenon which is ubiquitous in neural systems: adaptation currents in neurons lead to a decreasing firing rate in response to a constant input current (Kohn, 2007), recurrent inhibition provides normalization of neural responses (Carandini et al., 1997), and synaptic resources deplete over time (Tsodyks et al., 1998). Stimulating the same spatial receptive fields may have led to adaptation in the retina or in the LGN, which may have been inherited by areas in visual cortex performing contour integration. On the cortical level, Carandini (2000) has shown that there are cellular mechanisms acting in individual cortical neurons which are responsible for adaptation processes after exposures to visual stimuli lasting several seconds, and that these can lead to impairment of our perception of subsequent stimuli. Motion adaptation effects which impair our perception have also been well documented and are attributed to a reduction of responsiveness in cells tuned to a certain aspect of a stimulus (Anstis et al., 1998). Hence, the extended presentation of rotating Gabor patches in the Long condition may have lead to decreasing firing rates and thus—assuming a constant background noise level—to a decreased ability to detect contours after a few seconds of presentation. However, one also has to take into account that Gabor rotation speeds were very slow compared to the time required for adaptation. Assuming adaptation takes place on a timescale of 50–100 ms, and a full rotation of a Gabor element takes at least 500 ms, orientation-selective neural populations in V1 would have almost fully recovered when a Gabor rotates through their preferred orientation a second time.
Our observation that top-down processes improve performance can be explained by electrophysiological findings (Galashan et al., 2013) showing strong firing rate increases caused by attention even if the corresponding neurons are in a sustained state of low activation after prolonged stimulus exposure. Such increases in activity are also compatible with psychophysical studies (Yeshurun and Levy, 2003; Yeshurun and Marom, 2008) observing that attention increases the perceived duration of a stimulus while decreasing temporal resolution.
Perceptual hysteresis, which has been defined as the tendency to stabilize a percept (Schwiedrzik et al., 2011), may be another explanation for the low performance in the Long condition. Perceptual hysteresis is different from adaptation: fMRI studies with human observers found that when a subject experiences hysteresis, ventral visual areas, superior parietal areas, and the frontal cortices tend to have a high degree of activation (Kleinschmidt et al., 2002; Schwiedrzik et al., 2012). In contrast, when observers experience adaptation in a similar task, areas V2 and V3 in the visual cortex tend to increase their activity (Schwiedrzik et al., 2012). In our case, in the Long condition percepts not related to the target contour might have emerged during extended stimulus presentation. Putative percepts could include proto-contours (i.e., contours of two or three accidentally aligned Gabors), or Gabor configurations with other regularities having a certain salience (i.e., star-like pattern). In the Peak condition, percepts other than the target contour itself would not have emerged because the contour was present from the beginning of the dynamic stimulation period and by construction it was the most salient feature. Furthermore, the short SOA of only 235 ms barely provides the necessary time for stable percepts to emerge.
The required attentional resources to successfully perform the contour integration task in the Long timing condition may have been higher than in the Short timing condition, since the Long condition demands observers to engage in the task for several seconds. Fisher (1984) and Joseph et al. (1997) have shown that performance in visual search tasks is modulated by the number of features to which an observer must attend to in order to perform the task correctly, and by the rate of stimulus presentation. In our experiments the number of features which required attention did not change between the timing conditions. However, it is possible that the stimulus presentation time acted in an analogous manner to the stimulus presentation rate in Joseph et al. (1997). If this is the case, this would explain the low performance in the Long timing condition. Furthermore, the Long timing condition is particularly demanding in terms of attentional load. In this condition observers have to closely monitor the whole stimulus and are not allowed to move their eyes over an extended period of time. This is a situation in which attentional fatigue might have occurred, which would have led to a narrowing of the spatial focus of attention. Subsequently, and throughout the remaining trial, an observer's attention might have shifted to different areas of the stimulus—either to systematically scan the visual field with the now smaller attentional “spotlight,” or automatically drawn to salient features such as the chance alignment of a few Gabor patches emerging in different positions in the visual field (“protocontours”). As a consequence, the focus of attention of our observers could not have been at the location where the target appeared and hence they might have failed to detect it correctly. As the Peak condition consisted of a brief presentation of the stimulus (235 ms), shifting of attention to a different location in the stimulus is unlikely to have occurred.
Performance could have been higher in the Peak timing condition due to the enhanced transient neural activity caused by the abrupt change from a blank screen to a screen full of Gabor patches (Figure 1, periods 4 and 5). It is well known that a transient in neural activity occurs at stimulus onset (Jonides and Yantis, 1988; Visscher et al., 2003). Most importantly, the transient goes along with a decrease in the Fano factor (Galashan et al., 2013), which implies an increase in the signal-to-noise ratio and hence enhanced information processing capabilities. In contour integration, one electrophysiological study even demonstrated that the initial transient of a V1 neuron's response is higher if the stimulus element inside its receptive field is part of a contour (Bauer and Heinze, 2002). In our case, contour integration in the Peak timing condition might have well profited from such an enhanced transient. The transient would of course also be present at the beginning of the stimulus presentation period in the Long timing condition, however, it typically lasts only about 50–100 ms (Galashan et al., 2013), and contours appeared in the Long timing condition no earlier than about 1.7 s after stimulus onset.
In the Long condition one can think of the irrelevant dynamic stimulus prior to Taligned as a forward mask. Forward masks have been shown to affect many perceptual tasks such as change detection (Wutz and Melcher, 2013) and feature integration (Herzog et al., 2001, 2003), thus the appearance of the target contour could have been missed by the observers due to this effect.
One could also think of the dynamic stimulus prior to the appearance of the contour in the Long condition as a continuously evolving noise signal. This signal is potentially integrated over time, as shown in Burr and Santoro (2001), Melcher et al. (2004), and Drewes et al. (2015). Most interestingly, Burr and Santoro (2001) demonstrated that this time scale is limited to about 2–3 s, implying that detection performance of a target presented with an increasing SOA decreases until it reaches a plateau at an SOA of about 2–3 s. The situation is akin to the Long condition in our experiment, in which observers were subjected to a “noise” signal for at least 1.9 s before they were presented with the target (in the Long condition three different presentation intervals were used, 1,882, 2,823, and 3,764 ms, with the contour reaching the point of perfect alignment 235 ms before the end of these intervals). Given that a plateau is already reached at about this time, this would explain why we observed no further decrease in performance between 1.9 and 3.8 s.
EEG data from a recent study (Castellano et al., 2014) suggests that in extended viewing conditions, contour-related neural activity emerges very slowly. The authors considered a similar paradigm to ours (Grzymisch et al., 2013), in which contours formed dynamically by Gabor elements rotating into alignment after two seconds into a trial, and stayed in alignment for about one second before the contour dissolved again. Neural signatures of contour integration became significantly different from activity related to stimuli not containing a contour only after 150–250 ms, and needed as much as 400–600 ms to fully develop. In our case, the contour was present for maximally 230 ms until masked, thus terminating these slow processes far before they could reach maximum salience. This is further indication for contour integration in extended presentations being dominated, or even suppressed, by other on-going processes in the visual system.
Contour integration is subject to perceptual learning (Li et al., 2008; Gilbert et al., 2009). We observed perceptual learning in both Long and Short conditions. When participants are first presented with the stimuli typically used in psychophysical contour integration studies they often find it hard to detect contours, and their performance is significantly lower than after several hours of training. This might be due to the artificial nature of the stimuli used in experimental settings; typically stimuli are created in a manner which allows researchers to precisely control the parameters of interest and exclude all other cues. In our experiment we ensured that the only feature which could indicate the presence of a contour was alignment between the individual Gabor elements. However, in natural images there is usually a combination of features (e.g., color, texture, etc.) which indicates whether an element belongs to a given figure or object. Thus, it seems that the visual system of an average person might not be accustomed to detecting contours in the same manner, and especially with so few cues, as it is forced to do so in psychophysical experiments.
We also observed that it is not only harder for the participants to detect contours in extended dynamic presentations, but that the perceptual learning which occurs under this condition is not as pronounced as that seen for brief presentation times (see Figure 6). There are two possible scenarios explaining this finding, namely that asymptotic performance in Long and Short conditions are identical, but learning rates are different, or that learning rates are identical, but asymptotic performance is different. Since we only have four data points describing the temporal evolution of the performance over experimentation time (Figure 6), we can not decide between these possibilities. One reason supporting different learning rates might be our observation that in extended presentations, contours are not always consciously perceived. If perceptual learning takes place through reinforcement, then in such a situation there is no error signal that the visual system could use to improve stimulus processing, thus slowing learning progress. An alternative scenario supporting identical learning rates with different asymptotic performances would emerge if in extended presentations, neural activity is lower (adaptation) or the “noise” level is higher (as it would be if there is interference from other on-going computational processes).
Our results suggest that to achieve a high contour integration performance in dynamic scenes, support from top-down processes is needed. This finding is in agreement with physiological studies showing that top-down influences from higher visual, or other cortical areas strongly support contour integration (Li et al., 2008; Mijovic et al., 2014). Initially, we did not expect to find such a large difference in the Long and Peak conditions as contour integration is typically described as a robust process occurring mainly in early visual areas (Li, 1998; VanRullen et al., 2001; Hess et al., 2003), and as such it should be predominantly driven by the current stimulus and less so by the current behavioral state. From our study we can not conclude whether attention or other top down processes are independent of contour integration, simply acting as an amplifier on top of a contour integration process performed in an early visual cortical area, or if these processes interact with contour integration, actively boosting the linking process of visual features. The perceptual learning we observed throughout our experiments support the suggestion that contour integration involves networks in the extrastriate cortex and not just early visual areas. We make this claim since the perceptual learning we observed is distinct from that shown to take place in V1. In V1 perceptual learning is location-specific (Shiu et al., 1992; Schoups et al., 2001) and requires extensive training over about 2,000–5,000 trials (Schoups et al., 2001). In our experiment perceptual learning took place although the location where contours could appear varied significantly (four quadrants and two orientations, with some freedom in the exact position where a contour could appear in the quadrant), and we had a much lower number of trials (in total, each subject was exposed to about 1,500 trials, however, since the location and orientation of the contours the subject saw was evenly distributed amongst the four quadrants and two orientations the relevant number of trials for perceptual learning would approximately be 200). In summary, we speculate that top down processes invoked by external cues always play a role in contour integration when the task is difficult—only when a contour is in a pop-out, or static configuration, it appears that bottom-up processes suffice to perform efficient contour integration.
In conclusion, to study contour integration in dynamic scenes we employed a novel experimental design (Grzymisch et al., 2013; Castellano et al., 2014) which disentangles contour integration from neural processes linked to stimulus onset transients (such as enhanced activity). This approach paves the way for future studies to answer the question of why contour integration becomes so difficult under extended viewing conditions. In particular, we suggest to record from neurons in areas V1/V2 and V4 in order to distinguishing between feedforward/recurrent and top-down processes. This would allow us to explore the questions of how Gabor rotation activates orientation-selective neurons in orientation hypercolumns, whether the rotation of Gabors causes strong adaptations of neural responses, and how response variability scales with adaptation state and presentation time. Furthermore, by observing neural signatures of attentional states in V4 it will be possible to quantify how an attentional state changes over time, and to relate fluctuations in changing cognitive states to behavior (i.e., successfully/unsuccessfully detected contours).