Sixteen healthy volunteers (aged 22–29 years, mean age 24.1 years) from Okayama University participated in this experiment. All participants had normal or corrected-to-normal vision and were right-handed, with no neurological or psychiatric disorders and no hearing problems. The experimental protocol was approved by the Ethics Committee of Okayama University.

Stimuli presentation and response collection were carried out using MATLAB Release 14 with the Psychophysics Toolbox (PTB-3, a free, open-source collection of MATLAB and GNU Octave functions designed for designing and executing experiments in psychology research) (Brainard and David, 1997). The experiment consisted of three stimulus types: unimodal visual, unimodal auditory, and bimodal audiovisual (auditory and visual stimuli were presented simultaneously). The visual (V) stimuli consisted of a Gabor grating (2° visual angle, 30% contrast) with three SFs: 1.00 c/d, 1.86 c/d, and 3.47 c/d. These three SFs were selected based on previous studies (Campbell and Robson, 1968). The visual stimuli were presented on a 17-in. CRT monitor (100 Hz, 1,280 × 1,024 pixels, with a background luminance of 10 cd/m²) approximately 4° below the fixation point and included two subtypes with different orientations: clockwise 10° and anticlockwise 10°. The auditory stimulus (A) was a 3,000 Hz pure tone (65 dB SPL, 40 ms in duration, 5 ms rise and fall periods) that was presented through earphones. Bimodal audiovisual (AV) stimuli were presented at three levels, with the visual stimuli of 1.00 c/d (AV1.00), 1.86 c/d (AV1.86), and 3.47 c/d (AV3.47), as shown in Figure 1.

The experiment was performed in a dimly lit, electrically shielded, and sound-attenuated room (laboratory room, Okayama University, Japan). Each subject participated in 12 blocks, with each block lasting approximately 6 min. In six blocks, 10° clockwise was defined as the target visual stimulus, and 10° anticlockwise was defined as the standard reverse stimulus in the other six blocks. The order of orientation for the target stimulus was balanced between participants. Each block consisted of 54 visual stimuli (3 SFs × 15 standards and 3 SFs × 3 targets), 54 audiovisual stimuli (3 SFs × 15 standards and 3 SFs × 3 targets), 15 auditory stimuli, and 15 catch trials. The interstimulus interval varied randomly between 800 and 1,200 ms. At the beginning of each block, the participants were presented with a fixation point for 3,000 ms, and then, the stimuli were presented randomly for 40 ms. The participants were instructed to press the button as accurately and quickly as possible when the target stimuli were presented regardless of whether an auditory stimulus was presented, as shown in Figure 1.

An EEG system (BrainAmp MR plus, Gilching, Germany) was used to record EEG signals through 32 electrodes mounted on an electrode cap (Easy Cap, Herrsching-Breitbrunn, Germany). Horizontal eye movements were measured by deriving the electrooculogram (EOG) from one electrode placed approximately 1 cm from the outer canthi of the left eye. Vertical eye movements and eye blinks were detected by deriving an EOG from an electrode placed approximately 1.5 cm below the participant’s left eye. All signals were referenced to the left and right earlobes, and the impedance was maintained below 5 kΩ. Raw signals were acquired at a sample rate of 500 Hz and stored for offline analysis.

Hit rates, false alarms, and RTs were computed separately for each participant. RTs were calculated based on the responses falling within the mean RT ± 2.5 standard deviations (SDs). Hit rate was calculated as the percentage of correct responses relative to the total number of target stimuli, and false alarm rate was calculated as the percentage of incorrect responses relative to the total number of standard stimuli. In addition, perceptual sensitivity (d’ = z(HR)-z(FA)) and response bias (c = −0.5 × [z(HR) + z(FA)]) were calculated for each participant. These measures were then analyzed using a 3 SF (1.00, 1.86, and 3.47 c/d) * 2 stimuli type (V and AV) repeated-measures ANOVA, followed by post-hoc tests with Bonferroni adjustment.

The EEG signals elicited by the standard stimuli were analyzed using BrainVision Analyzer software (version 1.0, Brain Products GmbH, Munich, Germany). First, the data were re-referenced to the average of left and right mastoids and bandpass-filtered from 1 to 30 Hz at a sample rate of 500 Hz. Next, the data were divided into epochs from −100 to 600 ms after stimulus onset, and baseline corrections were made from −100 to 0 ms. Then, epochs with a voltage exceeding ± 100 μV at any electrode location were excluded from the analysis. In addition, responses associated with false alarms were rejected. Finally, grand-averaged ERPs were obtained across all participants for each stimulus type. Three participants were excluded from further analysis due to the loss of more than 70% of the epochs over at least one stimulus type.

To establish the presence of audiovisual integration, statistical analysis was conducted in three steps, following previous studies (Ren et al., 2018; Senkowski et al., 2011). First, the ERPs for bimodal AV stimuli were compared with the linear summation of unimodal auditory and unimodal visual ERPs (A+V) via pointwise running t-tests (two-tailed) for each electrode under each condition (1.00, 1.86, and 3.47 c/d). Significant differences were plotted when at least 12 consecutive data points met the alpha criterion of being < 0.05 (24 ms at a 500 Hz digitization rate was defined as audiovisual integration, Figure 2; Ren et al., 2018; Yang et al., 2015). Then, the regions of interest (ROI) and integration time intervals where and when significant audiovisual integration occurred were chosen based on the statistical analysis and topographical response patterns (Figure 3). In the second level of analysis, repeated-measures ANOVA were conducted for the three SFs of visual stimuli for the time interval and were selected based on an overview of the significant differences observed in the first step. Mean amplitude data were analyzed while accounting for the between-subject factors of stimulus types, conditions, electrodes, and time intervals. If a significant interaction between stimulus type, condition, or electrode and time interval was observed, the third phase of the analysis was carried out. In the third step, ANOVA were conducted separately for each of the ROI using the factor stimulus type (AV, A+V).

The RTs, hit rates, false alarm rates, perceptual sensitivity (d’), and response bias (c) are shown in Table 1. The 3 SF (1.00, 1.86, and 3.47 c/d) * 2 stimuli type (V and AV) repeated-measures ANOVA on RTs showed a main effect of stimuli type [F (1, 15) = 22.99, p = 0.000, η_p² = 0.605], indicating a faster response for AV stimuli than for V stimuli. In addition, a significant interaction was observed between stimuli type and SF [F (2, 30) = 8.86, p = 0.001, η_p² = 0.371]. Post-hoc comparisons showed significantly faster responses for AV versus V stimuli at SF 1.00 c/d (p < 0.01) and SF 1.86 c/d (p < 0.001), but not at SF 3.47 c/d. The 3 SF (1.00, 1.86, and 3.47 c/d) * 2 stimuli type (V and AV) repeated-measures ANOVA on hit rates revealed a significant main effect of SF [F (2, 30) = 13.65, p = 0.002, η_p² = 0.477], with a higher accuracy for SF 1.00 c/d (p = 0.005) and SF 1.86 c/d (p = 0.006) than for SF 3.47 c/d. Furthermore, a main effect of stimuli type was observed [F (1, 15) = 9.05, p = 0.009, η_p² = 0.376], where AV stimuli elicited larger hit rates than V stimuli. This auditory enhancement was significant only for SF 1.86 c/d (p = 0.049) and SF 3.47 c/d (p = 0.037). However, the interaction between SF and stimuli type was not significant [F (2, 30) = 1.55, p = 0.233, η_p² = 0.094]. The 3 SF (1.00, 1.86, and 3.47 c/d) * 2 stimuli type (V and AV) repeated-measures ANOVA on false alarm rates revealed a significant main effect of stimuli type [F (1, 15) = 7.39, p = 0.016, η_p² = 0.330], with higher false alarms for AV stimuli than for V stimuli. The 3 SF (1.00, 1.86, and 3.47 c/d) * 2 stimuli type (V and AV) repeated-measures ANOVA on d’ showed a main effect of SF [F (2, 30) = 4.58, p = 0.035, η_p² = 0.234], indicating decreased sensitivity with increasing SF. No significant interaction was observed [F (2, 30) = 1.78, p = 0.192, η_p² = 0.106] though a marginal difference between V and AV stimuli emerged at SF 3.47 c/d (p = 0.062). The 3 SF (1.00, 1.86, and 3.47 c/d) * 2 stimuli type (V and AV) repeated-measures ANOVA on c showed a main effect of SF [F (2, 30) = 9.16, p = 0.004, η_p² = 0.379] and a main effect of stimuli type [F (1, 15) = 11.29, p = 0.004, η_p² = 0.429], but no interaction was observed.

The average ERPs across the three experimental conditions are shown in Figure 2. The results of pointwise running t-tests comparing AV and A+V conditions at the three different SFs are shown in Figure 3. Based on the statistical analysis and topographical response patterns, six ROI (frontal: F7, F3, Fz, F4, F8; fronto-central: FC5, FC1, FCz, FC2, FC6; central: C3, Cz, C4; parietal: P3, Pz, P4; occipital: O1, Oz, O2; and temporal: T7, T8) and three integration time windows (60–90 ms, 230–320 ms, and 350–380 ms) were selected. Due to significant lateralization effects in the early time interval of 60–90 ms (Figure 4 topography), bilateral electrodes within each ROI were selected for subsequent analyses (left: F3, FC1, T7, C3, P3 and O1; right: F4, FC2, T8, C4, P4 and O2). A 3 SF (1.00, 1.86, and 3.47 c/d) * 2 stimuli type (AV and A+V) * 3 time interval (60–90 ms, 230–320 ms, and 350–380 ms) * 6 ROI (frontal, fronto-central, temporal, central, parietal, and occipital) * 2 electrode (left and right) repeated-measures ANOVA on mean amplitudes revealed a significant five-way interaction [F (20, 240) = 2.64, p = 0.016, η_p² = 0.180]. This indicates the distinct audiovisual integration patterns for different SF conditions in different time intervals. Therefore, these differences were analyzed in detail in the following sections.

At the behavioral level, our results showed that responses to audiovisual stimuli were faster and more accurate than responses to visual stimuli, suggesting that the presence of a synchronous auditory stimulus facilitates visual orientation discrimination (Busse et al., 2005; Frassinetti et al., 2002; Li et al., 2015; Lippert et al., 2007). This facilitating effect on RTs significantly increased as the SF increased from 1.00 to 1.86 c/d, whereas it disappeared at 3.47 c/d. It seems that part of the attention necessary for cross-modal processing (Busse et al., 2005; Donohue et al., 2011) shifted to identify visual orientation as the SF increased, resulting in the disappearance of audiovisual facilitation. This explanation is consistent with sensitivity (d’) results, which decreased with increasing SF for both audiovisual and visual stimuli (1.00 > 1.86 > 3.47, all p < 0.05), suggesting that the participants’ orientation discrimination ability decreased as the SF increased from 1.00 to 3.47 c/d (Musselwhite and Jeffreys, 1985; Pérez-Bellido et al., 2013). However, the facilitating effect was significantly higher with increasing SF in d’. This observation is in line with those of previous studies (Bolognini et al., 2010; Corneil et al., 2002; Noesselt et al., 2010), which demonstrated an inverse relationship between audiovisual benefits and stimulus intensity. In addition, behavioral results revealed that there may be differences in the effects of SFs on audiovisual integration during different stages.

As shown in Figures 2, 3, our results revealed that audiovisual integration occurred as early as 60–90 ms over temporal-occipital regions. This early latency was in line with previous studies that reported low-level sensory interactions, which are considered the earliest stage of multisensory processing within 100-ms post-stimulus onset, allowing the brain to automatically select and encode external inputs that can facilitate simultaneous stimulus encoding (Cappe et al., 2010; Giard and Peronnet, 1999; Meo et al., 2015; Molholm et al., 2002; Senkowski et al., 2011; Talsma et al., 2007; Van der Burg et al., 2011). Therefore, the task-irrelevant auditory stimulus in the present study can interact with the simultaneous visual stimulus during the early visual sensory processing stage, thereby enhancing the perceptual salience of the visual stimulus. Importantly, our results revealed a left lateralization effect over temporal-occipital regions. Some studies have reported ERPs similar to the left lateralization effect over posterior regions under visual or auditory selective attention (Starke et al., 2017; Van der Burg et al., 2011). For example, in the study by Van der Burg et al. (2011), participants were asked to perform a visual search task to investigate early audiovisual stimuli and were required to report the orientation of the target visual stimulus that was present or not present with the auditory stimulus (Van der Burg et al., 2011). Their results revealed early audiovisual integration at 50–60 ms over the left parieto-occipital region. Furthermore, Starke et al. (2017) used combined functional magnetic resonance imaging and EEG methods to investigate the neural basis of the visually induced enhancement of auditory detection via auditory detection tasks (Starke et al., 2017). They reported that visual-induced auditory enhancement involved left lateralization for low-intensity sounds over the superior temporal sulcus. However, Senkowski et al. (2011) reported that even if attention is directed to both modalities simultaneously, the left lateralization effect occurs during the earliest integration (Senkowski et al., 2011). Their results revealed an ERP component at 40–60 ms over the left posterior and right anterior regions for the low-intensity condition. Therefore, both attention and stimulus intensity may influence early audiovisual stimuli integration over occipital-temporal regions, and further electrophysiological studies are required to investigate this observation in detail.

Notably, the earliest audiovisual integration was found in the 3.47 c/d condition but was absent in the 1.00 and 1.86 c/d conditions (Figure 4). This phenomenon may be related to the intensity of the visual stimulus; a high-SF visual signal tends to be perceived as having a lower intensity than low-SF visual signals when contrast is constant (Georgeson and Sullivan, 1975). Our behavioral results confirmed this finding, showing that visual perceptual ability (d’) decreased with increasing SF from 1.00 c/d to 3.47 c/d (Table 1). Some studies have shown that stimulus intensity can modulate audiovisual integration (Fort et al., 2002; Senkowski et al., 2011; Stevenson and James, 2009). For instance, Fort et al. (2002) manipulated the orientation of visual stimuli (horizontal and vertical) and frequency of sounds (540 Hz and 560 Hz with a low intensity of 50 dB) to investigate the neural mechanism of audiovisual integration in simple detection tasks (Fort et al., 2002). Their results revealed that early integration occurred at 45–85 ms over the occipital-parietal regions when their participants were presented with a low-intensity auditory stimulus. Furthermore, Senkowski et al. (2011) manipulated the intensity of auditory, visual, and audiovisual stimuli of low, middle, and high levels to investigate the effects of stimulus intensity on multisensory audiovisual processing. Their results also revealed an early audiovisual interaction (40–60 ms), showing that the integration effect occurred particularly for low-intensity inputs but not for stimuli with middle and high intensities. Overall, in these studies, the earliest integration occurred when at least one of the presented input modalities (auditory and/or visual) was relatively low in stimulus intensity, which follows the principle of inverse effectiveness. Therefore, we speculated that a high-SF-evoked audiovisual integration at the early sensory stage was dependent on stimulus intensity. However, in the present study, only SFs ranging from 1.00 to 3.47 c/d were presented; thus, our study does not allow us to draw conclusions about how high an SF needs to be to evoke an early integration effect. Further electrophysiological studies are needed to elucidate the neural mechanisms of integration under more detailed visual SF conditions.

Audiovisual integration was significantly delayed with increased SFs over the frontal and fronto-central regions (Figure 5). Audiovisual integration was observed at 230–260 ms, 230–320 ms, and 260–320 ms at 1.00 c/d, 1.86 c/d, and 3.47 c/d, respectively. This phenomenon may be related to the two visual processing pathways of the dorsal and ventral streams (Fang and He, 2005; Haxby et al., 1991). A visual stimulus with low-SF information is primarily processed through the dorsal stream, which responds relatively faster, whereas high-SF information is projected chiefly to the ventral stream, with fine resolution but slow responses (Parker, 1980; Tootell and Nasr, 2017; Zhang et al., 2015). Consistent with our findings, previous behavioral investigations revealed that audiovisual integration occurs through both the dorsal and ventral pathways by processing low SFs (Jaekl and Soto-Faraco, 2010) and high SFs (Jaekl and Harris, 2009), respectively. Thus, audiovisual stimuli of 1.00 c/d allow the stimulus to reach high-order areas rapidly, primarily via the dorsal visual stream, and integrate with the auditory stimulus, whereas audiovisual stimuli of 3.47 c/d require more time to receive high-order areas, primarily via the ventral visual stream, and integrate with the auditory stimulus, resulting in delayed audiovisual integration. Importantly, no stimulus is processed exclusively through the ventral or the dorsal pathway. Both visual and auditory stimuli have been shown to be processed in parallel processing streams (Merigan and Maunsell, 1993; Nassi and Callaway, 2009). This parallel processing has subsequently been confirmed in audiovisual integration (Ahveninen et al., 2016; Kaposvári et al., 2015). Therefore, audiovisual stimuli of 1.86 c/d allow parallel processing in two pathways, leading to a wider integration time interval (230–320 ms), which combines 1.00 c/d with 3.47 c/d. Overall, we speculate that integration over frontal and fronto-central regions was influenced by the visual processing pathway and that a high SF delayed audiovisual integration.

In addition, our results revealed another instance of audiovisual integration at 350–380 ms over the parietal and occipital regions. Similar to our observations, audiovisual integration activity over the parietal and occipital regions has also been reported in some previous studies in which participants were asked to perform visual-direction-related tasks (Kayser et al., 2017; Yang et al., 2013). Yang et al. (2013), using the same visual orientation discrimination task to investigate the influence of sound location on audiovisual integration, reported significant integration in the parietal and occipital regions at 360–400 ms (Yang et al., 2013). More recently, Kayser et al. (2017) used the visual motion direction discrimination task and reported that sound facilitates visual motion discrimination by enhancing occipital visual representations at approximately 350 ms after stimulus onset. However, integration in these regions was absent for 3.47 c/d. This absence may be related to visual perceptual load, which has been elucidated to significantly influence audiovisual processing (De Niear et al., 2016; Macdonald and Lavie, 2011). Macdonald and Lavie (2011) reported that participants were less able to notice the presence of a simple auditory tone in the last trial while they were performing a high-visual-load task than when they were performing a low-visual-load task. Furthermore, Gibney et al. (2017) provided further evidence for the effect of visual perceptual load on audiovisual integration using audiovisual speed detection tasks via dual-task paradigms. Their results revealed that audiovisual integration occurred under no and low perceptual load conditions but was absent under high perceptual load conditions. Indeed, in the present study, discriminating visual orientation with under the 3.47 c/d condition was the most difficult task (Musselwhite and Jeffreys, 1985; Pérez-Bellido et al., 2013), and participants needed to pay more attention to this orientation, which was also necessary for cross-modal audiovisual processing to identify the orientation (Busse et al., 2005; Donohue et al., 2011), resulting in the absence of audiovisual integration. Specifically, our results revealed that the activity in the parietal and occipital regions was particularly relevant to the enhancement of behavioral performance (the absence of audiovisual benefits in RTs). It was previously noted that the activity over the parietal and occipital regions results in task-relevant representations. Therefore, we speculated that modulations in the parietal and occipital regions may be dependent on feedback from higher association areas, which guide audiovisual influences based on task requirements.