Twenty-four patients with the DSM-IV diagnosis SCZ were recruited from outpatient units of the Charité – Universitätsmedizin Berlin. In addition, 24 age, education, gender, and handedness matched HC participants, who were screened for mental disorders with the German version of the Structural Clinical Interview for DSM-IV-R Non-Patient Edition (SCID), participated in the study. Due to the insufficient number (i.e., <30) of illusion or no-illusion trials following the EEG artifact rejection, data of nine SCZ (mean illusion rate = 71.43%) and seven HC (mean illusion rate = 81.53%) were excluded from further analyses. Participants were excluded due to the following reasons: lack of illusion perception (i.e., <30 illusion trials; N = 2 SCZ, N = 1 HC), lack of no-illusion perception (i.e., <30 no-illusion trials; N = 7 SCZ, N = 6 HC). The illusion rates for the excluded SCZ and HC did not significantly differ (Mann–Whitney U test = 20, p = 0.223). For the final data analysis, the 15 best matching HC were selected (based on age, education, gender, handedness). All patients met DSM-IV-TR criteria for SCZ. The psychiatric diagnosis was assessed by a senior psychiatrist at the recruiting institution. The study was conducted in accordance with the Declaration of Helsinki and approved by the ethics committee of the Charité – Universitätsmedizin Berlin. All participants provided written informed consent, had normal hearing, normal or corrected to normal vision, and no neurological disorders, alcohol or substance abuse. A random sample of 40% of all participants underwent a multi-drug screening and all of those tested had negative results. Severity of symptoms in SCZ was assessed with the Positive and Negative Syndrome Scale (PANSS; Kay et al., 1987). To test cognitive performance, the Brief Assessment of Cognition in Schizophrenia (BACS) was assessed (Keefe et al., 2004). Table 1 provides an overview on demographic data, cognitive performance, and clinical scores of the study participants.

The experiment was conducted in a sound-attenuated electrically shielded chamber. Stimuli were presented on a CRT monitor with a background luminance of 21 cd/m². Six stimulus combinations were presented: A₀V₁, A₀V₂, A₁V₁, A₂V₀, A₂V₁, A₂V₂, where the indexed numbers denote the number of auditory (A) and visual (V) inputs. Participants fixated a central white cross while being presented with stimuli of the SIFI paradigm (Figure 1). The participants’ task was to report the number of perceived visual stimuli by pressing a button with the index, middle, or ring finger of their right hand to indicate whether they perceived 0, 1, or 2 flashes, respectively. Each visual stimulus was presented for 10 ms and consisted of a white disk subtending 1.6° with a luminance of 89 cd/m². Visual stimuli were presented at 4.1° centrally below the fixation cross. Each auditory stimulus was presented for 7 ms and consisted of a 73 dB (SPL) 1000 Hz sine wave tone. Auditory stimuli were presented from a central speaker below the screen. Three hundred SIFI trials and 150 trials per control condition were presented in random order in eight blocks.

For all stimulus combinations, the numbers of reported zero, one or two flashes were calculated relative to the total number of trials in each condition (Supplementary Figure 1). For the critical A₂V₁ condition, illusion rates were calculated as the percentage of two perceived flashes in relation to the total number of A₂V₁ trials. Within SCZ these values were related to the psychopathology scores (PANSS) by using Pearson correlations. To statistically control for the influence of anti-psychotic medication, medication dosage was converted to chlorpromazine equivalent level (Gardner et al., 2010) and entered as covariate to partial correlation analyses in the patient group. We calculated t-tests for each condition to compare the behavioral results between SCZ and HC. To account for multiple comparisons, the statistical outcome was Bonferroni corrected.

EEG was recorded using a 128-electrodes active system (EasyCap, Herrsching, Germany), including one horizontal and one vertical EOG electrode to monitor eye movements. Data were recorded against nose reference with a pass band (0.016–250 Hz) and digitized at a sampling rate of 1000 Hz. Preprocessing and offline data analysis were performed using EEGlab (Delorme and Makeig, 2004), Fieldtrip (Oostenveld et al., 2011), and custom-made Matlab scripts (MathWorks, Natick, MA, USA). In our analyses we followed previously reported procedures (Gross et al., 2013; Gross, 2014; Keil J. et al., 2014). Data were offline high-pass filtered (1 Hz, FIR), low-pass filtered (125 Hz, FIR), and notch-filtered (49.0–51 Hz, 4th order two-pass Butterworth filter). Moreover, data were down-sampled to 500 Hz.

For the data analysis, epochs of 4 s (-1 to 3 s around the onset of the first auditory stimulus) were extracted and those containing muscular artifacts were rejected by visual inspection. In total, 1050 trials were presented. Trials containing remaining artifacts with amplitudes of ±100 μV were rejected automatically. After artifact rejection, we used on average 960.13 (SD = 53.40) trials per individual in the SCZ group for further analyses. In the HC group we used on average 952.73 (SD = 62.24) trials for further analyses. For the multisensory A₂V₁ trials, we used on average 152.93 (SD = 67.61) illusion trials and 104 (SD = 54.73) no-illusion trials per individual in the SCZ group. In the HC group we used on average 145.47 (SD = 56.25) illusion trials and 113.53 (SD = 56.08) no-illusion trials. For the statistical analysis, the number of A₂V₁ trials was equalized between conditions, i.e., illusion and no-illusion, using the lowest number of available trials in either condition. This was done separately for each individual.

Independent component analyses were conducted to correct for EOG and ECG artifacts (extended runica; Lee et al., 1999). On average, 14.73 ± 5.06 (SD) independent components for SCZ and 15.67 ± 6.17 independent components for HC were rejected. Next, noisy electrodes were interpolated using spherical interpolation (average SCZ = 1.40 ± 1.50 electrodes; HC = 1.27 ± 1.58 electrodes). Finally, the epoched data were re-referenced to common average and the epoch mean was removed from each epoch.

For the ERP analysis of multisensory A₂V₁ trials, data of SCZ and HC were filtered (35 Hz low-pass, 4th order two-pass Butterworth filter), averaged over trials and a baseline correction was performed (-500 to 0 ms prior to stimulus onset). ERPs were analyzed in a time interval from 0 to 400 ms following stimulus onset. Since no prior EEG study has investigated the SIFI in SCZ, we did not have ad hoc hypotheses about the ERP components that could be altered in this patient group. For this reason, we performed non-parametric tests with cluster-based correction for multiple comparisons (Maris and Oostenveld, 2007; Maris, 2012). This was done separately between the different percepts (factor Perception: illusion vs. no-illusion trials), between groups (factor Group: SCZ vs. HC), and for the Perception by Group interaction (i.e., by comparing the differences between illusion vs. no-illusion trials between SCZ and HC). Significant interactions were followed up by post hoc t-tests. If an effect was significant, Cohen’s d was calculated, as marker of the effect size (Table 2).

To test if there are differences between the processing of multisensory and unisensory stimuli, unisensory visual A₀V₁ and A₀V₂ trials, as well as unisensory auditory A₂V₀ trials were analyzed accordingly. Again, we performed non-parametric tests with cluster-based correction for multiple comparisons between conditions (factor Condition: A₀V₁ vs. A₀V₂), between groups (factor Group: SCZ vs. HC), and for the Condition by Group interaction (i.e., by comparing the differences between A₀V₁ vs. A₀V₂ trials between SCZ and HC). For the auditory A₂V₀ trials, a similar comparison between groups was conducted. Previous studies on the SIFI have used the additive approach to investigate multisensory interactions in ERPs (Mishra et al., 2008, 2010). In this approach, ERPs to multisensory stimuli are linearly combined with activity from trials in which no stimuli were presented (so called ”No-Stim” trials, Busse and Woldorff, 2003; Senkowski et al., 2007). These combined responses, i.e., multisensory AV plus ‘No-Stim’ ERPs, are then compared with the combined ERPs to the respective unisensory stimuli, e.g., unisensory A plus unisensory V. In the present study, we did not include ‘No-Stim’ events and did therefore not apply the additive approach.

In agreement with the finding of a previous study in healthy individuals (Mishra et al., 2007), the analysis of neural oscillations explicitly focused on the examination of beta/gamma band power over the occipital cortex. To this end, time-frequency representations (TFRs) of single trials were calculated for each subject and for each relevant condition (i.e., multisensory: A₂V₁, unisensory: A₀V₁, A₀V₂, A₂V₀). Morlet Wavelets with a width of seven cycles per frequency were used to calculate spectral estimates at each point of the time-frequency window ranging from -1000 to 1500 ms (10 ms steps) in the time domain and 2 to 100 Hz (1 Hz steps) in the frequency domain. TFRs were baseline-corrected with respect to the interval ranging from -500 to -100 ms prior to stimulus onset (absolute baseline). In accordance with Mishra et al. (2007), for the statistical analysis individual mean 25 to 35 Hz total power in the 100 to 150 ms interval and individual mean 32 to 40 Hz total power in the 200 to 240 ms interval was extracted from a sensor-level ROI consisting of 24 electrodes. The ROI closely matched the 25–35 Hz power topography reported by Mishra et al. (2007). Individual mean power was then submitted to repeated measures 2 × 2 ANOVAs with the within-subject factor Perception (illusion vs. no-illusion) and the between-subjects factor Group (SCZ vs. HC). To test if there are differences between the processing of multisensory and unisensory stimuli, an analogous analysis was carried out for unisensory visual stimuli with the within-subject factor Condition (A₀V₁ vs. A₀V₂) and the between-subjects factor Group (SCZ vs. HC). Significant main effects or interactions were followed up by post hoc t-tests. Finally, for the unisensory auditory condition (i.e., A₂V₀) a t-test was used to test the main effect of Group.

Schizophrenia reported perceiving two flashes in 55.7% of the critical A₂V₁ trials, HC in 55.4% of the A₂V₁ trials. The illusion rates did not significantly differ between SCZ and HC [t(28) = 0.028, p = 0.978]. Moreover, the behavioral analyses for the other stimulus types revealed that participants correctly reported the number of presented visual and auditory inputs (Supplementary Figure 1). Furthermore, the comparison of the percepts in the control conditions did not reveal any group differences (Supplementary Figure 1). The partial correlation analyses between SIFI illusion rates and clinical symptoms (i.e., PANSS subscale scores), in which medication dose in SCZ served as control variable, were not significant (all p-values > 0.2).

The non-parametric dependent samples test for multisensory A₂V₁ trials, which was conducted for all electrodes, revealed no significant differences between illusion and no-illusion A₂V₁ trials (Figure 2A). This observation was somewhat surprising and motivated us to conduct an exploratory running non-parametric dependent samples test to examine whether there are ERP differences between illusion and no-illusion A₂V₁ trials in HC only, as previously reported (Mishra et al., 2007). We found significant differences between illusion and no-illusion trials in the 102 to 172 ms and 314 to 384 ms intervals, which indicate perception-related modulations of the visual N1 and P3 components specifically in healthy individuals (Supplementary Figure 2). The non-parametric independent samples test between SCZ and HC revealed significant group differences between 88 and 186 ms. More negative amplitudes in HC compared to SCZ were found at a central cluster comprising of 69 electrodes (Figure 2B; Table 2, upper panel). Notably, the non-parametric independent samples test between the Perception differences for SCZ and HC revealed a significant Perception by Group interaction (Figure 2C). The amplitude differences between illusion and no-illusion trials were significantly larger for HC than for SCZ, especially between 106 and 164 ms. The interaction patterns were most robust at a cluster comprising of 34 central to posterior electrodes. Follow-up post hoc t-tests revealed no differences between illusion compared to no-illusion trials in SCZ [t(14) = 0.980, p = 0.344]. However, there were significant differences between illusion compared to no-illusion trials in HC [t(14) = -5.220, p = 0.000]. Further testing revealed group differences in the 106 to 164 ms interval for illusion trials [t(28) = 3.796, p = 0.001], as well as for no-illusion trials [t(28) = 2.131, p = 0.042]. The ERP amplitudes for both percepts were lower in SCZ compared to HC.

To further examine the effect of visual stimulation alone we conducted non-parametric tests for the unisensory visual conditions (i.e., A₀V₁ vs. A₀V₂). For the interval between 164 and 268 ms we found significant differences between conditions (Figure 3A; Table 2, middle panel). The amplitudes in the A₀V₂ trials compared to A₀V₁ trials were more negative at a cluster comprising of 46 central and posterior electrodes. No main effect of Group or Perception by Group interaction was found (Figures 3B,C). Finally, we compared ERPs to unisensory auditory trials (i.e., A₂V₀) between groups. This analysis revealed no significant group differences (Supplementary Figure 3).

The analysis of neural oscillations focused on occipital 25–35 Hz power between 100 and 150 ms and occipital 32–40 Hz power between 200 and 240 ms. These time-frequency windows were selected based on findings of multisensory interactions in the SIFI obtained by Mishra et al. (2007). The repeated measures 2 × 2 ANOVA for multisensory A₂V₁ trials in the 100 to 150 ms time-frequency window revealed no main effect of Perception [F(1,28) = 0.813, p = 0.375] or Group [F(1,28) = 1.133, p = 0.296]. However, a significant Perception by Group interaction for the early time-frequency window between 100 and 150 ms was found [F(1,28) = 4.940, p = 0.035; Figure 4A]. Follow-up t-tests revealed significantly larger 25–35 Hz power in illusion compared to no-illusion trials in HC [t(14) = 2.502, p = 0.025], but no power differences between illusion and no-illusion trials in SCZ [t(14) = -0.845, p = 0.412]. The analysis of the 32-40 Hz power in the 200 to 240 ms time-frequency window did not reveal any significant effects. Analogous to multisensory trials, we calculated power analyses for unisensory visual trials (i.e., A₀V₁ vs. A₀V₂; Figure 4B). The analyses of the 100–150 ms time-frequency window revealed a significant main effect of Condition [F(1,28) = 6.903, p = 0.014], due to higher power in A₀V₂ compared to A₀V₁ trials. No main effect of Group [F(1,28) = 1.834, p = 0.187] or Condition by Group interaction [F(1,28) = 1.418, p = 0.244] was found. The analyses of the 200–240 ms time-frequency window also revealed a significant main effect of Condition [F(1,28) = 4.521, p = 0.042], due to higher power in A₀V₂ compared to A₀V₁ trials. No main effect of Group [F(1,28) = 2.926, p = 0.098], or Condition by Group interaction [F(1,28) = 0.034, p = 0.856] was found. Finally, we compared 25–35 Hz power in unisensory auditory stimuli (i.e., A₂V₀; Supplementary Figure 4) between groups. This analysis did not reveal any significant differences between SCZ and HC [t(28) = 1.240, p = 0.225 for the 100–150 ms time-frequency window; t(28) = 1.900, p = 0.068 for the 200–240 ms time-frequency window]. For exploratory purposes we also analyzed alpha-band power between 100 and 300 ms in the frequency range of 8–13 Hz. The 2 × 2 ANOVA did not reveal any significant effects.

On average, both SCZ and HC perceived the SIFI in about 55% of the critical A₂V₁ trials. Additionally, there were no perceptual differences between SCZ and HC in the control conditions. In these conditions, both SCZ and HC reported the correct numbers of visual stimuli in 73–95% of trials, indicating that both groups maintained attention throughout the experiment. Hence, the behavioral data suggest no substantial perceptual alterations in SCZ. Given that previous studies have reported aberrant multisensory processing in SCZ (de Boer-Schellekens et al., 2014), the lack of perceptual alterations in the current study is somewhat surprising. In unisensory processing, aberrant neural processing often coincides with altered perception, which is reflected in behavioral deficits (Senkowski and Gallinat, 2015; Uhlhaas and Singer, 2015). Therefore, we expected that differences in multisensory processing in the SIFI would affect the perceptual outcome between the groups. However, it is possible that SCZ show different multisensory processing patterns compared to HC, which still lead to the same perceptual outcome. These different multisensory processing patters might be indicative of compensatory mechanisms, altered signal fidelity or unspecific stimulus processing deficits.

Nevertheless, other studies did also not find differences in the perception of multisensory stimuli between SCZ and HC (Surguladze et al., 2001; Pearl et al., 2009; Martin et al., 2013; Wynn et al., 2014; Roa Romero et al., 2016a). For example, examining the McGurk illusion, Martin et al. (2013), as well as Roa Romero et al. (2016a) observed comparable illusion rates in SCZ and HC. Together with the present findings, these studies suggest that aberrant multisensory processing is not necessarily linked to aberrant multisensory perception. In summary, our behavioral data did not reveal alterations in SIFI perception rates in SCZ.

Within HC, we observed larger ERP amplitudes around 140 ms and around 350 ms to illusion compared to no-illusion multisensory trials. This observation is comparable with previous findings in healthy individuals (Mishra et al., 2007; Keil A. et al., 2014). In the current study, HC compared to SCZ showed larger ERP amplitudes, irrespective of whether the critical SIFI trials were perceived as illusion or no-illusion. This indicates general group differences in the processing of multisensory stimuli. The ERP amplitudes around 140 ms were larger in HC compared to SCZ. Reduced EEG data quality in patients could have contributed to the reduced amplitudes. However, reduced ERP amplitudes in SCZ have been also found in numerous previous studies (e.g., Stone et al., 2011; Onitsuka et al., 2013; Stekelenburg et al., 2013). Furthermore, the number of rejected EEG trials in the analysis did not differ between groups. After the artifact rejection, there were on average 960 trials in the SCZ group and 953 trials in the control group. Therefore, we propose that the reduced ERP amplitudes in SCZ reflect processing alterations in patients and are not due to group differences in the EEG data quality. Taken together, reduced amplitudes in SCZ compared to HC following multisensory stimulation are found in early ERP components.

Importantly, we observed differences in multisensory integration effects, i.e., illusion vs. no-illusion trials, between groups. The integration effects, which were found around 135 ms after the onset of the first auditory and visual stimulus, were more robust in HC compared to SCZ (Figure 2). In unisensory processing, aberrant neural processing in SCZ often coincides with altered perception (Senkowski and Gallinat, 2015; Uhlhaas and Singer, 2015). Therefore, we expected that differences in multisensory processing in the SIFI would also affect the perceptual outcome. However, this was not the case. Recently, Stekelenburg et al. (2013) and Roa Romero et al. (2016b) also reported group differences in multisensory ERPs without showing effects in behavioral data. In an interesting framework on multisensory processing, Bizley et al. (2016) suggested different stages of multisensory integration, in which the initial integrative processing can occur, at least partially, independent of later perception-related processing. In line with this proposal, the initial processing and integration could be disturbed in the SIFI trials in SCZ, whereas the later perceptual stages might be still intact. Thus, there could be later mechanisms in the perception of multisensory stimuli in SCZ that compensate for earlier alterations in sensory processing by recruiting additional brain networks or additional regions within the same network (Rentrop et al., 2011). However, the assumption that compensatory mechanisms might support multisensory perception in SCZ requires further empirical testing. For instance, transcranial magnetic stimulation (TMS) could be applied at early and late time intervals following the presentation of SIFI trials. If there were compensatory mechanisms in SCZ then TMS applied specifically at late intervals should affect perception in patients. In sum, our study revealed altered processing of multisensory stimuli in SCZ. However, this did not affect the SIFI perception rate.

To better understand the observation of reduced multisensory processing in SCZ, the ERP findings in the critical multisensory SIFI trials can be compared to the findings in unisensory trials. Across groups, the ERP amplitudes of unisensory visual stimuli around 215 ms were larger in trials with two visual inputs compared to trials with one visual input. This effect was found in both groups, which might indicate that unisensory processing differences between 1 and 2 visual stimuli are comparable between SCZ and HC and that aberrant processing predominantly occurs in the processing of multisensory stimuli. Previous studies also found evidence for alterations in multisensory processing, but not in unisensory processing in SCZ (Ross et al., 2007). For example, Ross et al. (2007) found specific deficits in the ability of SCZ to integrate visual and auditory speech. SCZ experienced less benefit from visual articulation when they tried to comprehend speech under noisy environmental conditions, whereas their performance in unisensory auditory speech perception remained fully intact. This could imply that sensory processing in SCZ is particularly impaired when stimuli are presented in a multisensory way. The integration process of multisensory information might be more complex than the processing of information that is presented to only one sense.

In the critical multisensory trials, HC displayed higher 25–35 Hz total power to illusion compared to no-illusion trials in the time window of 100 to 150 ms after stimulus onset. The observation of enhanced occipital power in healthy individuals replicates the previous finding by Mishra et al. (2007). Interestingly, the current study revealed that SCZ displayed no power differences at 25–35 Hz between illusion compared to no-illusion trials. Specifically, our study showed that patients lack an enhancement of neural oscillations over the occipital cortex to illusion trials. Interestingly, our data did not show general group differences for the 25–35 Hz responses to multisensory and unisensory visual or unisensory auditory stimuli. It might be that the absence of significant main effects between groups relates to a lack of statistical power. Indeed, previous studies have reported diminished evoked beta band (13–30 Hz) and gamma band (>30 Hz) oscillations in SCZ (Gallinat et al., 2004; Leicht et al., 2010, 2015; Lenz et al., 2011; Keil et al., 2016). For example, Leicht et al. (2010) investigated auditory evoked gamma band oscillations in SCZ and found a significant reduction of power in SCZ compared to HC around 50 ms after stimulus presentation. Keil et al. (2016) recently replicated this effect. Another study by Lenz et al. (2011) examined evoked gamma band responses in SCZ during an auditory oddball-paradigm and reported diminished responses to standard and deviant stimuli in SCZ around 40 to 90 ms after stimulus onset. These studies support our finding of diminished 25–35 Hz oscillations to illusory multisensory SIFI trials in SCZ. Our findings might indicate an early binding deficit in SCZ (Singer and Gray, 1995; Engel et al., 2001). Taken together with previous findings, our study suggests that altered 25–35 Hz oscillatory responses reflect aberrant multisensory processing in SCZ.