Multisensory integration (MSI) is the ability to process and combine information from different sensory modalities into a unified percept. It is a fundamental ability of human perception as it leads to improved stimulus detection and localization, as evidenced by shortened reaction times (1), which provides an evolutionary advantage by allowing earlier detection of environmental stimuli (2). Even though MSI is a crucial ability of the human brain, underlying mechanisms are not yet fully understood (3–6).

Multimodal illusions provide insight into MSI processes. One of the most extensively studied examples is the McGurk effect in which the presentation of different visual (lip movements, ‘ga’) and auditory (speech, ‘ba’) speech information leads to the awareness of a new percept (‘da’) (7). Since its discovery, the McGurk effect has been replicated and explored across a range of linguistic, cultural, and developmental contexts, offering important insights into how the brain resolves sensory conflicts (8–10). Moreover, research has shown individual differences in susceptibility to the effect, such as those associated with age, cognitive factors, or specific neurological conditions (11–13). The McGurk effect thus highlights the brain’s tendency to integrate conflicting sensory information into a coherent percept, which is a fundamental aspect of MSI.

Another illusion-based paradigm that reveals further dimensions of MSI is the sound-induced flash illusion (SIFI) (14). Here, one flash is presented with two short beep sounds with varying stimulus onset asynchronies (SOAs) between the beep sounds. This often results in the perception of two flashes with probability of illusory perception declining with increasing intervals between the two beep sounds (14, 15). This paradigm illustrates the importance of temporal proximity between stimuli for MSI.

The relevance of this temporal proximity is reflected in the temporal binding window (TBW). The TBW illustrates the temporal window in which integration of stimuli from different modalities becomes more likely (1, 16). For audiovisual stimuli, the TBW is supposed to have a bimodal width of about 200ms (16) indicating that only stimuli within this time window can be effectively integrated. This temporal width is particularly important for complex stimuli like speech, as a more flexible and broader integration is required to capture the complex properties of such stimuli. Thus, the TBW is wider for complex stimuli than for simple stimuli like light flashes and beeps (17). Additionally, the TBW differs interindividually and changes across the life span with a narrowing TBW from childhood into adulthood and a broadening TBW in older age (18).

The width of the TBW also plays an important role in different neurodevelopmental and psychiatric disorders. For example, children with autism spectrum disorders integrate audiovisual information over a longer period of time (larger TBW) which is associated with a reduced perceptional validity (19, 20). Another subgroup of patients with altered audiovisual perception are individuals with schizophrenia (SZ). Schizophrenia is a complex mental disorder affecting essential aspects of psychological experience, marked by characteristic disturbances in thinking, perception, self-functions, affectivity, drive, and psychomotor activity (21). Positive symptoms include persistent hallucinations and delusions, while negative symptoms involve blunted affect and psychomotor disturbances. Neuropsychological deficits, particularly in early stages, are a core feature of SZ, affecting attention, memory, and executive functions (22). Meta-analyses indicate significant impairments in overall cognition and specific domains such as processing speed and verbal memory (23–25).

Individuals with SZ exhibit specific functional deficits in auditory and visual perception, often experiencing auditory or visual hallucinations, which impair social communication (26–28). These perceptual impairments highlight the importance of understanding sensory information processing in SZ (29, 30). Accordingly, recent studies have identified impairments in unimodal visual perception in SZ, such as in contrast threshold recognition or temporal order discrimination (31, 32). Further studies have also indicated impairments in unimodal auditory processing in SZ (5, 33, 34). Conversely, review articles also report intact unisensory processing in SZ while multisensory processing is impaired, even with very simple, low-threshold stimuli (35). Intact unisensory processing in SZ, coupled with impaired audiovisual integration (AVI), was demonstrated, for instance, in a language-based study comparing healthy controls (HC) with SZ (33, 36). In this study, SZ did not differ from HC in unimodal auditory processing (sound localization). However, SZ benefited less from the additional visual component (lip-reading) when processing audiovisual stimuli (speech), indicating an integration deficit in audiovisual processing with complex stimuli in SZ.

As outlined above, as audiovisual integration depends on the temporal proximity of the unimodal stimuli, it can also be depicted through TBW in behavioral experiments. Many studies demonstrate that individuals with SZ have a broader TBW (i.e. integration over longer SOAs) compared to HC (37–39). This aberration is often associated with typical symptoms of schizophrenia concerning sensory dysfunction as for example the severity of hallucinations (5). However, little is known about the underlying mechanisms that account for this enlargement. For an improved understanding of audiovisual integration in SZ the exploration of underlying neural mechanisms seems promising.

EEG studies provide insights into the temporal dynamics of audiovisual integration. The N1 component, a negative deflection at about 100 ms post-stimulus, is sensitive to stimulus features and attentional processes (40). The P2 component, a positive deflection at about 200 ms post-stimulus, is associated with later stages of sensory processing, including attentional allocation and sensory integration (41).

For SZ event-related potential (ERP) studies revealed aberrant short-latency ERPs in response to the McGurk illusion compared to HC (42). Specifically, different studies involving HC as well as SZ have linked the amplitudes of early evoked N1 and P2 components to audiovisual processing (43–46). However, ERP-studies investigating audiovisual integration in SZ report inconsistent findings (47). Stekelenburg et al. (48) compared N1/P2 suppression effects in SZ and HC for auditory vs. audiovisual speech stimuli, reporting a lack of audiovisual N1 suppression and a diminished congruency effect of P2 amplitude in the SZ group. Other studies found similar effects regarding N1 suppression (49, 50). However, Wynn et al. (46) did not find any group differences for audiovisual integration at the neural level in a multisensory target detection task.

In summary, the current state of research on audiovisual integration processes in SZ remains highly inconsistent. This is likely due to the strong variation in utilized stimuli, ranging from simple to complex. Therefore, further systematic investigations of audiovisual integration processes in SZ and its underlying mechanisms are necessary.

In the presented study, we employed a simple paradigm for audiovisual simultaneity perception while recording EEG data, since such paradigms are highly useful for evaluating audiovisual integration processes (51, 52). Specifically, we use a simultaneity judgment task that includes ecological valid and invalid conditions. Ecological valid conditions comprise trials in which a beep sound is preceded by a flash while ecological invalid conditions comprise trials in which a flash follows a beep sound. This ecological valid condition reflects everyday life experience in which the light, travelling with ~300000000m/s, is the leading stimulus while the sound follows, traveling only with ~340m/s (53, 54). Therefore, people show a higher tolerance of lagging auditory than visual information in simultaneity judgement tasks (52, 55). This approach allows for the identification of group differences in TBW as a function of ecological validity. Moreover, we expand the existing literature on the relationship between early ERPs and audiovisual perceptual processes in SZ. Based on the current literature, we anticipate to replicate findings of a widened TBW in SZ compared to HC as well as altered amplitudes in early ERP components.

In the present study, we investigated audiovisual integration in patients with schizophrenia (SZ) compared to healthy controls (HC) using a simultaneity judgment paradigm. During the behavioral experiment, electrophysiological data were collected via EEG, allowing for a comparison of early ERP components between the groups.

First, regarding the Gaussian distribution that was fitted to our behavioral data, we found a sufficient goodness of fit for both groups (mean R² = 0.78, with mean R² = 0.85 for HC and mean R² = 0.69 for SZ) (64). This is in line with previous literature demonstrating a Gaussian or sigmoid distribution for simultaneity judgement tasks (65–67). Moreover, for the control condition (simultaneous presentation of flash and beep) we did not find any group differences, indicating that groups did not differ systematically in response bias. These findings show that our paradigm was sufficient to evoke the intended results.

One objective of the present study was to replicate findings from previous studies and demonstrate differences in temporal aspects of audiovisual perception for individuals with SZ compared to HC (47). Significant differences were found between the groups in the perception of simultaneity of the presented audio and visual stimuli for certain SOA conditions. Specifically, SZ perceived stimuli as simultaneous significantly more often at long SOA conditions (200ms to 350ms) compared to HC. These findings indicate an impaired ability to separate audiovisual stimuli in SZ compared to HC, which is consistent with earlier studies that found alterations in the perception of audiovisual stimuli in SZ at longer SOAs (37, 39, 68). The point of subjective simultaneity (PSS) did not differ between groups, however, the temporal binding window for SZ was significantly larger than for HC. This indicates that SZ perceive asynchronously presented audiovisual stimuli as belonging together over a longer period of time compared to HC. Our findings also support previous research reporting a broader temporal window for integrating audiovisual stimuli in SZ (17, 37, 39). Additionally, we observed a positive correlation between symptom severity and width of the TBW in SZ, which is in line with research reporting associations between deficits in multisensory temporal processing and aberrations in sensory perception (5).

Interestingly, we found a trend for significance for the interaction of ecological validity and group. The presented results suggest that HC have greater difficulty separating audiovisual stimuli under ecologically invalid conditions compared to valid conditions. No such difference is observed in individuals with SZ. It is likely that this potential difference for HC reflects our everyday experiences of differences in physical characteristic of stimuli (light is mostly faster than sound). Therefore, one might assume that the perceptual system should be more effective in separating or integrating stimuli that correspond to the natural environmental perception, and thus in the ecologically valid condition. In the ecologically invalid condition, however, we cannot rely on perceptual experience and unconscious learning processes, making differentiated perception more difficult.

No such difference is observed in individuals with SZ. The lack of difference between ecologically valid and invalid conditions might imply an aberration of perceptual experience in SZ. Correspondingly, our findings might support the application of the predictive coding model as a useful framework for understanding multisensory integration deficits observed in SZ (69, 70). Predictive coding theory posits that the brain continuously anticipates incoming sensory information based on an internal model derived from learned statistical regularity in sensory experience. This model is refined through a hierarchical, bidirectional signaling system: top-down predictions, informed by prior knowledge, to account for sensory inputs, while bottom-up prediction errors are generated to adjust the internal model when discrepancies arise (71, 72). Previous studies have provided evidence for predictive mechanisms in audiovisual speech processing (e.g., 73–76), yet few have focused on simple stimulus synchronicity, as used in our paradigm. Here, we explored ecologically valid conditions where visual stimuli precede auditory stimuli, aligning with typical sensory experiences. According to predictive coding, visual stimuli would facilitate the prediction of corresponding auditory inputs via top-down signaling. Conversely, in invalid conditions, an unexpected sequence would likely trigger a feedforward signal to update the internal model due to prediction error. Our data indicate that HC participants show notable differences between valid and invalid conditions, likely due to effective predictive mechanisms. However, SZ participants displayed no differences, possibly reflecting an imbalance in predictive precision. Recent research suggests that SZ involves an abnormal weighting between the precision of predictions and sensory inputs, with SZ associated with either a reduced precision of prior beliefs or an increased precision of sensory data. This imbalance may shift perceptual processing towards sensory inputs rather than prior expectations, potentially leading to aberrant salience of sensory information and contributing to the atypical multisensory integration observed in SZ.

To further elucidate relevant processes of multimodal perception examining neural correlates is a promising account. For audiovisual integration processes, a connection between early event-related potentials N1 and P2 has consistently been reported (43, 77, 78). Stekelenburg and Vroomen (78) used ecologically valid auditory and visual stimuli to examine audiovisual integration processes. They found reduced auditory N1 amplitudes for audiovisual stimuli indicating facilitated processing for multisensory stimuli. Similar effects of amplitude suppression were found for the P2 component when examining speech-related stimuli (43). The N1 component is highly associated with auditory perception and is involved in early sensory processing, while the P2 component is associated with cognitive processes related to sensory processing such as attention allocation (79). However, the ERP components may reflect different neural mechanisms dependent on stimulus features and task requirements (79, 80). In the present study, we found significant differences between SZ and HC in neurophysiological data. In line with Stekelenburg and Vroomen (81), the N1 component was significantly smaller in valid conditions for SZ compared to HC. Regarding the N1/P2-complex, we find reduced amplitudes for SZ compared to HC for both validity conditions. These results can be interpreted in line with previous research regarding audiovisual integration since an amplitude reduction in the N1/P2 complex is associated with audiovisual integration processes (43, 77, 78). This interpretation is in line with behavioral data showing an enlarged TBW for SZ compared to HC indicating integrational processes over a longer period.

However, some of our results contrast with previous findings challenging the aforementioned interpretation. First, we find a significantly larger N1 amplitude for SOA conditions inside the TBW. As mentioned before, previous research consistently reported a N1 suppression associated with audiovisual integration (45, 46, 78). Therefore, a reduced N1 amplitude would be expected for those SOAs inside the TBW compared to outside since audiovisual integration processes take place inside the TBW. Another surprising result is that we do not find any significant correlations between ERP amplitudes and simultaneity perception on a behavioral level or between ERP amplitudes and symptom severity in SZ. The present task required subjects to separate auditory and visual stimuli, instead of integrating them. It is possible that different mechanisms are crucial to be able to separate auditory and visual stimuli. For this reason, typical ERP changes associated with audiovisual integration processes might not have manifested, as other neural mechanisms may be involved. In this context, the N1 amplitude could reflect the increased attentional resources required to separate auditory and visual stimuli when SOA conditions are short. Accordingly, we observed enhanced N1 amplitudes for valid conditions only in HC. As outlined in the behavioral results, it is possible that, through everyday experience, HC develop mechanisms that facilitate the separation of multisensory stimuli, which is reflected in an enlarged N1 amplitude. Since invalid conditions are not typically encountered in regular experience, such mechanisms may not develop, leading to the absence of N1 enhancement in these cases. Interestingly, for SZ no differences between valid and invalid conditions were observed which may reflect a deficient ability to activate mechanisms that support separation of multisensory stimuli. This may be highly associated with typical symptoms of schizophrenia and may play a crucial role in misperceptions such as hallucinatory symptoms (5). However, this explanation is highly speculative since we did not include unisensory control conditions to ensure that reduced amplitudes in SZ are associated with audiovisual integration. In addition, as outlined above, we did not find any correlations between symptom severity and N1 amplitude.

Surprisingly, we did not observe any group differences for the isolated ERP component P2, nor did we find significant correlations between ERP components and behavioral data. Additionally, our findings regarding the N1 component differ from previous studies, which have reported reduced N1 amplitudes for short SOAs within the same simultaneity judgment paradigm (63, 82). It is important to consider differences in study design, as early ERPs are highly sensitive to specific stimulus characteristics (41). The aforementioned studies utilized different stimulus sets, often involving greater complexity; for example, Kaganovich and Schumaker (63) presented a drawing of an explosion. Furthermore, discrepancies in study samples may have contributed to the differing results, as the participants in our study were older than those in previous research, and age is known to influence multisensory integration (16). Future research should therefore emphasize the impact of specific task requirements, stimuli, and sample characteristics to deepen the understanding of MSI. Additionally, it is necessary to acknowledge the limitations of the present study.

The present study examined audiovisual simultaneity judgements under ecologically valid and invalid conditions for SZ compared to HC. We found an enlarged temporal binding window for SZ for valid as well as invalid conditions, which is in line with previous research. Regarding EEG-data, we found contradictory results. We did not find relations between amplitude of early ERPs and simultaneity perception, which is in contrast to previous studies. Additionally, we found larger N1 amplitudes for HC compared to SZ in positive SOA conditions, while overall, an N1 enhancement was observed for SOA conditions inside the TBW compared to outside, irrespective of group condition. One possible explanation may be that different neural mechanisms are involved in audiovisual separation and integration processes. However, it remains unclear what exact mechanisms may underlie audiovisual separation. More research is needed that combines tasks demanding audiovisual separation and integration. For SZ, we found reduced amplitudes in the N1/P2-complex compared to HC. These differences may be associated with stronger audiovisual integration in SZ.

All in all, it became evident that patients with SZ differ in the perception of audiovisual stimuli under both ecologically valid and invalid conditions on behavioral and neurocognitive measures. However, more research is needed to fully understand perceptional aberrancies in schizophrenia.