Audio-visual onset differences are used to determine syllable identity for ambiguous audio-visual stimulus pairs - Ten Oever, Sanne; Sack, Alexander T; Wheat, Katherine L; Bien, Nina; Van Atteveldt, Nienke (Frontiers in Psychology, 2013)
Content and temporal cues have been shown to interact during audiovisual (AV) speech identification. Typically, the most reliable unimodal cue is used more strongly to identify specific speech features; however, visual cues are only used if the audiovisual stimuli are presented within a certain temporal integration window (TWI). This suggests that temporal cues denote whether unimodal stimuli belong together, that is, whether they should be integrated. It is not known whether temporal cues also provide information about the identity of a syllable. Since spoken syllables have naturally varying audiovisual onset asynchronies, we hypothesize that for suboptimal AV cues presented within the TWI, information about the natural AV onset differences can aid in speech identification. To test this, we presented low-intensity auditory syllables concurrently with visual speech signals, and varied the stimulus onset asynchronies (SOA) of the audiovisual pair, while participants were instructed to identify the auditory syllables. We revealed that specific speech features (e.g. voicing) were identified by relying primarily on one modality (e.g. auditory) Additionally, we showed a wide window in which visual information influenced auditory perception, that seemed even wider for congruent stimulus pairs. Finally, we found a specific response pattern across the SOA range for syllables that were not reliably identified by the unimodal cues, which we explained as the result of the use of natural onset differences between audiovisual speech signals. This indicates that temporal cues not only provide information about the temporal integration of audiovisual stimuli, but additionally convey information about the identity of audiovisual pairs. These results provide a detailed behavioral basis for further neuro-imaging and stimulation studies to unravel the neurofunctional mechanisms of the audio-visual-temporal interplay within speech perception.