Front. Psychol.Frontiers in PsychologyFront. Psychol.1664-1078Frontiers Media S.A.10.3389/fpsyg.2024.1399084PsychologyMini ReviewThe impact of visual information in speech perception for individuals with hearing loss: a mini reviewChoiAhyeon1KimHayoon1JoMina1KimSubeen1JoungHaesun1ChoiInyong12LeeKyogu134*1Music and Audio Research Group, Department of Intelligence and Information, Seoul National University, Seoul, Republic of Korea2Department of Communication Sciences and Disorders, University of Iowa, Iowa City, IA, United States3Interdisciplinary Program in Artificial Intelligence, Seoul National University, Seoul, Republic of Korea4Artificial Intelligence Institute, Seoul National University, Seoul, Republic of Korea
Edited by: Dan Zhang, Tsinghua University, China
Reviewed by: Tongxiang Diao, Peking University People's Hospital, China
This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner(s) are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.
This review examines how visual information enhances speech perception in individuals with hearing loss, focusing on the impact of age, linguistic stimuli, and specific hearing loss factors on the effectiveness of audiovisual (AV) integration. While existing studies offer varied and sometimes conflicting findings regarding the use of visual cues, our analysis shows that these key factors can distinctly shape AV speech perception outcomes. For instance, younger individuals and those who receive early intervention tend to benefit more from visual cues, particularly when linguistic complexity is lower. Additionally, languages with dense phoneme spaces demonstrate a higher dependency on visual information, underscoring the importance of tailoring rehabilitation strategies to specific linguistic contexts. By considering these influences, we highlight areas where understanding is still developing and suggest how personalized rehabilitation strategies and supportive systems could be tailored to better meet individual needs. Furthermore, this review brings attention to important aspects that warrant further investigation, aiming to refine theoretical models and contribute to more effective, customized approaches to hearing rehabilitation.
Speech perception in individuals with hearing loss, presents a complex and multifaceted challenge. These individuals often rely on visual cues to compensate for their auditory limitations in everyday situations (Moradi et al., 2017; Tyler et al., 1997; Kaiser et al., 2003; Moody-Antonio et al., 2005; Kanekama et al., 2010). The seamless integration of auditory and visual information is particularly crucial for effective speech perception.
Previous research related to audio-visual speech perception has reported that visual cues such as lip movements, facial expressions, and gestures play a decisive role in understanding spoken language for those with hearing loss (Mitchel et al., 2023; Zhang et al., 2021; Regenbogen et al., 2012; Peelle and Sommers, 2015; Munhall et al., 2004). However, not all individuals with hearing loss benefit equally from visual information. The dependency on visual cues can significantly vary depending on factors such as the degree of hearing loss, age, and experience with communication aids like cochlear implants (CI) (Altieri and Hudock, 2014; Iler Kirk et al., 2007; Kanekama et al., 2010).
Contrary to the majority of studies that report benefits from visual information, some research shows no correlation between hearing loss and visual dependency (Lewis et al., 2018; Tye-Murray et al., 2007; Blackburn et al., 2019; Stevenson et al., 2017; Pralus et al., 2021). These studies may have been influenced by specific factors such as the subjects' aging (Tye-Murray et al., 2007), the experimental environment (Lewis et al., 2018; Blackburn et al., 2019), and the characteristics of the language stimuli (Stevenson et al., 2017; Pralus et al., 2021). Indeed, various factors, including age, experimental design, and hearing loss-related factors can contribute to the inconsistency in research outcomes (Bergeson et al., 2010).
To address these factors, this review aims to compare existing studies to gain comprehensive insights into how individuals with hearing impairments process and integrate visual cues in speech perception. Specifically, we conducted a thorough literature search using relevant keywords, such as “audiovisual” AND “speech perception” AND (“hearing impairment” OR “auditory disorder” OR “hearing defect”). The search was performed across Google Scholar and Scopus databases without restrictions on publication dates to ensure a comprehensive collection of relevant literature.
Through reviewing the extensive pool of studies collected, we identified key factors that contribute to varying outcomes in audio-visual speech perception research. These factors were categorized based on age, linguistic stimuli, and hearing loss-related factors including onset, intervention timing, and duration. Age was further classified into categories such as infants, children, adolescents, adults, and the elderly to assess the influence of developmental and aging processes. The stimuli were analyzed at different linguistic levels, including phonemes, syllables, words, and sentences, with experimental results organized into tables for detailed analysis. Additionally, cases of hearing loss were categorized based on onset, intervention timing, and duration, with interpretations of experimental results informed by these characteristics.
By evaluating various experiments from multiple perspectives according to these criteria, this review aims to provide a broad understanding of how individuals with hearing impairments process and integrate visual cues in speech perception. Ultimately, the findings from this review are expected to identify research gaps and offer valuable insights that could inform future studies in audio-visual speech perception for individuals with hearing loss.
2 Age-related variations in individuals with hearing loss
The first criterion we focused on is the age of individuals with hearing loss. Previous research has shown diverse age groups among participants, raising the possibility that visual information may or may not have an impact depending on age. Given the dynamic nature of language development, especially in early childhood, participant age becomes a crucial factor in integrating various sources of information. Therefore, we categorized participants into age groups encompassing infants, children, adolescents, adults, and seniors, exploring potential age-related changes in audiovisual speech perception.
Studies on infants and young children with hearing loss suggest that they become more adept at using visual information as they grow (Tona et al., 2015; Iler Kirk et al., 2007; Oryadi-Zanjani et al., 2017; Taitelbaum-Swead and Fostick, 2017). Literature frequently identifies the age of around 6 as a critical period for the initiation of sophisticated visual information utilization based on the onset of the McGurk effect and the beginning of audiovisual integration in the superior temporal sulcus (STS) (Tona et al., 2015; Iler Kirk et al., 2007; Oryadi-Zanjani et al., 2017).
Another study highlights 3.25 years as a significant age for beginning to integrate auditory and visual cues for speech perception (Holt et al., 2011). This difference can be attributed to the developmental stages of sensory integration, where basic multisensory integration starts around 3-4 years and becomes more refined by the age of 6. These developmental milestones indicate a gradual enhancement in utilizing visual information for speech perception. Compared to typically developing children, those with hearing loss are observed to rely more significantly on visual information during these early stages (Lalonde and McCreery, 2020; Oryadi-Zanjani et al., 2015, 2017; Taitelbaum-Swead and Fostick, 2017; Yamamoto et al., 2017), underscoring the role of early auditory information loss in fostering increased dependence on visual cues.
Despite the abundance of studies on infants and young children with hearing loss, research focusing specifically on adolescents is relatively scarce. While some studies include a range from infancy to adolescence, indicating that those with hearing loss may benefit more from visual information than their hearing counterparts (Tyler et al., 1997; Lamoré et al., 1998), there is a notable gap in studies dedicated exclusively to the adolescent group. This lack of research on adolescents with hearing loss suggests a need for more targeted investigation in this age group to better understand their unique challenges and benefits in utilizing visual information for speech perception.
Studies on individuals beyond the completion of language development, i.e., adults and seniors, do not pinpoint a specific age as crucial. Instead, they often analyze the influence of aging by comparing younger and older adult groups. While some studies report that older participants utilize visual information more as they age (Taitelbaum-Swead and Fostick, 2017; Puschmann et al., 2019; Hällgren et al., 2001), others suggest that the benefit of visual information remains consistent across age groups (Lasfargues-Delannoy et al., 2021; Hay-McCutcheon et al., 2005; Rigo and Lieberman, 1989). Studies specifically targeting seniors present conflicting results on how aging influences audiovisual integration, with some indicating potential benefits (Puschmann et al., 2019) and others suggesting no significant impact (Brooks et al., 2018; Tye-Murray et al., 2007). This variability underscores the importance of considering aging as a factor, dependent on experimental design and participant characteristics.
However, investigations into the neuromodulatory effects of hearing aid use and auditory training on audiovisual integration in seniors have revealed that these interventions can strengthen functional connectivity in the STS, similar to patterns observed during the developmental stages of audiovisual processing (Yu et al., 2017). While these findings are based on experiments with only two participants, they underscore the need for further research with larger sample sizes. These results highlight the necessity of considering the extent of hearing aid use and auditory training when assessing the impact of aging on audiovisual speech perception.
3 Linguistic level differences in individuals with hearing loss
The stimuli used in speech perception experiments are highly diverse, and changes in the linguistic level of the stimuli can significantly impact the utilization of visual information. Therefore, we categorized the experiments based on linguistic levels, such as phonemes, syllables, words, sentences, and suprasegmental elements. We then performed a comprehensive literature review, synthesizing findings from various studies to identify and describe trends in visual information utilization according to the linguistic level of the stimuli (Table 1). This approach allowed us to qualitatively assess and summarize how different linguistic levels influence the reliance on visual information in speech perception.
Summary of visual salience per linguistic stimuli.
Stimuli
Subgroup
Description
Visual saliency
Reference
Phoneme
Labial consonants
/p/, /b/, /m/
High visual saliency due to distinct lip closure and release
(Lamoré et al., 1998; Tsao, 2019)
Non-labial consonants
/t/, /d/, /k/, ...
Less distinct than labials but still detectable through articulatory movements
(Tye-Murray et al., 2007)
Vowels
/a/, /e/, /i/, /o/, /u/
Sustained vowels providing moderate to high visual saliency depending on the articulatory features
(Busby et al., 1984; Lei et al., 2008)
Syllables
–
CVC, VCV, CV, VC
High visual saliency, particularly for syllables with labial consonants
(Suess et al., 2022; Leybaert and LaSasso, 2010; Kishon-Rabin et al., 1997; Sato et al., 2020)
Words
–
Common or nonsense words
Moderate visual saliency, influenced by the recognizability of phoneme elements, particularly labial phonemes
(Puschmann et al., 2019; Taitelbaum-Swead and Fostick, 2017; Mantokoudis et al., 2013; Tye-Murray et al., 2016; Tsao, 2019)
Sentences
–
Simple declarative, complex sentences
Low to moderate visual saliency; effectiveness of visual information diminishes with increased linguistic complexity
(Moberly et al., 2020; Moody-Antonio et al., 2005; Hennesy et al., 2022; Tye-Murray et al., 2016; Tsao, 2019)
Phonemes, as a foundational linguistic element, is particularly influenced by visual information, leading to notable improvements in speech perception. Researches consistently demonstrate that visual information significantly enhances phoneme perception for individuals with hearing loss (Tona et al., 2015; Huyse et al., 2013; Rouger et al., 2008). Both vowel identification (Busby et al., 1984) and consonant identification (Tye-Murray et al., 2007) experiments performed better in the audiovisual than audio-only or visual-only conditions. Researchers differ in their claims about whether vowels or phonemes are generally more visually salient to further aid audiovisual integration. Some studies argue that vowels benefit more from visual cues due to sustained articulatory features and clearer formant structure (Lei et al., 2008). In contrast, others assert consonants are more salient due to distinct visual cues (Van Soeren, 2023; Moradi et al., 2017). This discrepancy may arise from the specific phonemes selected for the experiments. In any case, studies consistently agree that labial consonants (e.g., /p/, /b/, /m/) are the most visually distinctive stimuli compared to vowels and coronal phonemes due to distinct articulatory features such as lip closure and release, which are easily detectable by lip-reading (Lamoré et al., 1998; Tsao, 2019). In the same vein, Rouger et al. (2008) reported that French-speaking CI users placed greater reliance on visual information for labial phonemes, particularly under conditions of auditory uncertainty.
At the syllable level, individuals with hearing loss also benefit from audiovisual (AV) integration, facilitating speech perception (Leybaert and LaSasso, 2010; Kishon-Rabin et al., 1997; Sato et al., 2020). Studies reveal that labial syllables containing labial consonants provide greater visual cues, enhancing perceptual accuracy. For instance, a study on auditory-visual integration with hearing loss focusing on consonant recognition training highlighted overall improvements across all consonants, with significant gains observed in labial consonants within nonsense syllables (Grant et al., 1998). These findings align with those of Tona et al. (2015) and Yamamoto et al. (2017), who observed that Japanese children with cochlear implants (CIs) showed improved recognition of labial syllables (/ba/, /pa/) due to the distinct visual information provided by lip movements, compared to non-labial syllables (/da/, /ga/, /ka/, /ta/). Similarly, Suess et al. (2022) reported that bilabial syllables contribute more to the recognition of hearing-impaired, indicating that the informativeness of each viseme category varies subtly.
The influence of visual information on word recognition presents mixed results, though it generally has a positive impact. Tye-Murray et al. (2007) suggests that the recognizability of a word can be predicted based on the independent recognition of its phoneme elements as transmitted components. Bernstein et al. (2000) examined how certain phonological features influence word recognition visually, pointing out that words with round and coronal phonological traits were more effectively transmitted than those with continuant or voicing traits. This suggests that labial phonemes play a crucial role in transmitting speech and lexical content in an audiovisual context. Accordingly, studies emphasize the positive impact of visual information on word recognition (Puschmann et al., 2019; Taitelbaum-Swead and Fostick, 2017; Mantokoudis et al., 2013). However, Altieri and Hudock (2014) have argued that audiovisual integration does not provide significant benefits over auditory-only input in word recognition, challenging the general assumption of the advantages of visual information.
Sentence recognition demonstrates a consistently lower visual information effectiveness than stimuli of smaller units. Some studies show increased reliance on visual enhancement during sentence recognition for individuals with severe hearing impairment (Moberly et al., 2020; Moody-Antonio et al., 2005; Hennesy et al., 2022). However, others suggest that the hypothesis that hearing impairment enhances lipreading ability is supported at the word level but not at the sentence level (Tye-Murray et al., 2007). Similarly, Tsao (2019) highlighted audiovisual benefits at the word level, but it remained unclear whether these benefits extend to larger linguistic units, such as sentences. Words and sentences involve additional cognitive processes like lexical access and syntactic parsing, which can either enhance or diminish the role of visual information depending on the context (Jackendoff, 2002). For instance, in sentence-level stimuli, visual cues related to prosody and facial expressions become more relevant, especially for conveying emotions or stress patterns, which are crucial for understanding the overall message (Buchan et al., 2008).
Apart from a trend in segmental units such as phoneme, syllable, word, and sentence, results indicate that for suprasegmental elements, such as intonation and stress within utterance components, visual information does not provide substantial aid (Stevenson et al., 2017; Tsao, 2019). According to Stevenson et al. (2017), CI users derive meaningful visual assistance in phonemic perception and, to a certain extent, for word and sentence recognition tasks. Yet, this assistance is not extended to suprasegmental information. Furthermore, a study on Chinese users (Tsao, 2019) exploring the enhancement of speech perception through AV integration does not demonstrate a discernible contribution of visual elements to tone differentiation.
Overall, smaller linguistic units consistently show benefits from visual information for individuals with hearing impairment, who are strongly affected by labial phonological features. However, as linguistic units increase in complexity, the influence of visual information becomes less pronounced. For smaller units like phonemes and syllables, listeners detect speech by assigning the signal more directly to a phonetic category. In contrast, sentence recognition requires listeners to access phonetic and lexical representations and make corresponding decisions.
4 Hearing loss factors: onset, intervention timing, and cross-modal plasticity
In the preceding two chapters, we reviewed how factors such as age and linguistic level can lead to varying outcomes in the utilization of visual information for speech perception among individuals with hearing loss. Another crucial factor to consider is the characteristics related to hearing loss itself. Specifically, we aim to compare the utilization of visual information based on the onset of hearing loss and the timing of cochlear implant (CI) and hearing aid (HA) interventions.
The timing of hearing loss onset and the point at which auditory devices are worn can significantly influence how individuals perceive speech, moderating the extent to which visual information is utilized (Tyler et al., 1997; Tona et al., 2015; Colmenero et al., 2004; Bavelier et al., 2006; Stevens and Neville, 2006). Additionally, since the onset of hearing loss and the timing of intervention can affect cross-modal plasticity (Kral and Sharma, 2023; Buckley and Tobey, 2011), it is important to examine how these factors impact the utilization of visual information. Therefore, this chapter aims to review (1) the onset and intervention timing of hearing loss, and (2) cross-modal plasticity on the utilization of visual information for speech perception among individuals with hearing loss.
4.1 Three cases based on the onset and intervention timing of hearing loss
To better understand the impact of intervention timing on speech perception outcomes in individuals with hearing loss, we have created a schematic representation of hearing loss onset and intervention timing across pre-lingual and post-lingual periods. Figure 1 presents three case categories based on these factors. The pre-lingual period, typically occurring before age 6, encompasses the critical period for language development. Case 1 involves pre-lingual hearing loss with corresponding pre-lingual intervention. Case 2 represents post-lingual hearing loss with post-lingual intervention, further divided into Case 2.1 (shorter duration before intervention) and Case 2.2 (longer duration). Case 3 illustrates pre-lingual hearing loss with delayed intervention after language acquisition.
Schematic representation of hearing loss onset and intervention timing across pre-lingual and post-lingual periods.
The onset of hearing loss, whether it occurs before or after language acquisition, plays a crucial role in shaping the development of cognitive and perceptual abilities. A study by AuerJr and Bernstein (2007) highlights that individuals with early-onset hearing loss exhibit superior speech-reading abilities compared to those with normal hearing. This finding suggests that these individuals may develop compensatory strategies that enhance their capacity to interpret speech through visual cues, such as lip movements and facial expressions.
Intervention timing is closely related to the duration of hearing loss, with early intervention corresponding to a “shorter duration” (e.g., Case 1, 2.1) and delayed intervention to a longer duration (e.g., Case 2.2, 3). According to research by Gilley et al. (2010), early intervention leads to better audiovisual (AV) integration abilities, even when cochlear implants are worn for the same duration. This suggests that a shorter duration of hearing loss before intervention results in better cognitive outcomes in AV speech processing.
Moreover, in the case of early intervention, particularly for early-onset hearing loss (e.g., Case 1), Jerger et al. (2020) compared children with pre-lingual hearing loss who began using hearing aids around age 2 to their normal-hearing peers. The study found that speech detection and attention significantly improved when visual information was added to auditory-only stimuli. Building on the McGurk effect, Tona et al. (2015) demonstrated that children who received cochlear implants before age 3 utilized visual information to supplement auditory information. Similarly, Iler Kirk et al. (2007) suggested that children with cochlear implants could significantly enhance word recognition through audiovisual (AV) integration facilitated by lip reading.
Comparing intervention timings for individuals with the same onset of hearing loss reveals significant differences in outcomes. Both cases involve early-onset hearing loss, with Case 1 receiving early intervention and Case 3 undergoing intervention later in life. Case 3 involves individuals who experienced pre-lingual deafness but received intervention only after acquiring some language skills, indicating a longer duration of hearing loss. Although such cases are relatively rare and have received limited research attention, Moody-Antonio et al. (2005) provide evidence that some adults with congenital hearing impairment show significant improvements in speech perception after receiving intervention. This suggests that even without early auditory language experience, individuals can integrate auditory information from the intervention with visual language cues, demonstrating potential for enhanced speech perception despite missing the 'critical period' for early intervention.
However, differences in the use of visual information due to the timing of hearing loss and intervention between Case 1 and Case 3 are noteworthy. Research indicates that earlier intervention generally leads to better outcomes compared to later intervention. For instance, Tyler et al. (1997) found higher scores in tasks like vowel recognition and word comprehension when intervention occurred before age 5. Kral et al. (2019) suggest that intervening before age 3 is most effective for language acquisition. Additionally, Kanekama et al. (2010) highlight that intervention within 5 years of onset is particularly beneficial.
Comparing cases with the same onset and intervention timing but differing durations highlights the impact of the duration of hearing loss on language recognition outcomes. Case 2.1 experienced intervention after a relatively short duration of hearing loss, while Case 2.2 had a longer duration before intervention. The decision to categorize Case 2.1 and Case 2.2 based on duration stems from the limitations in existing research. In the study by Baskent and Bazo (2011), the wide age range of participants complicates the ability to isolate the effects of duration due to cognitive decline and age-related factors. Additionally, many studies do not provide specific details about the duration of hearing loss, making it difficult to draw clear conclusions about the effects of different durations within the post-lingual hearing loss group.
Bernhard et al. (2021) suggest that language recognition, assessed through sentence and single-word identification, shows a negative correlation with the duration of hearing loss before cochlear implantation. Lee et al. (2021) confirmed that the duration of hearing loss significantly negatively impacts both audio-only and audio-visual speech recognition. These findings indicate that even with similar onset and intervention timing, a shorter duration of hearing loss (Case 2.1) is more advantageous for language recognition compared to a longer duration (Case 2.2).
In this section, we have analyzed interventions broadly, including both cochlear implants (CI) and hearing aids (HA). However, further detailed comparative analysis of each type of assistive device is warranted. Additionally, it would be valuable to explore not only the duration of hearing loss but also the impact of the length of time using hearing aids or cochlear implants on training outcomes. Furthermore, to address the research gaps identified in the three cases discussed, there is a need for targeted additional studies for each case to better understand their specific contexts and outcomes.
4.2 Cross-modal plasticity with hearing loss patient
Cross-modal plasticity is commonly observed in individuals with hearing impairments and continues to be a factor even after cochlear implantation (Kral and Sharma, 2023; Lomber et al., 2010; Stropahl and Debener, 2017; Bavelier and Hirshorn, 2010; Campbell and Sharma, 2016). when individuals with hearing impairments are deprived of auditory input, much of the auditory cortex is commandeered by visual processing to compensate for the lack of auditory input. This cross-modal compensatory reorganization, where increased emphasis is placed on processing visual information, is a crucial aspect of adaptation in those with hearing impairments and has been pointed out as an important factor in the individual differences in their audio visual speech perception.
Previous studies have often shown that cross-modality may disrupt the integrative functions of the auditory cortex, potentially hindering speech perception adaptation in CI users (Liang et al., 2017). However, plasticity does not refer to a fixed or unchanging property, but rather to the ability to change for adaptation. In other words, the reorganization of the auditory cortex, which has been largely occupied to compensate for adaptation and deprivation before cochlear implant (CI) implantation, can change again after implantation. For example, a study on cross-modal plasticity before and after implantation due to visual speech revealed that cross-modal activation in the auditory regions after implantation is associated with better speech understanding, and that the simultaneous development of activation in the auditory cortex by both visual and auditory speech provides adaptive benefits (Anderson et al., 2017). Another study demonstrated that the imbalanced cross-modal activity in the auditory cortex observed in individuals with hearing impairments returned to normal levels comparable to those of the control group after cochlear implant (CI) implantation. Additionally, the activation of Broca's area, which showed lower activity during speechreading compared to that of normal individuals, increased after CI implantation. Broca's area is known as a brain region that contributes to speech production. This suggests that the recovery of auditory input through CI can plastically adjust the imbalance in the auditory cortex (Rouger et al., 2012).
There is also research suggesting that the use of visual cues varies depending on the proficiency of CI users. For example, Kelly N. Jahn observed that proficient CI users process visual cues differently than non-proficient users, supported by brain imaging studies showing distinct patterns of brain activation in response to visual cues (Jahn et al., 2017; Doucet et al., 2006; Sharma et al., 2015). However, the differences in visual cue utilization patterns suggest a complex relationship with cross-modality, but the exact correlation remains unclear. This is a topic that will require further exploration and research in the future.
The divergence in findings regarding the changes and adaptiveness of cross-modality may be attributed to various factors such as the timing of hearing loss onset–whether it is prelingual or postlingual, the duration of hearing impairment-how long one has experienced hearlong loss. Although previous studies have categorized participants based on the onset of hearing loss, they have observed cross-modality in both groups. Variables like age, language, and the nature of the tasks employed in the studies may also contribute to these varied results. To unravel this complexity, future research must refine the control of these variables. While there is extensive literature on the cross-modality of CI users, particularly concerning the clinical impact of CI on speech perception outcomes, more in-depth studies are needed on the long-term evolution of cross-modality after cochlear implantation and its influence on the integrated perception of speech. Additionally, with advancements in cochlear implant technology, the time required to achieve adaptive and balanced cross-modality may decrease. This remains a promising topic for further exploration and should be actively discussed in future research.
5 Discussion
This review has explored the intricate dynamics of AV speech perception in individuals with hearing loss, with a particular focus on the influences of age, stimuli diversity, and hearing loss factors. Our qualitative analysis reveals that these factors significantly modulate the reliance on and the efficacy of visual information in supplementing auditory cues for speech perception. Building on these findings, we discuss the research gaps related to each factor, propose directions for future studies, and highlight the implications and potential applications of our results.
5.1 Age-related variation
Based on the review of literature across different age groups, it is evident that early life stages of language and cognitive development, as well as the later stages of aging, are particularly critical periods that can significantly influence the use of visual information in speech perception. Specifically regarding aging, considering the variability in results due to task differences and sensory loss (Brooks et al., 2018), additional research that controls for task and age-related characteristics of participants in a consistent manner across age groups is warranted. Furthermore, given the lack of literature on adolescents, recruiting participants from all age groups–infants, children, adolescents, adults, and the elderly–and measuring AV speech perception using the same experimental design would provide clearer insights into age-related effects. Additionally, considering the neuromodulatory effects of aging on sensory integration in speech perception (Yu et al., 2017), there is a need for age-tailored rehabilitation strategies for individuals with hearing loss.
5.2 Linguistic diversity
The variability in response to different linguistic stimuli highlights the complexity of AV speech perception. Labial consonants, for example, are more readily enhanced by visual cues, pointing to the significant role of visually salient speech features in improving perception. However, the effectiveness of AV integration in speech perception is influenced by language-specific viseme configurations, with languages having denser phoneme spaces (e.g., English, French) showing a higher visual dependency. Investigating the efficacy of such tailored interventions across diverse linguistic contexts remains an essential avenue for future research. Also, there is a need for further comparative analysis across linguistic levels, particularly examining the relationship between phoneme recognition and higher linguistic units, as discussed in studies such as Lamoré et al. (1998) to understand how visual information contributes to speech perception as linguistic complexity increases. Moreover, this finding calls for a nuanced approach to auditory rehabilitation, suggesting tailored AV training programs emphasizing visually salient features to enhance speech perception for hearing loss such as Schumann and Ross (2022), or even further, strengthening the connection between these phonetic and lexical representations (Tye-Murray et al., 2016).
5.3 Hearing loss factors
This review has highlighted the critical influence of the onset of hearing loss and the timing of intervention on the utilization of visual information in speech perception. Early intervention, particularly in cases of pre-lingual hearing loss, significantly enhances speech perception by optimizing the use of visual cues. In contrast, later interventions, whether in pre-lingual or post-lingual cases, show varied outcomes influenced by factors such as cognitive decline and the duration of hearing loss. A key factor in these processes is cross-modal plasticity, where the auditory cortex is reorganized to process visual information in response to auditory deprivation. Although this reorganization can initially support speech perception through enhanced visual reliance, its persistence post-cochlear implantation may disrupt the balance between auditory and visual inputs. Therefore, understanding the long-term effects of cross-modal plasticity, particularly how it evolves after CI implantation, is crucial. Future research should investigate the impact of the duration of hearing loss before intervention and how individual differences in cross-modal plasticity influence speech perception outcomes. Additionally, the development of adaptive technologies and tailored rehabilitation strategies that consider these factors could significantly improve audiovisual integration for CI users, ultimately enhancing their speech perception abilities in diverse listening environments.
5.4 General discussion
Moving forward, future research should prioritize the practical applications of visual information in hearing rehabilitation. Developing guidelines for personalized rehabilitation plans that consider age, type of stimuli, and specific characteristics of hearing loss will be crucial. One promising approach involves the integration of AI-assisted training systems, which can customize learning paths based on the user's individual characteristics, such as age, degree of hearing loss, and the use of assistive devices. These systems can provide real-time feedback and adaptive learning by analyzing how effectively a user utilizes visual cues during speech perception tasks. For instance, if a user struggles with specific phonemes or words, the AI can detect this and adjust the training regimen, offering additional practice or emphasizing visual cues more heavily to optimize learning outcomes.
Moreover, AI systems can support the learning of various linguistic and phonetic features, tailoring exercises to the user's linguistic background. For example, in languages with dense phoneme spaces like English, the system could focus more on tasks that rely heavily on visual cues, whereas in vowel-centric languages, it could emphasize the visual recognition of vowels. Additionally, these systems can strengthen the integration of visual and auditory information by using training methods based on phenomena like the McGurk effect, helping users more effectively combine these sensory inputs. The long-term tracking and analysis of a user's progress would allow for continuous refinement of personalized learning strategies, which is particularly beneficial for those receiving later-stage interventions.
By focusing on these areas, we can create more effective, tailored interventions that fully leverage both auditory and visual cues. Such advancements will not only improve speech perception and communication abilities but also significantly enhance the quality of life for those affected by hearing loss. Through this review and discussions, we have gained a deeper understanding of how visual information influences speech perception in individuals with hearing loss, particularly in relation to the critical roles of age, linguistic diversity, and the timing of hearing loss. By integrating these variables into future studies on audiovisual speech perception, we can refine theoretical models, bridge gaps in current research, and enhance the predictive power of future investigations. The ongoing exploration of these factors is essential for advancing our knowledge of sensory integration processes and developing more holistic and personalized approaches to hearing rehabilitation.
The author(s) declare financial support was received for the research, authorship, and/or publication of this article. This work was partly supported by the Institute of Information & Communications Technology Planning & Evaluation (IITP) grant funded by the Korea government (MSIT) (No. 2022-0-00641, XVoice: Multi-Modal Voice Meta Learning, 1/2) and (No. RS-2021-II212068, Artificial Intelligence Innovation Hub, 1/2).
The authors acknowledge the use of OpenAI's ChatGPT (version 4.0) for language suggestions and assistance in drafting portions of the manuscript.
Conflict of interest
The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.
Publisher's note
All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.
ReferencesAltieriN.HudockD. (2014). Hearing impairment and audiovisual speech integration ability: a case study report. Front. Psychol. 5:678. 10.3389/fpsyg.2014.0067825071649AndersonC. A.WigginsI. M.KitterickP. T.HartleyD. E. (2017). Adaptive benefit of cross-modal plasticity following cochlear implantation in deaf adults. Proc. Nat. Acad. Sci. 114, 10256–10261. 10.1073/pnas.170478511428808014AuerJrE. T.BernsteinL. E. (2007). Enhanced visual speech perception in individuals with early-onset hearing impairment. J. Speech Lang. Hear. Res. 50, 1157–1165. 10.1044/1092-4388(2007/08017905902BaskentD.BazoD. (2011). Audiovisual asynchrony detection and speech intelligibility in noise with moderate to severe sensorineural hearing impairment. Ear Hear. 32, 582–592. 10.1097/AUD.0b013e31820fca2321389856BavelierD.DyeM. W.HauserP. C. (2006). Do deaf individuals see better?Trends Cogn. Sci. 10, 512–518. 10.1016/j.tics.2006.09.00617015029BavelierD.HirshornE. A. (2010). I see where you're hearing: how cross-modal plasticity may exploit homologous brain structures. Nat. Neurosci. 13, 1309–1311. 10.1038/nn1110-130920975752BergesonT. R.HoustonD. M.MiyamotoR. T. (2010). Effects of congenital hearing loss and cochlear implantation on audiovisual speech perception in infants and children. Restor. Neurol. Neurosci. 28, 157–165. 10.3233/RNN-2010-052220404405BernhardN.GaugerU.Romo VenturaE.UeckerF. C.OlzeH.KnopkeS.. (2021). Duration of deafness impacts auditory performance after cochlear implantation: a meta-analysis. Laryngoscope Investigat. Otolaryngol. 6, 291–301. 10.1002/lio2.52833869761BernsteinL.TuckerP.DemorestM. (2000). Speech perception without hearing. Percept. Psychophys. 32, 233–252. 10.3758/BF0320554610723205BlackburnC. L.KitterickP. T.JonesG.SumnerC. J.StaceyP. C. (2019). Visual speech benefit in clear and degraded speech depends on the auditory intelligibility of the talker and the number of background talkers. Trends Hear. 23:2331216519837866. 10.1177/233121651983786630909814BrooksC. J.ChanY. M.AndersonA. J.McKendrickA. M. (2018). Audiovisual temporal perception in aging: The role of multisensory integration and age-related sensory loss. Front. Hum. Neurosci. 12:192. 10.3389/fnhum.2018.0019229867415BuchanJ.PareM.MunhallK. (2008). The effect of varying talker identity and listening conditions on gaze behavior during audiovisual speech perception. Brain Res. 1242, 162–171. 10.1016/j.brainres.2008.06.08318621032BuckleyK. A.TobeyE. A. (2011). Cross-modal plasticity and speech perception in pre-and postlingually deaf cochlear implant users. Ear Hear. 32, 2–15. 10.1097/AUD.0b013e3181e8534c20829699BusbyP. A.TongY. C.ClarkG. M. (1984). Underlying dimensions and individual differences in auditory, visual, and auditory-visual vowel perception by hearing–impaired children. J. Acoust. Soc. Am. 75, 1858–1865. 10.1121/1.3909876747096CampbellJ.SharmaA. (2016). Visual cross-modal re-organization in children with cochlear implants. PLoS ONE11:e0147793. 10.1371/journal.pone.014779326807850ColmeneroJ.CatenaA.FuentesL.RamosM. (2004). Mechanisms of visuospatial orienting in deafness. Eur. J. Cognit. Psychol. 16, 791–805. 10.1080/09541440340000312DoucetM.BergeronF.LassondeM.FerronP.LeporeF. (2006). Cross-modal reorganization and speech perception in cochlear implant users. Brain129, 3376–3383. 10.1093/brain/awl26417003067GilleyP.SharmaA.MitchellT.DormanM. (2010). The influence of a sensitive period for auditory-visual integration in children with cochlear implants. Restor. Neurol. Neurosci. 28, 207–218. 10.3233/RNN-2010-052520404409GrantK.WaldenB.SeitzP. (1998). Auditory-visual speech recognition by hearing-impaired subjects: Consonant recognition, sentence recognition, and auditory-visual integration. J. Acoust. Soc. Am. 103, 2677–2672. 10.1121/1.4227889604361HällgrenM.LarsbyB.LyxellB.ArlingerS. (2001). Evaluation of a cognitive test battery in young and elderly normal-hearing and hearing-impaired persons. J. Am. Acad. Audiol. 12, 357–370. 10.1055/s-0042-174562011500010Hay-McCutcheonM. J.PisoniD. B.KirkK. I. (2005). Audiovisual speech perception in elderly cochlear implant recipients. Laryngoscope115, 1887–1894. 10.1097/01.mlg.0000173197.94769.ba16222216HennesyT.CardonG.CampbellJ.GlickH.Bell-SouderD.SharmaA. (2022). Cross-modal reorganization from both visual and somatosensory modalities in cochlear implanted children and its relationship to speech perception. Otol. Neurotol. 43, e872–e879. 10.1097/MAO.000000000000361935970165HoltR. F.KirkK. I.Hay-McCutcheonM. (2011). Assessing multimodal spoken word-in-sentence recognition in children with normal hearing and children with cochlear implants. J. Speech Lang. Hear. Res. 54, 632–657. 10.1044/1092-4388(2010/09-0148)20689028HuyseA.BerthommierF.LeybaertJ. (2013). Degradation of labial information modifies audiovisual speech perception in cochlear-implanted children. Ear Hear. 34, 110–121. 10.1097/AUD.0b013e318267099323059850Iler KirkK.Hay-McCutcheonM. J.Frush HoltR.GaoS.QiR.GerlainB. L. (2007). Audiovisual spoken word recognition by children with cochlear implants. Audiol. Med. 5, 250–261. 10.1080/1651386070167389219823696JackendoffR. (2002). Foundations of Language: Brain, Meaning, Grammar, Evolution. Oxford: Oxford University Press.15377127JahnK. N.StevensonR. A.WallaceM. T. (2017). Visual temporal acuity is related to auditory speech perception abilities in cochlear implant users. Ear Hear. 38:236. 10.1097/AUD.000000000000037927764001JergerS.DamianM. F.KarlC.AbdiH. (2020). Detection and attention for auditory, visual, and audiovisual speech in children with hearing loss. Ear Hear. 41, 508–520. 10.1097/AUD.000000000000079831592903KaiserA. R.KirkK. I.LachsL.PisoniD. B. (2003). Talker and lexical effects on audiovisual word recognition by adults with cochlear implants. J. Speech Lang. Hear. Res. 46, 390–404. 10.1044/1092-4388(2003/032)14700380KanekamaY.HisanagaS.SekiyamaK.KodamaN.SamejimaY.YamadaT.. (2010). “Long-term cochlear implant users have resistance to noise, but short-term users don't,” in Auditory-Visual Speech Processing 2010 (Makuhari: International Speech Communication Association (ISCA)), 1–4.Kishon-RabinL.HarasN.BergmanM. (1997). Multisensory speech perception of young children with profound hearing loss. J Speech Lang. Hear. Res. 40, 1135–1150. 10.1044/jslhr.4005.11359328885KralA.DormanM. F.WilsonB. S. (2019). Neuronal development of hearing and language: cochlear implants and critical periods. Annu. Rev. Neurosci. 42, 47–65. 10.1146/annurev-neuro-080317-06151330699049KralA.SharmaA. (2023). Crossmodal plasticity in hearing loss. Trends Neurosci. 46, 377–393. 10.1016/j.tins.2023.02.00436990952LalondeK.McCreeryR. W. (2020). Audiovisual enhancement of speech perception in noise in school-age children who are hard-of-hearing. Ear Hear. 41:705. 10.1097/AUD.000000000000083032032226LamoréP. J.HuiskampT. M.van SonN. J.BosnianA. J.SmoorenburgG. F. (1998). Auditory, visual and audiovisual perception of segmental speech features by severely hearing-impaired children. Audiology37, 396–419. 10.3109/002060998090729929888195Lasfargues-DelannoyA.StrelnikovK.DeguineO.MarxM.BaroneP. (2021). Supra-normal skills in processing of visuo-auditory prosodic information by cochlear-implanted deaf patients. Hear. Res. 410:108330. 10.1016/j.heares.2021.10833034492444LeeH. J.LeeJ. M.ChoiJ. Y.JungJ. (2021). The effects of preoperative audiovisual speech perception on the audiologic outcomes of cochlear implantation in patients with postlingual deafness. Audiol. Neurotol. 26, 149–156. 10.1159/00050996933352550LeiJ.FangJ.WangW.MeiY. (2008). The visual-auditory effect on hearing-handicapped students in lip-reading chinese phonetic identification. Psychol. Sci. 31, 312–314.LewisD. E.SmithN. A.SpaldingJ. L.ValenteD. L. (2018). Looking behavior and audiovisual speech understanding in children with normal hearing and children with mild bilateral or unilateral hearing loss. Ear Hear. 39:783. 10.1097/AUD.000000000000053429252979LeybaertJ.LaSassoC. J. (2010). Cued speech for enhancing speech perception and first language development of children with cochlear implants. Trends Amplif. 14, 96–112. 10.1177/108471381037556720724357LeybaertJ.LaSassoC. J. (2010). Cued speech for enhancing speech perception and first language development of children with cochlear implants. Trends. Amplif.14, 96–112. 10.1177/108471381037556720724357LiangM.ChenY.ZhaoF.ZhangJ.LiuJ.ZhangX.. (2017). Visual processing recruits the auditory cortices in prelingually deaf children and influences cochlear implant outcomes. Otol. Neurotol. 38, 1104–1111. 10.1097/MAO.000000000000149428727651LomberS. G.MeredithM. A.KralA. (2010). Cross-modal plasticity in specific auditory cortices underlies visual compensations in the deaf. Nat. Neurosci. 13, 1421–1427. 10.1038/nn.265320935644MantokoudisG.DählerC.DubachP.KompisM.CaversaccioM. D.SennP. (2013). Internet video telephony allows speech reading by deaf individuals and improves speech perception by cochlear implant users. PLoS ONE8:e54770. 10.1371/journal.pone.005477023359119MitchelA. D.LuskL. G.WellingtonI.MookA. T. (2023). Segmenting speech by mouth: The role of oral prosodic cues for visual speech segmentation. Lang. Speech66, 819–832. 10.1177/0023830922113760736448317MoberlyA. C.VasilK. J.RayC. (2020). Visual reliance during speech recognition in cochlear implant users and candidates. J. Am. Acad. Audiol. 31, 030–039. 10.3766/jaaa.1804931210633Moody-AntonioS.TakayanagiS.MasudaA.Auer JrE. T.FisherL.BernsteinL. E. (2005). Improved speech perception in adult congenitally deafened cochlear implant recipients. Otol. Neurotol. 26, 649–654. 10.1097/01.mao.0000178124.13118.7616015162MoradiS.LidestamB.DanielssonH.NgE. H. N.RönnbergJ. (2017). Visual cues contribute differentially to audiovisual perception of consonants and vowels in improving recognition and reducing cognitive demands in listeners with hearing impairment using hearing aids. J. Speech, Lang. Hear. Res. 60, 2687–2703. 10.1044/2016_JSLHR-H-16-016028651255MunhallK. G.JonesJ. A.CallanD. E.KuratateT.Vatikiotis-BatesonE. (2004). Visual prosody and speech intelligibility: head movement improves auditory speech perception. Psychol. Sci. 15, 133–137. 10.1111/j.0963-7214.2004.01502010.x14738521Oryadi-ZanjaniM. M.VahabM.BazrafkanM.HaghjooA. (2015). Audiovisual spoken word recognition as a clinical criterion for sensory aids efficiency in persian-language children with hearing loss. Int. J. Pediatr. Otorhinolaryngol. 79, 2424–2427. 10.1016/j.ijporl.2015.11.00426607564Oryadi-ZanjaniM. M.VahabM.RahimiZ.MayahiA. (2017). Audiovisual sentence repetition as a clinical criterion for auditory development in persian-language children with hearing loss. Int. J. Pediatr. Otorhinolaryngol. 93, 167–171. 10.1016/j.ijporl.2016.12.00928109491PeelleJ. E.SommersM. S. (2015). Prediction and constraint in audiovisual speech perception. Cortex68, 169–181. 10.1016/j.cortex.2015.03.00625890390PralusA.HermannR.CholvyF.AgueraP.-E.MoulinA.BaroneP.. (2021). Rapid assessment of non-verbal auditory perception in normal-hearing participants and cochlear implant users. J. Clin. Med. 10:2093. 10.3390/jcm1010209334068067PuschmannS.DaeglauM.StropahlM.MirkovicB.RosemannS.ThielC. M.. (2019). Hearing-impaired listeners show increased audiovisual benefit when listening to speech in noise. Neuroimage196, 261–268. 10.1016/j.neuroimage.2019.04.01730978494RegenbogenC.SchneiderD. A.GurR. E.SchneiderF.HabelU.KellermannT. (2012). Multimodal human communication–targeting facial expressions, speech content and prosody. Neuroimage60, 2346–2356. 10.1016/j.neuroimage.2012.02.04322487549RigoT. G.LiebermanD. A. (1989). Nonverbal sensitivity of normal-hearing and hearing-impaired older adults. Ear Hear. 10, 184–189. 10.1097/00003446-198906000-000082744255RougerJ.FraysseB.DeguineO.BaroneP. (2008). Mcgurk effects in cochlear-implanted deaf subjects. Brain Res. 1188, 87–99. 10.1016/j.brainres.2007.10.04918062941RougerJ.LagleyreS.DémonetJ.-F.FraysseB.DeguineO.BaroneP. (2012). Evolution of crossmodal reorganization of the voice area in cochlear-implanted deaf patients. Hum. Brain Mapp. 33, 1929–1940. 10.1002/hbm.2133121557388SatoT.YabushitaT.SakamotoS.KatoriY.KawaseT. (2020). In-home auditory training using audiovisual stimuli on a tablet computer: Feasibility and preliminary results. Auris Nasus Larynx47, 348–352. 10.1016/j.anl.2019.09.00631708168SchumannA.RossB. (2022). Adaptive syllable training improves phoneme identification in older listeners with and without hearing loss. Audiol. Res. 12, 653–673. 10.3390/audiolres1206006336412658SharmaA.CampbellJ.CardonG. (2015). Developmental and cross-modal plasticity in deafness: evidence from the P1 and N1 event related potentials in cochlear implanted children. Int. J. Psychophysiol. 95, 135–144. 10.1016/j.ijpsycho.2014.04.00724780192StevensC.NevilleH. (2006). Neuroplasticity as a double-edged sword: deaf enhancements and dyslexic deficits in motion processing. J. Cogn. Neurosci. 18, 701–714. 10.1162/jocn.2006.18.5.70116768371StevensonR. A.SheffieldS. W.ButeraI. M.GiffordR. H.WallaceM. T. (2017). Multisensory integration in cochlear implant recipients. Ear Hear. 38, 521–538. 10.1097/AUD.000000000000043528399064StropahlM.DebenerS. (2017). Auditory cross-modal reorganization in cochlear implant users indicates audio-visual integration. NeuroImage: Clini. 16, 514–523. 10.1016/j.nicl.2017.09.00128971005SuessN.HauswaldA.ZehentnerV.DepireuxJ.HerzogG.RöschS.. (2022). Influence of linguistic properties and hearing impairment on visual speech perception skills in the german language. PLoS ONE17:e275585. 10.1371/journal.pone.027558536178907Taitelbaum-SweadR.FostickL. (2017). Audio-visual speech perception in noise: Implanted children and young adults versus normal hearing peers. Int. J. Pediatr. Otorhinolaryngol. 92, 146–150. 10.1016/j.ijporl.2016.11.02228012517TonaR.NaitoY.MorotoS.YamamotoR.FujiwaraK.YamazakiH.. (2015). Audio-visual integration during speech perception in prelingually deafened japanese children revealed by the mcgurk effect. Int. J. Pediatr. Otorhinolaryngol. 79, 2072–2078. 10.1016/j.ijporl.2015.09.01626455920TsaoY.-W. (2019). Audiovisual Speech Perception in People With Hearing Loss Across Languages: A Systematic Review of English and Mandarin. Tucson, AZ: The University of Arizona. Available at: http://hdl.handle.net/10150/633080Tye-MurrayN.SommersM. S.SpeharB. (2007). Audiovisual integration and lipreading abilities of older adults with normal and impaired hearing. Ear Hear. 28, 656–668. 10.1097/AUD.0b013e31812f718517804980Tye-MurrayN.SpeharB.SommersM.BarcroftJ. (2016). Auditory training with frequent communication partners. J. Speech Lang. Hear. Res. 59:871. 10.1044/2016_JSLHR-H-15-017127567016TylerR. S.Fryauf-BertschyH.KelsayD. M.GantzB. J.WoodworthG. P.ParkinsonA.. (1997). Speech perception by prelingually deaf children using cochlear implants. Otolaryngol.-Head Neck Surg. 117, 180–187. 10.1016/S0194-5998(97)70172-49334763Van SoerenD. (2023). The role of word recognition factors and lexical stress in the distribution of consonants in spanish, english and dutch. J. Linguist. 59, 149–177. 10.1017/S0022226722000081YamamotoR.NaitoY.TonaR.MorotoS.TamayaR.FujiwaraK.. (2017). Audio-visual speech perception in prelingually deafened japanese children following sequential bilateral cochlear implantation. Int. J. Pediatr. Otorhinolaryngol. 102, 160–168. 10.1016/j.ijporl.2017.09.02229106867YuL.RaoA.ZhangY.BurtonP. C.RishiqD.AbramsH. (2017). Neuromodulatory effects of auditory training and hearing aid use on audiovisual speech perception in elderly individuals. Front. Aging Neurosci. 9:30. 10.3389/fnagi.2017.0003028270763ZhangY.FrassinelliD.TuomainenJ.SkipperJ. I.ViglioccoG. (2021). More than words: Word predictability, prosody, gesture and mouth movements in natural language comprehension. Proc. Royal Soc. B. 288:20210500. 10.1098/rspb.2021.050034284631