Visual and auditory information are the two main sources of external information for humans, accounting for about 90 percent of the total (Treichler, 1967). The phenomenon of audio-visual integration exists everywhere in people’s lives, such as the integration of sound and subtitles when watching TV, the integration of speech and oral information when communicating with others, and the effective integration of information from the visual and auditory channels into a unified and coherent perceptual (Tang et al., 2016; Parker and Robinson, 2018). Audiovisual integration is able to integrate ambiguous information to make better inferences about external connections (Ernst and Bülthoff, 2004; Noppeney, 2021) and thus plays an important role in the human brain’s understanding of external information.

Audiovisual integration is influenced by many factors, including spatial (Nidiffer et al., 2016; Spence and Frings, 2020), temporal (Stevenson and Wallace, 2013), and attentional (Donohue et al., 2015; Mole, 2020) factors. Attention has top-down or bottom-up control over multisensory integration (Meyer et al., 2018), and endogenous, goal-driven attention, mediated by the dorsal attention network, is particularly effective in enhancing audiovisual integration under attended conditions (Talsma et al., 2007; Talsma et al., 2010). Whereas exogenous attention, often controlled by the ventral attention network, tends to override endogenous cues, resulting in reflexive spatial localization (Tang et al., 2016). These attentional processes guide the orientation of attention, with studies showing that the frontoparietal network, including both dorsal (for spatial attention) and ventral (for stimulus-driven attention) pathways attention (Landry et al., 2024), proposed by Posner (1980) in 1980, demonstrating that when cues produce short intervals between the cue and the target (< 250 ms), prior exogenous cues facilitated the subject’s response to the next target, but may also produce Inhibition of return (Van der Stoep et al., 2017). An fMRI study (Yan et al., 2015) measured cortical areas activated in the visual and auditory domains in a cue-target spatial attention paradigm. The results suggest that interactions between multisensory inputs can lead to enhancement or inhibition of cortical responses with top-down spatial attention. Thus audiovisual integration and attention are both important mechanisms, capable of recognizing the external world by enhancing sensory perception (Spence, 2010), and their interaction is particularly important (Talsma et al., 2010; Tang et al., 2016).

In the field of experimental psychology, it is common to use mean response time (RT) and proportions of correct responses (PC) to ideally quantify subject behavioral performance, however, it does not take into account the subject’s tendency to answer the questions, whether their strategy is to be more precise or faster at completing the task, which brings us to the speed-accuracy trade-off (SAT) (Luce, 1991; Heitz, 2014). SAT suggests that intra-or inter-subject trade-offs are often unpredictable (Gueugneau et al., 2017), For example, some researchers have focused on the ‘posterior response delay’ that occurs when subjects make a mistake (Thura et al., 2017). Thus, RT or PC are often very vague concepts, and by integrating them we can further focus on behavioral changes in one direction or another (Liesefeld et al., 2015). Inverse Efficiency Score (IES), Rate-Correct Score (RCS), Linear Integrated Speed-Accuracy Score (LISAS) and Balanced Integration Score (BIS) are important composite behavioral indicators for evaluating behavior in the field of psychology (Liesefeld and Janczyk, 2019).

Functional connectivity has been shown to correlate with transient mental states (Quinn et al., 2018), and the extent of its influence on task performance can also be established (Sadaghiani et al., 2015). However, this approach only reveals fixed intrinsic connectivity and behavioral properties at rest-state. In contrast, task-state fMRI allows for a more accurate characterization of behavioral states in a given cognitive direction and establishes a more direct link to behavioral contributions through local neural activity. Despite this, it remains unclear how different types of attention affect the behavioral strategies for audiovisual integration, particularly the shifting focus between accuracy and speed. Furthermore, how brain regions within the dorsal and ventral attention pathways fine-tune their functions and correlate with behavioral performance still requires further exploration.

Therefore, our study aims to establish the link between brain activity and behavioral representations of audiovisual integration under endogenous attentional conditioning. By combining basic and composite behavioral indicators, the study applies task-state fMRI to design three tasks—auditory, visual, and audiovisual—under three types of endogenous attention: spatial, temporal, and neutral, based on the classical Posner experimental paradigm. Behavioral and fMRI data were analyzed to explore differences in behavioral indicators and activated brain regions across the A, V, and VA tasks under spatial, temporal, and neutral attention conditions. The goal is to uncover the functional characteristics of the dorsal and ventral attention pathways in audiovisual integration and to analyze the key brain regions associated with behavioral indicators in the attention network, clarifying their regulatory mechanisms.

In our study, we examined the relationship between brain activation and behavioral representations during audiovisual integration under endogenous attention, specifically focusing on how different types of endogenous attention—spatial, temporal, and neutral—affect behavioral strategies and modulate brain activity along the dorsal and ventral attention pathways. The auditory, visual, and audiovisual tasks based on the classical Posner experimental paradigm were designed. Behavioral data and task-state fMRI data were analyzed to explore differences in activation levels across subjects for the A, V, and VA tasks under different attention conditions. Additionally, we examined whether there were variations in brain region activation, particularly within the dorsal and ventral attention pathways, and explored correlations between activation levels and behavioral representations in each task. This approach highlights the focus on how endogenous attention modulates behavior and neural activity, emphasizing the functional roles of the dorsal and ventral attention pathways in audiovisual integration.

The Visual spatial cue (an arrow), first developed by Posner (1980), providing a spatial hint of the location of an upcoming target stimulus (Ristic and Kingstone, 2006), which are widely used to study the top-down spatial attention. We used 20% ineffective spatial cues to increase visual sensitivity (Solomon, 2004) to enhance attention-directing effects. We used circles as visual temporal cues that provide short or long-time intervals between the cue and the target stimulus, which would guide top-down time-related neural correlations. A study (Olk, 2014) showed that at around 300 ms, the study indicated the presence of a strong cueing effect, whereas greater than 1,500 ms, the cueing effect declined and a clear pattern of inhibition of return emerged. Thus temporal contextual differences in our experiments will affect the allocation of attention, revealing the importance of time-related implicit factors in attention (Olson and Chun, 2001). Neutral cues, on the other hand, are used as nonspecific processes to reveal purely attentional processes. The study demonstrated that we found extensive activation of FEF and IPL in brain activations for all three types of tasks, confirming that the dorsolateral frontoparietal (dFPN) network regulates top-down attention (Ptak et al., 2017) in visual and auditory targets with similarity in neural activation (Smith et al., 2010), suggesting that our experimental design successfully guided the attentional processes. We conducted a practice/training course with all subjects prior to the MR scan, which was conducted to ensure that subjects were familiar with the details of the tasks, and the correctness rate for all three tasks was above 95%, suggesting that the complexity and difficulty of the tasks were balanced. According to the analysis of the behavioral data on the process of audiovisual integration under endogenous attention, the results of the pairwise comparisons of the A, V, and VA tasks showed that there were significant differences in the psychological behavioral indicators including RT, IES, RCS, LISAS, and BIS, and that there was a significant difference in VA versus A and VA versus V for PC. Expressed in the VA task subjects responded more slowly and less correctly, expended more energy on average, responded less efficiently correctly, were less able to switch tasks, and were less able to make judgements about the difficulty of the task. Overall behavioral performance on the VA task was weaker. The classical MLE model assumes that visual and auditory signals of the same origin are integrated into a unified representation and weighted for sensory reliability, but sensory weights are subject to top-down modulation (Vercillo and Gori, 2015). The VA task is more complex than the single-channel tasks, as the brain must attend to information from both the visual and auditory channels, and then make a judgment about whether the stimuli are from the same side before issuing a response. This process is more complex and time-consuming, with sensory weights adjusted due to the spatial inconsistency that may arise in the VA task (Ernst and Banks, 2002; Rohe and Noppeney, 2018). This selective control reduces the gains from multiple senses, and being stimulated on different sides creates challenges in allocating attentional resources (Vatakis et al., 2007). Our findings highlight how the VA task requires different behavioral strategies, particularly in balancing speed and accuracy. The lower values in composite behavioral indicators like LISAS and BIS suggest that the increased complexity of integrating auditory and visual information leads to greater cognitive load, requiring more attention. The differences in these behavioral indicators emphasize the challenge of managing the trade-off between speed and accuracy in the VA task.

Differences in activated brain regions for the three tasks A, V, and VA under spatial, temporal, and neutral cues showed consistency in neural-level characterization, including the frontal eye field, inferior parietal lobule., supplementary motor area, superior temporal gyrus, middle occipital gyrus and cuneus brain regions. The frontal eye field, inferior parietal lobule and supplementary motor area belong to the dorsal frontoparietal network, an endogenous attentional task-activated brain region (Tang et al., 2016). These regions are involved in the regulation of spatial attention and task-related processing. The frontal eye field, inferior parietal lobule and superior temporal gyrus belonging to the association cortex, which processes information from visual & auditory channels and is responsible for audiovisual semantic integration (Li et al., 2016). The middle occipital gyrus and cuneus are primarily associated with visual information processing and belong to the primary visual cortex. These regions’ activation patterns highlight how attention networks, particularly the dorsal pathway, help allocate cognitive resources during task processing and facilitate the integration of audiovisual information.

In a correlation analysis of activation levels with behavioral indicators (Table 5), we used two basic behavioral metrics and four composite metrics and found that brain activity in multiple brain regions was behaviorally relevant and overlapped with task-related activation in three tasks, A, V, and VA, under spatial, temporal, and neutral cues. These overlapping brain regions suggest a direct link between their activity patterns during task performance and behavioral performance. The cuneus, lingual gyrus, and middle occipital gyrus are associated with visual information processing, and the inferior temporal gyrus and middle temporal gyrus are associated with visual information processing and belong to more advanced regions of visual processing. The angular gyrus and supramarginal gyrus are engaged in auditory information processing, with the angular gyrus also playing a role in decision-making by directing attention to relevant reward-related information in the visual environment (Studer et al., 2014). The fusiform gyrus, part of the multimodal association cortex, integrates sensory inputs from both visual and auditory channels. The frontal eye field, inferior parietal lobule, and superior parietal gyrus belong to the dorsal frontoparietal network and are involved in spatial attention and task-related focus (Chica et al., 2013), playing a crucial role in directing attentional resources. We found correlations with PC and BIS in the overlapping left Inferior parietal lobule, suggesting that IPL relies on accuracy in auditory tasks guided by visual endogenous attention, while top-down integration is enhanced in visual or audiovisual to balance overall task performance. The ventral attention network, including regions like the angular gyrus and superior temporal gyrus, is more stimulus-driven, helping the brain shift attention to relevant stimuli. We found in the overlapping bilateral Superior temporal gyrus that participation in endogenous attentional guidance in a single-channel task improves task completion by modulating information processing in a single modality to accomplish bottom-up allocation of attention. In addition, the hippocampus, parahippocampal gyrus, and temporal pole are associated with memory storage (Wagner et al., 2020; Setton et al., 2022), but also contribute to spatial perceptual learning and recognition of landmark objects (Jeffery, 2018; Sun et al., 2021), reflecting broader attentional modulation. These findings highlight how both dorsal and ventral attention pathways are engaged in managing cognitive load and balancing the integration of visual and auditory information.

In RT and PC correlations with brain activation we found broader significance results, with PC being more broadly significant than RT, suggesting that better PCs (higher grades) require more mobilization of perceptual resources than better RTs (faster speeds) (Helfrich et al., 2019; Hunter, 2021). In terms of RT (faster), we only observed a negative correlation of brain region activation accompanied by RT in neutral attention, which does suggest that increased information processing leads to slower reaction times when more brain activation intervenes, which may be related to the deliberate decision-making performed by the subjects (average task PC > 95%) (Vallesi et al., 2021; Schneider et al., 2022). Unlike single-channel stimuli, we observed differences in the correlation between RT and PC for spatial & temporal attention in the VA task, where increased brain activated areas resulted in increased RT and decreased PC, which is the opposite of neutral attention, and is manifested in areas dominated by sensory integration and visuo-spatial processing, suggesting that increased top-down spatio-temporal modulation mobilizes additional cognitive resources and thus affects the task performance strategy (Vidaud-Laperrière et al., 2022; Carmona et al., 2024). Improvements in task performance require greater mobilization of perceptual resources, while increased task complexity under different attentional modalities may alter an individual’s execution strategy, thereby affecting the relationship between reaction time and performance scores. Spatial and temporal attention are modulated, and individuals not only mobilize more cognitive resources, but may also respond to the challenges of the task by adjusting their task strategies.

In the IES we did not obtain a significance correlation in the A task, and in the V task we obtained an increase in spatial and neutral attention for the average energy consumed by the system over trials (Townsend, 1983) in the visual processing area. This exacerbates the energetic deepening of visual space in spatial attention, which differs from the monitoring function of visual targets in neutral attention, and follows the same trend as in VA, as evidenced by increased activation of visually and spatially relevant brain regions consuming energy in spatial and temporal attention, and energy savings in higher cognitive regions in neutral attention. In correlation with the RCS then characterized the efficiency of correct response trials, both of which negatively correlated with brain activation in the V task, as demonstrated by the processing of basic visual sensory information in spatial attention and the processing of higher cognition in temporal & neutral attention. This indicates that under spatial attention, the processing of basic visual information is associated with higher cognitive processing under temporal and neutral attention, suggesting that there are differences in top-down attentional modulation of cognitive strategies (Hahn et al., 2006; Li et al., 2012). Both LISAS and BIS are balanced for speed-accuracy, and the correlation differences between these two sets of metrics reveal differences between linear and nonlinear processing patterns in balancing cognitive resources in different brain regions. Our results reveal that the introduction of spatio-temporal attentional modulation in the A task increases visuo-spatial orienting processing would show accuracy, whereas in neutral attention it affects behavioral performance. This is the same in both indicators, whereas the LISAS and BIS show differential processing patterns after the addition of visual stimuli (V&VA task) and are completely reversed in the VA task, suggesting a dynamic modulation of the top-down attentional orientated processing audiovisual integration (Todd and Manaligod, 2018; Zeraati et al., 2023). Different attentional modalities not only modulate the brain’s workload, but also influence the efficiency of task performance, especially the balance between visuo-spatial and higher cognitive processing. Furthermore, audiovisual integration performance was affected by the modulation of top-down attention, suggesting that the brain optimizes task speed and accuracy through flexible resource allocation in the face of multimodal information.

Three audiovisual tasks based on the Posner experimental paradigm were designed to calculate behavioral indicators, brain region activation differences, and correlations between brain activation and behavioral representations under spatial, temporal, and neutral cues, respectively, for each task. Significant differences in behavioral performance were found across tasks, with the dual-channel task performing weaker than the single-channel task. Consistent and widespread brain activation was demonstrated in the frontal eye field, inferior parietal lobule, supplementary motor area, superior temporal gyrus, middle occipital gyrus and cuneus brain regions. In the correlations between behavioral indicators and brain activation, PC had broader differential results. In audiovisual tasks, spatio-temporal attentional modulation mobilizes additional cognitive resources and influences executive strategies compared to neutral attention. IES reveals endogenous attentional modulation of mean energy expenditure in dual-channel tasks, and the RCS reveals between-task differences in the efficiency of responding correctly. LISAS and BIS show different patterns of speed-accuracy balance in audiovisual tasks, indicating dynamic modulation of cognitive processes by attention networks. Dual attention pathways are involved in managing cognitive load, balancing the integration of visual and auditory information, and influencing task performance strategies.