Global field power, as a reference-independent measurement metric, was used to evaluate the overall electrical activity of brain topographic maps. Since local maxima of the GFP curve typically correspond to stable topological configurations and high signal-to-noise ratios, this study first calculated GFP values at each time point within 0–800 ms and selected scalp potential maps corresponding to local GFP peaks as the initial clustering input prior to cluster analysis. GFP was derived by computing the sum of squared differences between all electrode potentials and the mean potential, with its value reflecting the spatial consistency of EEG signals. Based on AV EEG data under different frequency band conditions, GFP calculation provided highly representative initial topographic maps for subsequent clustering.

Subsequently, a Topographical Atomize and Agglomerate Hierarchical Clustering-based microstate analysis was performed. First, local GFP peak points were identified, with their potential distributions treated as candidate microstates. Initially, the potential map of each local peak was considered an independent cluster. Cluster similarity was measured using Pearson correlation coefficients, and the most similar clusters were iteratively merged to form microstate templates. This bottom-up process gradually reduced the number of clusters, ultimately generating microstate templates for each candidate cluster number.

The selection of the optimal cluster number requires balancing model complexity and interpretability. Common evaluation methods include GEV that maximizes the cumulative GEV to ensure selected clusters sufficiently explain spatial features of EEG signals; cross-validation (CV) which partitions data into training/validation sets to assess model stability across cluster numbers, selecting the number with minimal generalization error; and KL criterion which identifies inflection points where the rate of change in cluster compactness (e.g., within-cluster similarity) significantly declines.

After determining the optimal cluster number, microstate backfitting and refinement were conducted. During backfitting, the potential map of each time point was matched to all microstate templates via Pearson correlation coefficients, assigning it to the most similar class to form continuous microstate time series. This process was iterated until assignments stabilized. Short-duration noise segments were removed using a window smoothing algorithm, yielding stable microstate sequences reflecting AV processing dynamics. This methodology enabled efficient spatiotemporal modeling and analysis of EEG signals. This workflow is illustrated in Figure 3.

In this study, we proposed a method for evaluating the clustering results of task-state EEG microstates. The KL_GEV evaluation metric was applied to AV EEG signals, and the microstate classifications selected under the KL_GEV, KL, and CV evaluation metrics were compared based on the variance ratio criterion (CH Index) and the silhouette coefficient.

The calculation principle of GEV is based on the idea of decomposing the total variance into the portion explained by the model and the portion unexplained by the model. By comparing the sizes of these two portions, the proportion of total variance explained by the model can be determined. Specifically, the formula is as follows:

Where T is the total number of time points, and y ( t ) is the observed value vector at time point t .

KL-GEV is an improved method based on the KL criterion. It determines the optimal number of microstate classifications by analyzing the rate of change of GEV as the number of classifications increases. Its core idea is to capture the “point of diminishing marginal returns” in GEV growth, i.e., the inflection point where the improvement in model-explained variance significantly weakens with the addition of more classifications. Its calculation formula is as follows:

The larger the K L GEV value, the smaller the impact of adding one more microstate to the growth of GEV. Therefore, the optimal number of microstate classifications can be considered as the point where K L GEV is maximized.

The KL-GEV microstate clustering number selection criterion was compared with CV and KL using evaluation metrics such as the CH index and the silhouette coefficient.

The CH index essentially represents the ratio of between-cluster distance to within-cluster distance, and its overall calculation process is similar to that of variance, hence it is also referred to as the variance ratio criterion.

The silhouette coefficient measures the separation between clusters by comparing the similarity of each object to its own cluster with its similarity to objects in other clusters. First, the mean distance (within-cluster distance) between sample point and all other sample points in its cluster is calculated, and then the ratio of this distance to the mean distance (between-cluster distance) between the sample point and all sample points in the nearest other cluster is computed. The silhouette coefficient ranges from −1 to 1, with higher values indicating better clustering performance.

In this study, we filtered the preprocessed EEG data during AV stimulus presentation to obtain AV processing EEG data in four frequency bands: delta, theta, alpha, and beta. Using the aforementioned methods for calculating EEG microstates during brain processing of AV tasks, microstates under both attended and unattended conditions were computed for each of the four frequency bands. Together with the original EEG data before frequency division, we ultimately obtained microstates for 2 (attended and unattended conditions) × 5 (original, delta, theta, alpha, and beta bands) = 10 types of AV processing, as shown in Figure 3.

Further calculations were performed to determine the following properties for each type of microstate under these 10 conditions: Duration, Coverage, Occurrence, and Transition Probabilities. Among these, Duration was used to describe the length of time each microstate persists in the EEG signal. Coverage was used to describe the proportion of a specific microstate in the entire EEG signal. Occurrence refers to the number of times a specific microstate appears per unit time. The Transition Probability between microstates refers to the likelihood of transitioning from a specific microstate to another microstate.

This study calculated the microstate properties (Duration, Coverage, Occurrence, and Transition Probabilities) under both attended and unattended conditions for the original EEG, as well as the delta, theta, alpha, and beta frequency bands. These microstate time series, which incorporate frequency-domain features, provide a time-frequency domain feature set for analyzing the AV processing stages and the regulatory role of attention. Based on the original EEG, we computed the microstates of AV processing, thereby dividing the processing stages into multiple sub-stages. Similarly, we were able to obtain the sub-stages of AV processing in the delta, theta, alpha, and beta frequency bands.

To explore the characteristics of the sub-stages of AV processing, we compared the microstate properties derived from the original EEG with those from each frequency band. For the microstate properties of Duration, Coverage, Occurrence, and Transition Probability, a repeated measures analysis of variance (ANOVA) (N Microstates conditions [MS1, MS2, … and MSn]) as within-subject factors was conducted separately. Paired-t tests were conducted in the presence of main or interaction effects, and for origin EEG and each frequency, all the microstate properties of Duration, Coverage, Occurrence, and Transition Probability were separately entered into a repeated measures ANOVA [two Attention conditions (attended and unattended)] as within-subject factors, to find significant differences caused by the attention mechanism. All statistical analyses were conducted using IBM SPSS software (version 22, IBM Inc., Armonk, NY) for Windows. Greenhouse–Geisser corrections were applied with adjusted degrees of freedom. Effects and correlations were considered significant when p < 0.05.

To validate that the time-frequency domain features calculated using multi-band microstates could effectively characterize the regulatory role of attention in AV processing, we applied several classical machine learning algorithms to classify brain activities under attended and unattended conditions based on these features. Additionally, we extended this method to classify brain activities during unimodal visual processing, unimodal auditory processing, and AV information processing, further verifying that the time-frequency domain features derived from multi-band microstates can effectively represent brain activities during different information processing tasks.

For the classification experiments in this study, six classical machine learning models were employed: SVM, Random Forest, Gradient Boosting, k-nearest neighbors (KNN), Logistic Regression, and Linear Discriminant Analysis (LDA). The parameter Settings of these six machine learning models are shown in Table 1. The specific experimental setup included an i7-9750H processor, an NVIDIA GeForce RTX 1660 Ti graphics card, 16 GB of memory, and the Windows 10 Pro 64-bit operating system. All experiments were conducted in the MATLAB R2022a environment, using 5-fold cross-validation to evaluate the classification performance of each model. Evaluation metrics included accuracy, precision, recall, and F1 score.

Current research typically divides microstates during the resting state of the brain into four categories: A, B, C, and D. However, when processing AV information, the brain is in a task state, and the changes in scalp electric field distribution are related to the neural mechanisms of AV information processing. Therefore, it is not appropriate to simply categorize microstates in the same way as in the resting state.

We proposed the use of KL_GEV to evaluate the number of microstate clusters, thereby determining the optimal number of microstates for AV information processing. Our KL_GEV calculation results are shown in Table 2. Under attended conditions, the optimal number of microstates for unfiltered AV information processing was six. For the delta, theta, alpha, and beta frequency bands, the optimal numbers of microstates were six, eight, four, and 11, respectively. Under unattended conditions, the optimal number of microstates for unfiltered AV information processing was four. For the delta, theta, alpha, and beta frequency bands, the optimal numbers of microstates were four, 14, five, and eight, respectively.

We employed the CH Score and silhouette coefficient as evaluation metrics for clustering performance, comparing them with two other classical methods for determining the optimal number of clusters: CV and KL.

When selecting the number of microstate categories based on CV, two main approaches were used. The first involved observing the slope changes in the CV curve. Although CV continues to decrease as the number of microstate categories increases, the rate of decrease may significantly slow down after a certain number of categories (the “elbow point”), which can be considered a candidate for the optimal number of categories. The second approach involved physiological constraints, referencing typical numbers of microstate categories in task states from existing literature (e.g., four to seven categories) to avoid selecting excessively high and less interpretable numbers.

The CH Index essentially represents the ratio of between-cluster distance to within-cluster distance, and its calculation process is similar to that of variance, hence it is also referred to as the variance ratio criterion. The silhouette coefficient measures the separation between clusters by comparing the similarity of each object to its own cluster with its similarity to objects in other clusters. The results are shown in Table 2 and Figure 4. These results showcase the CH index and silhouette coefficient of the selected clusters for each microstate clustering method across each frequency band.

From the above results, it can be observed that the brain processes AV information regardless of whether attentional resources are engaged. Under attended conditions, the AV information processing can be represented by an alternating sequence of six microstates, while under unattended conditions, it is represented by four microstates. This may suggest that the processing of AV information by the brain is more refined and complex when attentional resources are engaged. This finding aligns with conclusions from some previous studies, further highlighting the importance and value of classifying brain microstates for research purposes. From the perspective of information processing, the handling of AV information by the brain is an extremely complex process involving numerous levels and types of neural activities. By classifying microstates, these complex neural activities can be systematically organized and categorized, clearly revealing the specific patterns of information processing under different conditions. For example, in this study, distinguishing between six microstates under attended conditions and four microstates under unattended conditions allows us to intuitively observe the impact of attentional resource allocation on the refinement and complexity of information processing, providing a framework for a deeper understanding of the information processing mechanisms of the brain. From the perspective of exploring neural mechanisms, different microstates may represent the activation of distinct neural functional modules or neural circuits. Classifying microstates helps us identify the specific neural regions and pathways involved in AV information processing. The alternation of different microstates may reflect the dynamic interactions between these neural regions. By analyzing these microstates, we can better uncover the mysteries of brain neural mechanisms and clarify the specific roles and interrelationships of different neural regions in information processing.

Further, the differing results of microstate clustering across frequency bands for AV information imply that neural oscillations in different frequency bands contribute to the processing of AV information, but the mechanisms vary across bands. By further analyzing the properties of microstate sequences in various frequency bands, we can obtain time-domain and frequency-domain features that characterize brain activity during AV information processing under both attended and unattended conditions.

As shown in Figures 4, 5, we used the KL_GEV evaluation method to cluster the microstates of AV information processing under attended conditions into six categories and those under unattended conditions into four categories. To facilitate a comparative analysis of the microstate properties under both conditions, we relabeled these microstates based on the similarity of their topographic distributions, as illustrated in Figure 6A. Many classic studies (Huang et al., 2018; Xi et al., 2020a,b; Matusz and Eimer, 2013; Brunellière et al., 2013) divide the AV information processing stages into early and late phases based on the timing of ERP presentations. However, this division lacks clear temporal boundaries and does not consider whether the scalp electric field distributions are consistent within the same phase. The scalp electric field distribution reflects the neural activity state of the brain during information processing and is closely related to cognitive processes. Therefore, we have proposed that processing stages with identical or similar scalp electric field distributions represent identical or similar cognitive processes.

In this study, based on the clustering results of AV microstates, we used the time series of the same microstate category to represent a sub-stage of AV information processing. Consequently, the AV information processing under attended conditions was divided into six and under unattended conditions into four sub-stages, as shown in Figure 6B.

Our findings demonstrate that the sub-stages of AV information processing, as delineated by microstate segmentation, do not follow a fixed sequential order but rather operate through dynamic alternation and collaboration to accomplish information processing. This suggests that the processing of external information by the brain involves complex mechanisms that likely encompass multiple cognitive and computational processes. We hypothesized that these sub-stages which are defined by microstates reflect the interactive dynamics of various processing and cognitive mechanisms. Notably, attended AV processing was segmented into six sub-stages (MS1–MS6), whereas unattended processing yielded four sub-stages (MS1–MS4). This marked difference indicates that the allocation of attentional resources significantly enhances processing complexity, potentially reflecting top-down regulatory mechanisms that facilitate refined integration of multimodal information (e.g., conflict resolution, task switching). The increased number of alternating sub-stages may correspond to a more sophisticated dynamic reorganization of cognitive functions. Even in the absence of attentional engagement, the brain maintains a basic processing of AV information (represented by four microstate clusters), albeit through a simpler mechanism characterized by fewer processing sub-stages. This likely reflects an automatic or passive processing mode that lacks the depth of integration and refinement afforded by attention-guided mechanisms. This reduced sub-stage complexity suggests fundamental differences in neural resource allocation and computational demands between the attended and unattended processing states.

It can be observed that MS1 and MS2 resemble the classical microstate D. Previous research suggests this microstate (particularly associated with the right temporoparietal junction, inferior parietal lobule, and the dorsal attention network) primarily orchestrates attentional resource allocation (Khanna et al., 2015). During audiovisual processing, it may participate in integrating visual and auditory information. MS3 and MS4 correspond approximately to microstate C. This microstate (typically linked to core regions of the default mode network, such as the posterior cingulate cortex/precuneus and medial prefrontal cortex) is generally associated with self-referential thinking (e.g., autobiographical memory, introspection) during rest (Brodbeck et al., 2012). During audiovisual processing, it may mediate the integration of emotion and perception. MS5 and MS6 are similar to microstate B. Previous studies indicate that microstate B (primarily involving the ventral attention network, including the temporoparietal junction, inferior frontal gyrus, and dorsolateral prefrontal cortex) is mainly associated with visuospatial information processing, attentional shifting, and the monitoring of exogenous stimuli (Michel and Koenig, 2018).