The data set used in this work was provided by Chen et al. (2023). The acquired data underwent text data processing using the Statistical Package for Social Sciences software (SPSS ver. 22.0, http://www01.ibm.com/software/analytics/spss/products/statistics/). During the data processing process, unknown and null values in MRI and EEG were estimated and filled using the weighted nearest neighbor algorithm Troyanskaya et al. (2001). Subsequently, min-max normalization was applied to all data within the range [0,1].

Given that both modes contain hidden features in addition to observed features, utilizing too many features could lead to significant overfitting problems in the model. Therefore, this work uses two feature selection methods to address this concern. MRMR Zhao et al. (2019) feature selection is used for dimensionality reduction of the MRI and EEG datasets. By examining the score values of different subsets of the data set, the highest and most optimal feature set is selected. To streamline the process, a grading strategy of 10 is used for feature selection. The MRMR algorithm is run for each grading label, and N optimal features are chosen within the range of 10 500 through evaluating the score value. Following the selection of the N best features, a feature importance algorithm is employed to verify and further optimize these selected features.

In this work, the sCNN-EEG is designed to extract the important and hidden feature extraction from AD EEG data. In Figure 1, the input undergoes convolution two kernels of size of 2, resulting in the generation of matrix X 1 . X 1 is then processed through two different branches, where stacked convolutions with kernel sizes of 3 and 4 are applied, producing X 2 and X 3 , respectively. Next, matrix multiplication is performed between X 1 and X 2 , yielding a matrix graph S 1 that contains both important and hidden features. A similar approach is used to combine X 1 and X 3 , resulting in the formation of S 2 . The matrix graph S 1 can be calculated by Equation 1 S 1 = X 1 × X 2 . (1)

The convolution kernels of the convolutional layer are all initialized with constants. A stride value of 1 is used to move the kernel window and perform the convolution across the entire input matrix. The sCNN-EEG maintains the size of the convolutional feature map, akin to the feature map of the previous convolution, and preserves the shape of the input data by setting the padding variables to be the same. The same padding variable can ensure that there will be no matrix problems during subsequent fusion. Since the second half of the pooling layer has the same structure, take the part of the model where S 1 is an example. The feature map S 1 is taken as two inputs, which are processed by the max pooling layer and the average pooling layer, respectively. After processing, two feature maps ( M 1 and M 2 ) are obtained. Both M 1 and M 2 subjected to an identical convolution process, employing a kernel size of 2, resulting in the creation of two additional feature maps, M 3 and M 4 . Then, a matrix multiplication operation is performed on M 3 and M 4 to yield the composite feature map M 5 . By using pooling operations such as max pooling and average pooling, important spatial information from the input feature maps is preserved while reducing dimensionality. These pooled feature maps, M 1 and M 2 , are then processed using convolution, which helps in extracting meaningful features. Finally, matrix multiplication is applied to combine these features, capturing the relationships between the different pooled representations and creating a composite feature map. This process is formally represented by Equation 2, which describes the mathematical operation involved in computing M 1 . M 1 = A V G S 1 × M A X S 1 , (2) where AVG here is the average pooling layer, and MAX is the maximum pooling layer.

The two deep feature maps are matrix multiplied and concatenated with the original features to form the final feature map. The purpose of this is to ensure the integrity of feature information and to dig out deep features. Finally, the final feature maps are passed to the max pooling and connection layers. At the same time, the stacked feature maps are flattened using a flattening layer before the connection layer. This allows the feature maps to be transformed into a one-dimensional representation. The connection layer consists of 150, 100, and 50 units, which the flattened feature maps are connected to. Additionally, there is a hidden layer with a 50% dropout rate, which helps prevent overfitting by randomly dropping out half of the units during training. To ensure that the features are non-linear, the connection layer uses the hyperbolic tangent function as the activation function and initializes its weights with the Glorot normal initializer Glorot and Bengio (2010). Therefore, the cost function can be obtained by L x t , y t = − 1 N × ∑ i = 0 N x t i × log y t − 1 − x t i × log 1 − x t + 1 2 × ∂ × ∑ k = 0 K W k 2 , (3) where L is cost function is the combination of the binary cross-entropy and L2 regularization term. ∂ is a hyper-parameter which represents the regularization coefficient. x t is true class label. y t is predicted class label. N is batch size. W k is the k − t h weight parameter of the model. K is the number of weight matrices.

In this work, the sCNN-sMRI is focused on important feature extraction and hidden feature extraction for sMRI. In Figure 2, the sMRI data is taken as input, first passing through two identical convolution processes, using a convolution kernel of size 2. Then it goes through the feature extraction modules of two different convolution kernels. One of the paths consists of a set of convolution kernels 3 and 4. It aims to create two different feature maps (represented as a 1 and a 2 respectively). The other path consists of convolution kernels 4 and 2, which focuses on obtaining two different feature maps (represented as a 3 and a 4 respectively). Multiple feature maps containing different feature information are created and fused through different feature extractors. Now all feature map features of the same feature extractor are fused together to form new feature maps Q 1 and Q 2 . The Q 1 and Q 2 can be obtained by Equations 4, 5 Q 1 = a 1 × a 2 , (4) Q 2 = a 3 × a 4 . (5)

In this model, multiple feature maps obtained from each branch are fused through matrix multiplication. Multiple feature maps Q 1 and Q 2 are concatenated to obtain the feature map A 1 . After cascading, A 1 is calculated by the maximum pooling and the average pooling respectively, and obtains a multi-pooling feature matrix map. Multi-pooling features are convolved to obtain deep feature maps (each convolution kernel size of 2). The final feature map A 2 is obtained by matrix multiplication of the maximum feature map and the average feature map. The A 1 and A 2 can be obtained by Equations 6, 7 A 1 = Q 1 © Q 2 , (6) A 2 = A V G A 1 ∗ W × M A × A 1 ∗ W , (7) where © is the concatenation symbol, AVG is the average pooling layer, and MAX is the maximum pooling layer, W is the convolution kernel size. Finally, the A 2 is calculated by the max pooling and fully connection layers. After the maximum pooling layer, the structure of the connection layer is consistent with the connection of sCNN-EEG. Both consist of 200, 150, 50% and 50% dropout. The training cost function for this model is the same as Equation 3.

During the experiment, it is found that in the multi-branch feature extraction process of MCNNRF, the feature extraction process of the two modalities is independent of each other, and the lack of relevant cooperation may cause the extracted features to be independent of each other. At the same time, it may cause poor representation ability after multimodal fusion. To this end, this work proposes a fusion module of MJA, which is mainly used to explore the deep cooperation of two modalities to enhance the representation ability of extracted features. The structure of the MJA fusion module is shown in Figure 3. The design of the module is inspired by the spatial attention mechanism in Fu et al. (2019). Specifically, the module takes the EEG data branching model (denoted as A) and the sMRI branching model (denoted as B) as input sources. To simplify the description, only the spatial attention unit of branch A is explained in detail in this paper. In the module, the structure of A and B is similar, and each branch consists of three convolutional layers and an S-shaped activation function; the convolutional layers are used to extract the features of each model, and the S-shaped activation function is used for nonlinear transformation. Given the input N ∈ R ( G × H × W ) , where G , H , and W denote the number of channels, height, and width of the features, respectively, firstly, A query , A key , and A value are generated by three 1 × 1 convolution operations, respectively, and the dimensionality of these outputs are all R ( G / 8 × H × W ) . In order to reduce the computational cost, the 1 × 1 convolution reduces the number of channels to 1/8 of the original, thus reducing the computational effort. Next, the attention score matrix is obtained by multiplying the transpose of A query with A key . Then, the sigmoid activation function is used to generate the spatial attention graph S a ∈ R ( N × N ) , which reflects the spatial importance of the input features. Next, for branch B, the same operation is performed to obtain the corresponding spatial attention map S b ∈ R ( N × N ) , which is used to characterize the spatial feature importance of branch B. Unlike the use of softmax to generate S a in the original spatial attention mechanism, this paper employs an S-shaped activation function, which is designed to capture hidden features over a wider range and to be consistent with the activation function in multimodal models. In addition, in the traditional spatial attention mechanism, S a is only used to refine A. In this design, S b is not only used to refine B, but also achieves a deep fusion of the two modes by combining it with S a , which enables the two modes to work more closely together, thus improving the overall feature representation capability. Therefore, the MJA fusion module is able to better coordinate the feature learning of the two modalities by introducing a joint attention mechanism, which not only weights the features of the respective modalities at the spatial level, but also interactively fuses between the two modalities, thus improving the model’s ability to comprehend and process multimodal data. This design allows the model to extract and fuse information more effectively, enhancing the robustness and accuracy of the final representation.

Specifically, first perform three 1 × 1 convolutions to generate A query , A key and A value respectively, and make their dimensions controlled at R ( G × N ) Then A value , B value , and the corresponding generated S a and S b are matrix multiplied to obtain two attention feature maps C a and C b . The specific formulas of C a and C b can be calculated by Equations 8, 9 C a = A value × S b , (8) C b = B value × S a . (9)

In the formula, C a ϵ R ( G × N ) is the stacked EEG feature guided by the sMRI feature, and C b ϵ R ( G × N ) is the stacked sMRI feature guided by the EEG feature. Finally, this work reshapes C a and C b into R ( G × H × W ) and performs feature concatenation on C a with A, and C b with B to obtain the final stacked features. It can be calculated by Equations 10, 11 A refined = N © C a , (10) B refined = M © C b , (11) where A refined ϵ R ( G × H × W ) and B refined ϵ R ( G × H × W ) represents the final EEG features and sMRI features, while N and M represent the initial input EEG features and sMRI features, respectively.

The final framework of this work is a combination of sCNN-EEG, sCNN-sMRI models, and fused modality models. The sCNN-EEG and sCNN-sMRI models are responsible for extracting features from the corresponding modalities. The MAJ module is used to solve the interconnection and matching between multimodal features and to fuse multimodal features. MCNNRF is shown in Figure 4. It is divided into three phases. The first stage is to extract features from the corresponding modalities using single modality models (i.e. sCNN-EEG and sCNN-sMRI). The second phase aims to address the lack of interconnectivity and fusion in the multimodal information extraction process. The features extracted from the two single models are used as the input source of the MAJ module. The third stage is the stacked features formed after fusion and the stacked RF is used for classification.

This work is implemented by using TensorFlow library on an NVIDIA RTX A6000 GPU. The dataset used in this work is provided by the research of Colloby et al. (2016). The dataset contains electroencephalogram (EEG) data from 99 Alzheimer’s disease (AD) patients. However, due to the lack of data in 5 cases, magnetic resonance imaging (MRI) scan images are only available for 89 patients. Among the available cases, there are 45 females with an average age of 75.8 ± 7.3 years. The EEG data of CNHCs (Healthy ControlsCognitively Normal) used is from a public static EEG dataset for epilepsy, and the MRI data is from the public ADNI dataset. Despite the data set being small, the model is relatively intricate, and this often culminates in overfitting of the model. Therefore, the experiments in this work use 10-fold cross-validation to deal with these problems. At the same time, the data set will be divided into 8:2 corresponding to the training set and the test set, where the training set is used for training the model, and the test set is used for testing and evaluating the model. The receiver operating characteristic curve (ROC) Tharwat (2018) is used as the main metric for hyperparameter tuning and finding the best model. This work also evaluated some secondary indicators such as sensitivity (Sn), specificity (Sp), accuracy (Acc), precision (Pre), and Matthew correlation coefficient (Mcc). These indicators can be calculated by Equations 12–16 S n = t p / t p + f n , (12) S p = t n / t n + f p , (13) P r e = t p / t p + f p , (14) A c c = t p + t n / t p + t n + f p + f n , (15) M c c = t p × t n − f p × f n t p + f n × t p + f p × t n + f n × t n + f p , (16) where t p is true positive, t n is true negative, f p is false positive and f n represents false negative values. They are calculated from the confusion matrix of the predicted results.

This work focuses on the feature extraction of EEG and sMRI datausing sCNN-EEG and sCNN-sMRI models, respectively. Prior to determining the sCNN-EEG and sCNN-sMRI models, this work designed some feature extraction model strategies for two modalities, named CNN-EEG and CNN-sMRI respectively. Compared with sCNN-EEG and sCNN-sMRI, CNN-EEG and CNN-sMRI only lacks different multiple pooling layer modules. In this section, the performance of different networks (CNN-EEG, sCNN-EEG, CNN-sMRI, and sCNN-sMRI) is compared. The Receiver Operating Characteristic (ROC) is shown in Figure 5, and the Area Under the Curve (AUC) is calculated. The AUC of sCNN-sMRI is 0.33% higher than that of CNN-sMRI, while the AUC of sCNN-EEG is 2.63% higher than that of CNN-EEG. Other performance indicators are shown in Table 1. Compared with CNN-sMRI, the precision sCNN-sMRI improves to 75.97%, an Mcc improves to 51.35%, and a relatively flat sensitivity. Compared with CNN-EEG, sCNN-EEG has a relatively flat accuracy and a sensitivity improvement of 18.76%. Compared with CNN-EEG/CNN-sMRI, sCNN-EEG, and sCNN-sMRI not only have more multi-branch different convolutional kernels for feature extraction but also strengthen the learning of weak features and use multi-pooling modules to extract deep-level features. At the same time, using stacked connections in the connection allows the extracted features to perform stacked features, which can better integrate hidden features into it.

After analyzing the various performance indicators of the centralized single-mode feature extractor, a model for the feature extractor is selected. The feature extractor is used to extract multimodal features, which are then stacked together. The multimodal features are fused and classified using different strategies. Initially, a simple concatenation matrix method is used to fuse the multimodal features, and the performance of the model is evaluated. However, it is observed that simple splicing methods does not directly improve and enhance the performance and classification ability of multimodal models. For this purpose, this work has designed several strategies for multimodal fusion. For example, using a bimodal attention mechanism for fusion. The multimodal model of EEG and sMRI can solve the correlation and cooperation between the modalities and can also classify AD well. Table 2 shows the performance indicators of each strategy, and their ROC curves are shown in Figure 6.

Figure 6 shows the performance curve of different multimodality models. Even with RF as the classifier, MCNNcRF accuracy is only 62.93%. MCNNBA uses dual attention fusion for multiple modalities, with an accuracy of 83.73%. Compared with MCNNcRF, the accuracy of MCNNBA is much higher than that of MCNNcRF. The main reason is that MCNNBA’s dual attention fusion module is more focused on the connections between multimodals. When MCNN uses Bi-Attention and adds RF for classification, the accuracy rate is 84.43%, because RF enhances its ability to classify stacked features. Finally, when MCNNBA adds RF, the accuracy rate reaches 94.75%. Compared with the previous strategies, this model uses the fusion module to cooperate and deeply fuse the features after extracting the two modal features. Table 2 shows the MCNNRF strategy outperformed all other strategy models in this experiment, demonstrating superior performance in terms of accuracy, precision, sensitivity and Mcc values. When compared with MCNNcRF, the accuracy of MCNNRF is elevated by 31.82%, precision is amplified by 11.44%, and sensitivity is increased by 39.21%. As shown in Table 2 and Figure 6, among different multimodal fusion strategies, MCNNRF with the MJA module shows the best performance in accuracy, precision, sensitivity and Mcc values. Although both MCNNBA and MCNNBARF use bimodal attention mechanisms for fusion, MCNNBARF outperforms MCNNBA in all performance metrics due to its utilization of RF to enhance classification capabilities. Nevertheless, MCNNBARF still falls short of MCNNRF’s performance, as the bimodal attention mechanism only enhances feature extraction capabilities without exploring and amplifying the intercommunication and complementarity between modalities.

EEG modalities and sMRI modalities are fed into the model by the multimodal model as output sources. Compared to the unimodal model, the multimodal model diversifies the input. At the same time, the multimodal model fuses the characteristic enhancement features between the different modalities, and obtains a higher accuracy. Their various performance indicators are shown in Table 3. It is obvious that the multimodal model is the optimal model, and the accuracy is increased by 8.91% and 16.32% compared with the sCNN-sMRI and the sCNN-EEGl, respectively. At the same time, other parameters have been greatly improved. Overall in this experiment, MCNNRF is far superior to the unimodal model.

To analyze the robustness of the proposed models, 10 independent experiments are performed on each model. The experimental results, measured in terms of accuracy, are presented in Figure 7 The MCNNRF model exhibits obvious advantages, outperforming both unimodal models (sCNN-EEG and sCNN-sMRI) consistently. Even the worst-performing MCNN model surpasses the best-performing sCNN-EEG and sCNN-sMRI models by 7.07% and 2.02%, respectively. Additionally, the curves of all three models demonstrate stability and consistency throughout the experiments.

This work is the first work to explore the application of deep learning in combination with EEG multimodality for AD diagnosis. Therefore, this work compares with most advanced multimodal methods. The performance comparison between the different models is shown in Table 5. Compared to CNN + ANN model, MCNNRF shows relatively lower accuracy and sensitivity. The reason is that CNN + ANN has conducted deep mining of clinical and biological information. Firstly, CNN is used to extract features from images, and then a fusion module is designed using ANN to fuse and classify features. MCNNRF model lacks the supplementation of auxiliary information like clinical data and does not utilize feature transformation techniques to reduce feature dimensionality differences. Compared to HMGD, MCNNRF shows relatively lower accuracy and sensitivity. The specific reason is that HMGD employs graph diffusion methods to enhance the representation capability of multimodal data, thereby strengthening the measurement of multimodal similarity. However, MCNNRF is more focused on cross-modal collaboration and correlation. In the comparison on Accuracy, the performance of MCNNRF is on par with OLFG. The difference between MCNNRF and OLFG lies in one utilizing a multimodal combination of EEG and MRI, while the other employs MRI and PET. OLFG focuses more on the variations of various information in brain images. MCNNRF considers changes in brain image information, while also focusing on information differences that occur over time. In summary, this work explores the application of multimodal EEG in AD. Compared to MCNNRF, the accuracy of 3D-CNN-BRNN increases 1.25% and the sensitivity value increases 11.12%. The main reason is that the 3D-CNN-BRNN dataset owns a clear time series, with mobile MRI data spanning 6 months. At the same time, bidirectional recurrent neural networks are used to recognize the time series. 3DCNN is used to extract MRI features, and then AD is classified by auxiliary information. However, MCNNRF has the different time span as this method for controlling datasets, and there is also no corresponding time series for recognition. Compared to MCNNRF, MCAD has a 0.68% lower accuracy, and MCAD uses MRI and PET as well as some auxiliary modal information. MCAD uses a cross attention mechanism to fuse modalities, while MCNNRF performs deep feature mining on modalities and finally performs fusion.