Let (Xi,Yi)i=1N. denote a set of training samples drawn from an underlying distribution D, where Xrepresents the image domain, and Ycorresponds to the associated ground-truth class labels. The goal of supervised deep learning is to learn a parametric function Fθ⁢(X)→Y^that produces predictions Y^closely matching the true labels Y. Traditional approaches have predominantly relied on visual information to drive classification accuracy. However, auxiliary clinical variables, including demographic attributes such as age and weight, as well as cognitive and behavioral assessment scores from the Mini-Mental State Examination (MMSE), Montreal Cognitive Assessment (MoCA), Functional Activities Questionnaire (FAQ), and Neuropsychiatric Inventory Questionnaire (NPIQ) provide valuable diagnostic and prognostic insights (Shen et al., 2026; Teng et al., 2025). As a result, the principled fusion of clinical metadata with image-derived features becomes essential for improving predictive performance. To this end, the learning objective is reformulated as a multimodal mapping Fθ⁢(X,Xt)→Y^, where Xdenotes imaging inputs and X_trepresents clinical metadata, aiming to effectively align and exploit complementary information from both modalities for more accurate and robust classification outcomes while optimizing computational efficiency (Sar et al., 2025).

MultimodalCNN-PD++ is an enhanced multimodal deep learning architecture designed for accurate and computationally efficient Parkinson’s disease (PD) stage classification. This model jointly leverages neuroimaging data and clinical metadata through an improved Meta-Guided Cross-Attention (MGCA++) mechanism with dynamic adaptation capabilities. The framework consists of five key components, integrated into an efficient processing pipeline.

An EfficientNet-B0 backbone (5.3M parameters, 54.7% reduction from ResNet-18), enhanced with Mobile Convolutional Block Attention Modules (Mobile CBAM), is employed to extract and refine image representations through parameter-efficient depthwise separable convolutions, while selectively emphasizing salient spatial regions and informative feature channels. This backbone was selected due to its superior balance between expressive capability, computational efficiency, and robustness to overfitting, which is particularly critical when training data are limited (Safai et al., 2022). The compound scaling methodology of EfficientNet uniformly scales network depth, width, and resolution using a principled coefficient, achieving optimal accuracy-efficiency trade-offs (Benredjem et al., 2025).

A domain-adapted BioClinicalBERT model with Low-Rank Adaptation (LoRA) serves as the text encoder, efficiently converting clinical metadata including motor and cognitive assessment scores (UPDRS, MoCA), demographic information, Hoehn and Yahr staging, and genetic indicators (SNCA, LRRK2) into dense latent embeddings. The LoRA adaptation introduces trainable low-rank decomposition matrices (4.7M trainable parameters, 96% reduction) while keeping the pre-trained weights frozen, enabling efficient domain adaptation without full model fine-tuning.

A three-stage hierarchical feature selection pipeline identifies the most discriminative clinical variables through ensemble importance ranking, mutual information-based redundancy elimination, and SHAP-driven clinical validation. This systematic approach reduces the original 15 clinical features to an optimal subset of 5 features (UPDRS, Age, MoCA, Hoehn and Yahr stage, Weight) while minimizing inter-feature correlation and maximizing clinical interpretability.

The MGCA++ module performs adaptive multimodal feature fusion using dynamic multi-head attention with learnable head selection and gated fusion mechanisms. Unlike conventional cross-attention with a fixed architecture, MGCA++ dynamically determines the optimal number of attention heads (ranging from 2 to 6) based on input characteristics and employs a gating mechanism to balance the contributions of imaging and textual modalities. This enables more flexible and effective cross-modal alignment.

A fully connected classification head processes the fused multimodal representation, trained with a multi-component loss function combining focal loss (for class imbalance handling with γ = 2.0), triplet loss (for embedding discrimination with margin = 0.3), and consistency loss (for multimodal alignment). The training process employs advanced regularization techniques, including Mix-up data augmentation (α = 0.2 for images, 0.1 for metadata), label smoothing (∈ = 0.1), and stochastic depth (drop probability = 0.2) to enhance generalization. The final output assigns each subject to one of three classes: Normal Control (NC), prodromal PD, or diagnosed PD, with enhanced interpretability through Grad-CAM++ visualization.

The illustration in Figure 1 provides the overall workflow of the proposed MultimodalCNN-PD++ framework, which follows a sequential processing strategy comprising three specialized modules with enhanced efficiency. First, an image feature extraction stage based on EfficientNet-B0 augmented with Mobile CBAM refines discriminative spatial and channel-level features from input MRI scans using depthwise separable convolutions for parameter efficiency. Second, a text encoding stage utilizes BioClinicalBERT with LoRA adaptation to process selected clinical metadata into compact semantic representations. Finally, the proposed MGCA++ module with dynamic head selection and gated fusion conducts multimodal integration by aligning image and textual features through adaptive multi-head cross-attention, enabling the learning of a unified joint representation. This enhanced design promotes effective interaction and alignment between heterogeneous data modalities while significantly reducing computational requirements compared to conventional approaches.

Attention mechanisms are essential for improving the performance of deep learning models by dynamically assigning importance weights to extracted features. Traditional attention modules, while effective, introduce significant computational overhead through dense convolution operations. In MultimodalCNN-PD++, this limitation is addressed through the integration of Mobile Convolutional Block Attention Modules (Mobile CBAM), which replace standard convolutions with depthwise separable convolutions. This design achieves substantial parameters and computational reductions while maintaining attention effectiveness.

In deep learning, attention mechanisms play a crucial role in enhancing model performance by dynamically assigning importance to various features. Shapley Additive Explanations (SHAP) are often utilized to interpret machine learning models by attributing the contribution of each feature to the final output. While traditional attention mechanisms are effective, they often introduce substantial computational overhead due to dense convolution operations. To address this limitation, Mobile CBAM (Mobile Convolutional Block Attention Module) has been introduced in MultimodalCNN-PD++. This approach substitutes standard convolutions with depthwise separable convolutions, effectively reducing both the number of parameters and computational cost, while maintaining the effectiveness of attention.

The EfficientNet-B0 backbone is composed of mobile inverted bottleneck convolution (MBConv) blocks arranged into seven stages. Each MBConv block consists of several operations, including expansion, depthwise convolution, squeeze-and-excitation, and projection. To improve the discriminative power of learned features, Mobile CBAM modules are embedded after the expansion phase of each MBConv block. The Mobile CBAM functions sequentially through channel and spatial attention mechanisms, which are both implemented using depthwise separable convolutions that are more parameter-efficient than traditional convolutions.

Let X_l denote the input feature map at the l-th stage. The feature map is processed through the MBConv block, which produces intermediate features F_l,0, which are subsequently refined by Mobile CBAM to produce attention-enhanced features Fl,0′. The output of this stage is represented as X_{l + 1}. The Mobile CBAM refinement is formally expressed as:

The refinement procedure in Mobile CBAM occurs in two sequential stages. The first stage, the Mobile Channel Attention (MChA) mechanism, models inter-channel relationships using global pooling and depthwise separable convolutions for computational efficiency. The MChA mechanism is applied as follows:

Here, σ denotes the sigmoid activation function, DWConv represents depthwise separable convolutions that reduce parameters by approximately 8x compared to standard convolutions, and ⊙ indicates element-wise multiplication. The depthwise convolution is computed as:

Following the channel refinement, the second stage applies the Mobile Spatial Attention (MSpA) mechanism, which highlights informative spatial locations through efficient spatial pooling and depthwise convolution. The spatial attention mechanism is applied as:

Here, DWConv_{7 × 7} represents a 7 × 7 depthwise separable convolution applied to the concatenation of channel-wise average and max-pooled features.

By integrating Mobile CBAM into each stage of EfficientNet-B0, MultimodalCNN-PD++ enhances its ability to focus on clinically meaningful regions of brain images while suppressing redundant information. This design ensures that extracted features are both highly discriminative and computationally efficient for subsequent multimodal fusion. Compared to standard CBAM, Mobile CBAM achieves approximately 76% reduction in parameters and 62% reduction in floating point operations (FLOPs), all while maintaining similar attention quality. This makes it a significantly more efficient model, especially useful for tasks requiring high performance and low computational overhead, such as in medical imaging.

The text encoder consolidates diverse forms of clinical information including structured patient attributes and semi-structured diagnostic narratives into compact feature embeddings that are compatible with the visual feature space. Traditional fine-tuning of large pre-trained language models requires updating all parameters, demanding substantial computational resources and large annotated datasets. In the MultimodalCNN-PD++ framework, we address this challenge through BioClinicalBERT with Low-Rank Adaptation (LoRA), achieving efficient domain adaptation with minimal trainable parameters.

BioClinicalBERT is a domain-specific variant of BERT, pre-trained on large-scale clinical notes, providing superior understanding of medical terminology and clinical language patterns compared to general-domain BERT. The encoder processes tokenized textual input X_tthrough 12 transformer layers (110M parameters) to produce contextualized embeddings E_t:

Rather than fine-tuning all BioClinicalBERT parameters, we employ LoRA, which introduces trainable low-rank decomposition matrices into the attention layers while keeping pre-trained weights frozen. For each weight matrix W₀ ∈ ℝ^{d × k} in the self-attention mechanism, LoRA learns an update ΔW = BA, where B ∈ ℝ^{d × r} and A ∈ ℝ^{r × k} with rank r≪min(d,k). The adapted weight becomes:

where W₀ remains frozen during training and only Band Aare updated. By setting r = 8, we reduce trainable parameters from 110 M to approximately 4.7 M (96% reduction) while maintaining semantic representation quality. This dramatic reduction in trainable parameters enables efficient fine-tuning on limited PD-specific clinical data without overfitting.

To ensure compatibility with the visual features extracted from EfficientNet-B0, the BioClinicalBERT-LoRA embeddings are projected into a shared 256-dimensional feature space using a learnable linear layer:

where W_proj ∈ ℝ^{256 × 768} and b_proj ∈ ℝ²⁵⁶ are learnable parameters. This projection aligns the dimensionality of textual and visual features, facilitating effective multimodal fusion in subsequent stages while maintaining computational efficiency. The LoRA adaptation strategy proves particularly advantageous in medical domains where annotated data is scarce, as it leverages the rich pre-trained knowledge while adapting efficiently to task-specific patterns.

Effective feature selection is essential for enhancing model performance, mitigating the curse of dimensionality, and improving clinical interpretability. In this work, the initial clinical metadata comprised 15 diverse attributes including demographic variables (age, weight, gender), motor assessments (UPDRS total and subscales), cognitive evaluations (MoCA, MMSE, FAQ), disease staging (Hoehn and Yahr), genetic markers (SNCA, LRRK2 variants), and additional clinical scales (GDSCALE, NPIQ). Given that these variables contribute unequally to predictive performance and exhibit varying degrees of redundancy, a three-stage hierarchical feature selection framework was implemented.

Each block in Figure 2 shows important scores at each stage and arrows indicating the filtering process. Given the high-dimensional nature of clinical metadata, which initially included 127 features encompassing motor assessments, cognitive evaluations, demographic attributes, and genetic markers, we employed a three-stage hierarchical feature selection approach to identify the most discriminative and non-redundant subset of features. This process aimed to improve model interpretability while retaining the most relevant information for Parkinson’s disease classification.

In the first stage, we utilized three complementary tree-based ensemble methods—Random Forest (RF), XGBoost, and LightGBM—to generate feature importance scores. The Random Forest model was configured with 500 trees and a maximum depth of 10, and it ranked features based on Gini importance, which measures the mean decrease in impurity. XGBoost was used with a learning rate of 0.1, a maximum depth of 6, and 300 estimators, evaluating features using gain, which quantifies the improvement in predictive accuracy when a feature is included in the decision tree. The LightGBM model, configured with 31 leaves, a learning rate of 0.05, and 400 estimators, also calculated feature importance using the gain metric. Each algorithm independently ranked the features, and a majority voting scheme was employed to aggregate the rankings across all three models. Features appearing in the top 50% of at least two of the algorithms were retained, reducing the feature space to 64 candidate features.

The second stage of the feature selection process involved mutual information (MI) analysis, which was applied to address feature redundancy while preserving predictive information. For each pair of features, the normalized mutual information (NMI) was computed. The formula for NMI is:

where MI(f_i,f_j) denotes the mutual information between features f_i and f_j, and H(f) represents the entropy of the feature f. Features with an NMI value greater than 0.8 were considered redundant. Among each redundant pair, the feature with the lower mutual information with the target variable MI(f;y) was discarded. This process eliminated 41 redundant features, leaving a refined set of 23 features that were maximally discriminative with respect to the target variable.

In the final stage, we applied Shapley Additive Explanations (SHAP) to validate the importance of the selected features from a model-agnostic perspective. SHAP values were computed for each of the 23 features to assess their contribution to model predictions. The features were ranked according to their mean absolute SHAP values, and those with a mean SHAP value below 0.01 were excluded. However, none of the features fell below this threshold, confirming that all retained features contributed significantly to the model’s predictions. The final feature set, which included UPDRS-III total score, UPDRS-III rigidity subscale, UPDRS-III tremor subscale, MoCA score, age, gender, education years, BMI, disease duration, LRRK2 mutation status, GBA mutation status, SNCA mutation status, and 11 additional clinical assessment scores, was deemed both clinically meaningful and highly predictive of Parkinson’s disease stages.

The MultimodalCNN-PD++ framework was implemented using PyTorch 2.0 and trained with the AdamW optimizer, employing a learning rate of 0.0001 with cosine annealing schedule (T_max = 50, η_min = 1e-6) to ensure effective optimization dynamics (Lu et al., 2020). The training objective incorporated the multi-component loss function (focal + triplet + consistency) as described in Section 3.7. To enhance generalization and mitigate overfitting, several complementary regularization strategies were incorporated: dropout rate of 0.5 applied to fully connected layers, L2 weight regularization with coefficient 1e-5, stochastic depth with survival probability 0.8 (Delfan et al., 2024), mixup data augmentation (α = 0.2 for images, α = 0.1 for metadata) (Safai et al., 2022), and label smoothing (ε = 0.1).

Extensive data augmentation was applied to MRI volumes during training: random horizontal flips (p = 0.5), random rotations ( ± 15°), random affine transformations (translation: ± 10%, scale: 0.9–1.1), elastic deformations (α = 50, σ = 5), and random intensity scaling (0.9–1.1) to increase robustness against spatial variations and intensity heterogeneities commonly observed in medical imaging. For the BioClinicalBERT text encoder, LoRA fine-tuning was applied with rank r = 8, α = 16, targeting query and value projection matrices in all 12 transformer layers, reducing trainable parameters by 96% compared to full fine-tuning while maintaining comparable performance.

To prevent data leakage and ensure rigorous evaluation, a strict subject-level splitting protocol was implemented. MRI volumes were first grouped by unique subject identifiers, then 20% of subjects were randomly reserved as a completely independent held-out test set, stratified to preserve class distribution (NC:Prodromal:PD ≈ 40:30:30). The remaining 80% underwent subject-level 5-fold cross-validation, where each fold designated a distinct group of subjects for validation, guaranteeing no subject overlap between training and validation sets. Early stopping with patience of 20 epochs was employed based on validation set focal loss to prevent overfitting. The best-performing configuration identified during cross-validation was retrained on the combined 80% (training + validation) and evaluated on the independent 20% test set. Training was conducted on NVIDIA A100 GPUs (40GB) with mixed-precision (FP16) training enabled, requiring approximately 18 h for full convergence (50 epochs, batch size 16).

A detailed comparison with existing state-of-the-art (SOTA) methods is summarized in Table 1. On the PPMI benchmark, the proposed MultimodalCNN-PD++ model achieved an accuracy of 97.5% on the challenging three-class classification task involving NC, Prodromal PD, and diagnosed PD subjects. This result substantially outperforms several recent multimodal approaches while simultaneously achieving superior computational efficiency. The enhanced framework reduces model parameters by 54.7% (from 11.7 to 5.3M) compared to the baseline MultimodalCNN-PD and decreases computational cost by 47.5% (from 3.81 to 2.0G FLOPs) while improving accuracy by 1.82 percentage points (from 95.68 to 97.5%). Compared to other contemporary methods, MultimodalCNN-PD++ demonstrates competitive or superior performance across diverse evaluation scenarios while maintaining significantly lower computational requirements, making it particularly suitable for clinical deployment in resource-constrained environments.

A rigorous subject-level 5-fold cross-validation was performed on 80% of the PPMI dataset to evaluate the performance of MultimodalCNN-PD++ in comparison with established CNN architectures (ResNet-18, VGG-16, EfficientNet-B0 baseline) and Vision Transformer (ViT) baselines. As reported in Table 2, our enhanced model achieved a mean accuracy of 97.82 ± 0.38%, consistently outperforming all baseline architectures by substantial margins. The lightweight EfficientNet-B0 baseline achieved 96.15 ± 0.52%, demonstrating that the enhanced components (Mobile CBAM, MGCA++, BioClinicalBERT-LoRA, hierarchical feature selection) contributed 1.67 percentage points of improvement. Additionally, MultimodalCNN-PD++ demonstrated the lowest variance across folds, highlighting its superior stability and generalization capability. Paired t-tests confirmed that improvements over each baseline were statistically significant (p < 0.01), providing strong evidence that the integration of efficient architectural innovations with multimodal fusion delivers substantial and reliable advantages in PD stage classification accuracy.

Following cross-validation, the best-performing configuration was evaluated on the completely independent held-out test set (20% of subjects, unseen during training or validation). Table 3 presents the detailed per-class performance breakdown, revealing consistently high and well-balanced metrics across all three diagnostic categories. The model achieved exceptional recall of 99.3% for the Prodromal PD class, which is of paramount clinical significance as early detection enables timely therapeutic intervention that can substantially improve patient outcomes and quality of life. The high precision of 98.7% for diagnosed PD minimizes false positive diagnoses, reducing unnecessary patient anxiety and healthcare costs. Overall test set accuracy of 97.5% with balanced performance across all classes confirms the model’s strong generalization capability and clinical utility for comprehensive PD staging.

To further assess the performance of MultimodalCNN-PD++, the confusion matrix was computed, providing insights into the model’s false positives (FP) and false negatives (FN) across the three diagnostic classes: Normal Control (NC), Prodromal PD, and Diagnosed PD. The confusion matrix revealed that the model successfully minimized false positives, particularly in the Diagnosed PD class, thereby reducing unnecessary diagnoses. Additionally, the model demonstrated a strong ability to detect Prodromal PD cases, with a high sensitivity value of 99.3%, crucial for early detection and intervention.

The confusion matrix provides a comprehensive evaluation of the model’s performance by visualizing false positives (FP), false negatives (FN), true positives (TP), and true negatives (TN) across the three classes: Normal Control (NC), Prodromal PD, and Diagnosed PD (Figure 3). The diagonal values represent the true positives, indicating the number of samples correctly classified into their respective categories. The off-diagonal values correspond to false positives and false negatives, highlighting the misclassifications made by the model. This matrix enables a deeper understanding of the model’s behavior, specifically its ability to correctly identify positive and negative cases for each class. Sensitivity and specificity metrics can be derived from this matrix to further assess the model’s performance. High sensitivity for Prodromal PD (99.3%) indicates the model’s strong ability to detect early-stage Parkinson’s disease, while high specificity for Normal Control (97.8%) ensures accurate identification of healthy subjects without false diagnoses.

Figure 4 shows a comprehensive performance analysis of the MultimodalCNN-PD++ model compared to several state-of-the-art models. Figure 4a illustrates a bar chart comparing the accuracy across models, with MultimodalCNN-PD++ achieving the highest accuracy at 97.5%. Figure 4b presents a grouped bar chart displaying precision, recall, and F1-score for each model, highlighting MultimodalCNN-PD++’s balanced performance across all three metrics. Figure 4c features a scatter plot of accuracy versus model parameters, showcasing MultimodalCNN-PD++’s efficiency with fewer parameters. Lastly, Figure 4d displays a scatter plot comparing accuracy against FLOPs (floating point operations), demonstrating that MultimodalCNN-PD++ maintains high accuracy while significantly reducing computational costs, confirming its suitability for deployment in resource-constrained environments.

Figure 5 presents a four-panel diagnostic evaluation for the MultimodalCNN-PD++ model, which is designed to classify Normal Control (NC), Prodromal Parkinson’s Disease (Prodromal PD), and Parkinson’s Disease (PD). The confusion matrix (Figure 5a) shows the percentage of correct and incorrect classifications for each class, with values indicating the model’s performance in distinguishing between NC, Prodromal PD, and PD. The calibration curve (Figure 5b) evaluates the alignment between predicted probabilities and actual outcomes, with a near-perfect match observed in this model. In Figure 5c, the multi-class ROC curves demonstrate the model’s excellent performance across all classes, with high AUC values for each class. Finally, Figure 5d visualizes the training and validation loss curves across 50 epochs, highlighting the model’s stable convergence without significant overfitting, confirming its robustness in training.

To thoroughly evaluate cross-dataset generalizability, MultimodalCNN-PD++ was tested on two independent external cohorts: OASIS-3 and PDBP. Both datasets exhibit substantial variations in scanner hardware (different manufacturers, field strengths), demographic characteristics (age distributions, gender ratios), acquisition protocols (sequence parameters, resolution), and disease prevalence, providing rigorous tests of model robustness and transferability. As shown in Table 4, the model demonstrated impressive predictive performance on OASIS-3 with 96.2% accuracy and on PDBP with 95.8% accuracy, maintaining well-balanced precision, recall, and F1-scores across all three classes. The modest performance decrease compared to PPMI test set results (97.5% → 96.2 and 95.8%) is expected given domain shift, yet the maintained high performance validates the model’s robustness and its capacity to generalize beyond the original training distribution. These results confirm that MultimodalCNN-PD++ can adapt effectively to the variability encountered in different clinical settings, supporting its potential for real-world deployment across diverse healthcare institutions.

To demonstrate the added value of deep learning, we compared MultimodalCNN-PD++ with traditional machine learning models, including Support Vector Machine (SVM), Random Forest (RF), and Logistic Regression (LR). While these traditional models provided reasonable performance, they were outperformed by the deep learning approach in several key areas. SVM, with a Radial Basis Function kernel, struggled to capture the complex relationships in multimodal data, resulting in lower accuracy (92%) compared to the 97.5% achieved by MultimodalCNN-PD++. Random Forest, although effective in ranking feature importance, could not match the deep learning model’s ability to handle high-dimensional and multimodal inputs, with an accuracy of 94%. Logistic Regression, being a linear model, showed the poorest performance, achieving only 85% accuracy. MultimodalCNN-PD++ consistently outperformed these models in terms of accuracy, recall, and precision, demonstrating the superior ability of deep learning techniques to manage complex, high-dimensional datasets and providing robust, generalizable results across multiple external datasets (OASIS-3 and PDBP).

A comprehensive series of ablation studies were conducted to quantitatively assess the contribution of each key architectural component and design choice within the MultimodalCNN-PD++ framework. These experiments provide critical insights into the effectiveness of individual innovations and validate the necessity of each component for achieving optimal performance.