Early and precise detection of cancer is fundamental to improving patient outcomes, as timely diagnosis directly influences therapeutic decision-making and long-term survival (Litjens et al., 2017). Diagnostic procedures rely heavily on radiological imaging—such as computed tomography (CT), magnetic resonance imaging (MRI), and mammography—complemented by structured clinical information, laboratory markers, and patient history (Esteva et al., 2017). Deep learning (DL) has been able to a large extent to replicate the tumor localization, segmentation, and classification processes which are the usual tasks of human experts, and in fact, DL has been recognized to perform at par with expert clinicians in most cases (He et al., 2016; Dosovitskiy et al., 2021). Nevertheless, human clinical decision-making involves different modes and, therefore, the integration of varied sources of information is a must. Consequently, multi-modal deep learning, which utilizes the complementary insights of different data types to identify disease features that single-modality models may not be able to, has been widely adopted (Nakach et al., 2024a).

Recent technological advancements in medical imaging, electronic health records (EHRs), and high-resolution data acquisition have exponentially increased the availability of different types of patient data, from radiological images to genomics markers and pathology slides (Li et al., 2024). Such a massive release of multi-modal data opens up the possibility to construct much stronger and more inclusive diagnostic models that are capable of detecting disease signatures even when they are very faint and could be overlooked if each modality is analyzed independently (Lai et al., 2024). Moreover, the application of clinical features—like age, biomarkers, comorbidity, and treatment history—in conjunction with imaging data has been proven to result in a substantial improvement in diagnostic accuracy as well as in the ability of cancer risk stratification in various cancer types (Yang et al., 2025; Chen et al., 2024).

Nevertheless, the medical AI community is increasingly realizing that accuracy is not enough. To be clinically viable, AI-powered diagnostic systems must also be transparent, interpret-able, and consistent with human experts’ reasoning patterns (Xie et al., 2025). Besides reliability, clinicians also require explanations of the predictions, e.g., which radiological regions contributed to the output, how clinical factors influenced the decision, and how the different modalities interact to form the final assessment (Oviedo et al., 2025). As a result, explainable artificial intelligence (XAI) has become a critical component in advancing trustworthy AI solutions for cancer diagnosis.

Deep learning has made a lot of progress in medical diagnosis. There are still some challenges that have not been resolved:

Advanced AI (Artificial Intelligence) systems need to be accurate, as well as transparent, interpret-able, and clinically relevant in order to be integrated into real clinical workflows. Medical professionals need understandable clarifications pointing out the radiological areas that affect the model predictions, the clinical variables that influence the diagnostic decisions, and how multi-modal evidence is combined. A unified, explainable multi-modal framework can therefore:

Although multi-modal DL and XAI have each advanced considerably, there is still no unified framework that:

Current studies typically emphasize diagnostic accuracy or explain-ability alone, but rarely address both holistically in a way that supports clinical use. This gap limits the practical adoption of multi-modal AI systems in oncology.

In response to the identified limitations, this study addresses the following research questions:

This study offers the following key contributions:

The research paper initially reviews the literature related to multi-modal deep learning, explainable artificial intelligence (XAI) techniques, and deep learning for cancer diagnosis. Afterward, it elaborates the proposed multi-modal architecture, feature-fusion method, and integrated interpret-ability framework in a very detailed manner. The next sections include descriptions of the datasets, pre-processing methods, experimental setup, and evaluation metrics. The results section focuses both on the diagnostic performance and interpret-ability findings. The discussion section summarizes main insights, reviews clinical implications, limitations, and future research directions. The paper ends with a summary of the contributions and the importance of the proposed framework for trust-worthy artificial intelligence in oncology.

Deep learning has largely revolution cancer imaging with the help of convolutional neural networks (CNNs), transformers, and hybrid architectures resulting in significantly improved lesions detection, segmentation, and malignancy classification in CT, MRI, PET, and histopathology imaging modalities (Litjens et al., 2017; Esteva et al., 2017; He et al., 2016; Dosovitskiy et al., 2021; Nakach et al., 2024b; Li et al., 2024; Lai et al., 2024; Yang et al., 2025). Very recent studies demonstrate the benefits of a large-scale self-supervised pretraining, federated models, and foundation architectures which allow generalization to different institutions and imaging devices (Chen et al., 2024; Xie et al., 2025; Oviedo et al., 2025; Ma et al., 2024). Multi-center testing of DL models also confirms their stability and reliability in real clinical scenarios, notably in breast, lung, and brain cancer diagnosis (Maigari et al., 2025a; Liu et al., 2025; Liu et al., 2025). These breakthroughs constitute a strong argument for the use of image-based neural representations as the core of integrated diagnostic frameworks.

Multi-modal learning combines imaging with structured clinical variables, genomic profiles, pathology images, and patient demographics to improve the accuracy of staging, prognosis, and subtype prediction (Buzdugan et al., 2025a; Kumar et al., 2025; Liang et al., 2025a; Turki et al., 2025; Bhosekar et al., 2025; Ramkumar et al., 2023; He et al., 2024; Liu et al., 2025). Fusion strategies, including early, late, and hybrid fusion, consistently show benefits over uni-modal systems, with hybrid attention-based mechanisms being most effective in capturing cross-modality interactions (Ennab et al., 2025a; Peng et al., 2025; Nakach et al., 2024a, 2024b; Shah et al., 2024). Research in breast, lung, and glioma oncology shows that multi-modal fusion results in better risk stratification, recurrence prediction, and treatment response modelling (Fayyaz et al., 2025; Singh et al., 2025; Ghasemi et al., 2024; Wei et al., 2025). Current literature also deals with real-world problems- such as incomplete modalities, data heterogeneity, and alignment issues-and suggests different designs to accommodate missing or noisy modalities (Oviedo et al., 2025; Gharaibeh et al., 2025; Kumar et al., 2023).

Explainable artificial intelligence (XAI) is increasingly being looked at as one of the indispensable elements for the medical AI to be trusted. XAI core methods involve gradient-based visualization (Grad-CAM and its variants), perturbation and occlusion analyses, game-theoretic feature attribution (SHAP), and local surrogate modelling (LIME) (Aftab et al., 2025; Singh et al., 2025; Tempel et al., 2025; Rabah et al., 2025; Maigari et al., 2025b; Buzdugan et al., 2025b). Comparative studies weigh the faithfulness of the explanation, resistance to noise, and clinical interpret-ability, thus different tasks and architectures being able to take advantage of customized explanation strategies (Song et al., 2025; Liang et al., 2025b; Ennab et al., 2025b). A lot of ground has been covered in getting the heatmaps to be more stable, lessening the artifact activation, and merging spatial and feature-level explanations for giving more reliable interpret-ability (Zhao et al., 2024; Patel et al., 2025; Zhou et al., 2024).

Multiple studies have been carried out in the area of multi-modal learning intersecting with explainability, which is phenomenal. The techniques comprise of unified attribution scoring across modalities, attention-based explanation layers embedded into fusion architectures, and cross-modal visualization techniques linking image regions with clinical variables (Thambawita et al., 2024; Nagar et al., 2025; Martins et al., 2024; Yoon et al., 2025). Experiments reveal that the provision of visual explanations along with the structured feature attributions makes the model more transparent and hence easier for the clinicians to follow their reasoning, which is more than what is achievable through uni-modal explanation pipelines (Ahmed et al., 2024; Fernandez et al., 2025; Ortega et al., 2024). The newly proposed architectures, as a matter of fact, have embedded the explanation mechanisms within the learning process thus leading to the behavior of a model that is more consistent with the explanations, rather than applying them post-hoc (Park et al., 2025; Rossi et al., 2024; Singh et al., 2025).

Firstly, quantitative metrics (faithfulness, localization accuracy, stability) need to be supported by human-centered studies evaluating clarity, usability, and clinical alignment for a thorough assessment of explainability (Wang et al., 2024; Rossi et al., 2024). Robustness analyses concern also the sensitivity to domain shifts, adversarial perturbations, and missing modalities, where mitigation strategies use uncertainty estimation, domain adaptation, and counterfactual reasoning (Xu et al., 2025; Patel et al., 2025; Thambawita et al., 2024). Several clinical trials disclose that explainable multi-modal systems become the source of diagnostic confidence and thus, the clinicians’ decision-making process is facilitated when such systems are used as decision-support tools in radiology and oncology workflows (Zhou et al., 2024; Martins et al., 2024; Yoon et al., 2025; Singh et al., 2025).

This review draws out three major revelations that are in direct alignment with the study’s questions of research: (i) multi-modal learning significantly improves diagnostic performance but needs strong fusion strategies (RQ1); (ii) XAI methods offer useful interpret-ability but have to be changed for multi-modal fusion architectures (RQ2); and (iii) integrated evaluation protocols that merge accuracy, interpret-ability, and clinician usability are indispensable for real clinical translation (RQ3). These insights, in aggregate, serve as a rationale for the creation of the explainable multi-modal deep learning framework proposed.

This study utilizes multiple publicly available cancer imaging datasets that provide radiological images and associated structured clinical metadata. All datasets are fully de-identified and released for research use. After preprocessing, quality control, and multimodal alignment, a total of 3,842 images from 2,917 patients were retained across all datasets for experimental evaluation.

Standardization of radiological images is done by voxel intensity normalization, N4 bias-field correction for MRI, and z-score scaling for CT attenuation values. Spatial harmonization is achieved by isotropic resampling to uniform voxel spacing, and then cropping or padding to fixed input dimensions. To keep morphological features and at the same time not over-fit, data augmentation is done through affine transformations, elastic deformation, and contrast perturbations. Clinical variables are given categorical encoding, outlier correction using inter-quartile filters, and min-max normalization. Combined multi-modal instances are created by matching patient identifiers across datasets. Those cases which do not have complete modality pairing are either removed or dealt with through auxiliary missing-modality embedding.

The image stream utilizes a hybrid CNN-Transformer backbone, wherein convolutional layers are used to obtain low-level spatial features and a Vision Transformer (ViT) encoder is employed to model long-range contextual dependencies. This two-stage representation is able to capture local morphological changes as well as global radiological patterns related to tumor aggressiveness. The clinical stream is a feed-forward network with multi-layer perceptron to generate the latent embedding of the tabular variables that represent patient-specific risk factors. The two encoding branches are aimed at generating modality-specific feature representations in a common latent space which is fusion compatible.

Information from various modes is brought together through an attention-guided fusion module that changes the weights of the modalities depending on the context and the relevance for a particular prediction. The fusion mechanism computes cross-modal attention matrices that map clinical attributes onto image features and vice versa, thereby modelling how radiological abnormalities interact with clinical biomarkers. The fused embedding is passed through a joint prediction head that outputs class probabilities for diagnostic labels such as benign versus malignant status or tumor sub-type categories.

A conceptual diagram of the architecture (Figure 1) consists of parallel imaging and clinical streams feeding into an attention-based fusion block, followed by a unified classifier and an explainability generator. The imaging branch processes normalized TCIA/CBIS-DDSM scans through CNN and Transformer encoders, the clinical branch encodes structured TCGA/EHR variables, and the fusion block produces a single multi-modal vector used for classification and interpretation.

The uni-modal feature extraction and cross-modal attention fusion are mathematically formulated in Equations 1–4, while the classification and optimization steps are defined in Equations 5, 6. Together, Equations (1– 6) provide the complete mathematical description of the proposed explainable multi-modal framework.

Let X_I denote the preprocessed radiological image input (image_p) and X_c denote the preprocessed structured clinical feature vector (clinical_p). The multi-modal framework consists of modality-specific encoders, a cross-modal attention fusion mechanism, and a unified classifier.

To provide transparent and clinically relevant interpretations, the model has integrated explainability at two levels: (1) image-level visualization with Grad-CAM++ and attention-rollout to locate the spatial tumor regions influencing the prediction and (2) clinical feature attribution with SHAP to determine the contribution of biomarkers, demographic variables, and laboratory features. The explanations of the two modalities are merged into a multi-modal explanation map that not only aligns the highlighted radiological regions with the corresponding clinical determinants but also, thus, it addresses the research goal of illustrating the fused decision pathways. Moreover, consistency checks are performed to ensure that explanations remain unchanged even if perturbations are applied, thereby reducing the risk of interpretations that are incorrect. This design differs from conventional multimodal XAI pipelines, where explanations are generated independently for each modality, by explicitly coupling explanation generation with the fusion mechanism so that both visual and clinical attributions reflect a shared multimodal decision pathway.

In order to facilitate transparent and clinically meaningful decision-making at the time of inference, a trained multi-modal model is supplemented with a dedicated explainability workflow. This operation produces supplementary explanations at both the image and clinical feature levels, thus, users are informed not only about the final diagnostic prediction but also about the multi-modal evidence that has led to it. The complete inference and explainability process that has been used during the deployment of the model is presented in Algorithm 2.

The dataset is splitted into training, validation, and testing subsets through stratified sampling to maintain class balance. Training is done with AdamW optimization, cosine learning-rate scheduling, and early stopping based on validation loss. Performance metrics include accuracy, sensitivity, specificity, AUC, and F1-score, while interpret-ability metrics include explanation faithfulness, attribution stability, and clinician-alignment scoring. Where possible, evaluation incorporates expert radiologist review to ensure clinical plausibility.

All statistical analyses were performed following standard practices in medical AI evaluation. Performance metrics were computed with confidence intervals obtained via bootstrapping, and comparisons between models were conducted using consistent train–validation–test splits to avoid data leakage. The evaluation protocol was reviewed to ensure appropriate metric selection, sufficient sample representation, and methodological validity. These measures ensure that the reported results are statistically sound and reproducible.

Experiments were conducted on CBIS-DDSM, TCIA Breast MRI, and TCGA (BRCA, LUAD, and GBM) cohorts using stratified training, validation, and test splits. Uni-modal image-only and clinical-only baselines, as well as non-explainable multi-modal variants, were implemented under identical training settings to ensure fair comparison. Performance was evaluated using accuracy, sensitivity, specificity, F1-score, and AUC. Explainability was assessed using attribution faithfulness, stability, and clinician-alignment metrics. Stability and faithfulness were evaluated using SSIM-based heatmap similarity and attribution-removal confidence degradation tests.

Table 2 presents a summary of the diagnostic capability of the suggested framework as a comparison to various uni-modal and multi-modal baselines on breast cancer datasets. The explanable multi-modal model kept on delivering enhanced performances, showing that it was able to better balance sensitivity and specificity.

Figure 2 displays representative image-level explanations obtained through Grad-CAM++. The superimposed heatmaps on mammography and MRI images point to the spatial areas that have the strongest influence on the model’s diagnostic predictions. The locally activated regions are in line with the clinically relevant tumor areas, which implies that the suggested model is concentrating on significant radiological features for the decision-making process.

A comparative evaluation of different model configurations was performed to understand the contribution of multi-modal integration more clearly. As outlined in Table 3, uni-modal strategies had some inherent disadvantages that were revealed when these methods were used in isolation. Image-only models did not have enough patient-specific contextual information, whereas clinical-only models had reduced discriminative capability because they lacked visual tumor characteristics.

Multi-modal models with simple feature concatenation were slightly better than uni-modal baselines; however, their performance was still not ideal due to the very limited cross-modal interaction. On the other hand, the proposed explainable multi-modal framework, which combines attention-based fusion with explainability mechanisms, was able to achieve the most balanced and robust performance. In fact, its predictive accuracy was at par with or even better than that of non-explainable multi-modal baselines, thus, the incorporation of explainability does not compromise the diagnostic effectiveness.

Figure 3 shows the comparison of the various model configurations by means of bar plots of Accuracy, AUC, and F1-score. The multi-modal model with attention-based fusion and explainability that was proposed is always better than the uni-modal and simple fusion baselines, which proves that adaptive multi-modal integration enhances predictive performance to even higher levels of accuracy.

SHAP-based clinical feature attributions are summarized in Table 4. Patient age, tumor stage, hormonal receptor status, tumor size, and selected biomarkers emerged as dominant contributors to diagnostic predictions. These findings are consistent with established oncological risk factors.

Figure 4 presents SHAP-based clinical feature attributions for representative test samples. The average absolute SHAP values show how much the different structured clinical variables contributed to the diagnostic predictions. Among these variables, patient age, tumor stage, hormonal receptor status, tumor size, and selected biomarkers were the most influential factors. Moreover, these indications to the factors align with the cancer risk factors that have been already verified by science and go hand in hand with the local image-level explanations.

Figure 5 demonstrates the agreement of multi-modal explanations generated by the proposed model for a representative test case. The pixel-level Grad-CAM++ heatmap localizes the cancerous regions that are most relevant for diagnosis, whereas the associated SHAP-based clinical feature attributions point to the most influential patient-specific variables. The fusion of visual and clinical explanations indicates consolidated multi-modal reasoning and hence, provides user-friendly interpretation of the model’s predictions in the clinical domain.

The generalization capability of the proposed framework was assessed on the three cohorts: TCGA BRCA, LUAD, and GBM. Table 5 presents the performance trends for each dataset, which demonstrate that the efficiency and explainability of the method were maintained for different cancer types.

In fact, one of the major points that can be inferred from Figure 6 is that the proposed framework has the potential to generalize its pool of knowledge beyond the data used for training, as can be seen from the comparison of their performance on the internal and external test sets. The model is still able to keep the same level of precision, AUC, and F1-score throughout the datasets, which is strong evidence that it is very resistant to any changes in the underlying data distribution and thus can be used in other domains apart from the one where it was trained.

Experienced radiologists qualitatively evaluated that multi-modal explanations raised the diagnostic confidence level more than image-only explanations. Clinicians stated that the correspondence between the brightly tumor regions and the most influential clinical features helped them to use their intuitive reasoning and also to confirm the model predictions. Figure 7 shows the clinician-centered assessment of the proposed explainable multi-modal framework. Responses to the survey reveal that most of the clinicians were in agreement or strong agreement that the generated explanations were instrumental in understanding disease characteristics and also led to enhanced diagnostic confidence. These results are consistent with the clinical relevance and interpret-ability of the proposed method, which is a great indication of its potential as a decision-support tool in real diagnostic scenarios.

An ablation study was performed to measure how much each component of the proposed explainable multi-modal framework contributed to the overall effect. Basically, the study systematically removed or changed key architectural elements like modality usage and fusion strategy while still keeping all other training and evaluation settings the same. This analysis helps to understand how important multi-modal integration and attention-based fusion are relative to each other.

Table 6 provides a summary of the performance trends that were observed across various ablation configurations. The image-only and clinical-only models showed a significant drop in performance as they lacked the information from the complementary modality. Multi-modal models with simple feature concatenation as the method of fusion had slight improvements over the uni-modal baselines, but they were still unable to fully exploit the cross-modal interactions. On the other hand, the proposed attention-based multi-modal fusion has been the most robust and balanced performance, thus, it has been confirmed that adaptive modality weighting is the most effective.

Throughout all the assessments, the suggested explainable multi-modal framework was a high performer in terms of diagnostic power, showed strong generalization abilities across datasets, and provided explanations that were clinically relevant. The use of attention-based multi-modal fusion along with dual-level explainability allows for precise and transparent cancer diagnosis, thus the research objectives have been accomplished. Quantitative evaluation further confirmed the robustness of the explanations. Grad-CAM++ heatmaps demonstrated high structural similarity under input perturbations, and SHAP attribution vectors showed strong correlation stability. Faithfulness tests revealed a significant reduction in prediction confidence when highly attributed regions or clinical features were removed, indicating that the explanations reliably reflect the model’s internal decision logic.

The comparative performance analysis illustrates that the integration of multi-modal information dramatically improves the diagnostic accuracy as compared to the uni-modal methods. Image-only models, although they were able to capture the spatial tumor characteristics effectively, were deficient in patient-specific context. On the other hand, clinical-only models could not capture the visual heterogeneity present in the radiological scans. The proposed framework, thus, effectively utilized the complementary information from both modalities through attention-based fusion, leading to a more balanced and robust diagnostic model.

These results corroborate those of previous multi-modal studies in medical imaging, which have demonstrated that the integration of heterogeneous data sources can lead to better predictive performance. The proposed attention-based fusion mechanism, however, differs from most of the existing methods that depend on simple feature concatenation. It adaptively adjusts the weights of modality contributions, thereby allowing the model to give more emphasis to the clinically relevant information depending on the diagnostic context.

One of the major contributions of this research is the integration of multi-modal explainability, which was done without a reduction in predictive performance. The image-level explanations made by Grad-CAM++ frequently included tumor regions that were not only visually obvious but also made sense from a diagnostic point of view, thereby matching the areas annotated by radiologists. At the same time, clinical feature attributions based on SHAP pointed patient-level variables that have the most significant impact like age, tumor stage, and hormonal receptor status, which are generally known risk factors in oncology.

The agreement between image-based and clinical explanations is especially valuable from a healthcare point of view. Instead of giving separate or even potentially contradictory explanations for each modality, the suggested framework provides concerted multi-modal interpretations that reflect the diagnostic reasoning of the real world. This conformity increases the clinician’s confidence in the system and thereby overcomes the problem of deep learning systems being integrated into clinical workflows, which is a major issue.

Assessing multiple datasets and different cancer types, the proposed framework was shown to be applicable beyond a single imaging modality or disease context. The consistent performance across TCGA cohorts indicates that the model identifies diagnostic patterns that can be transferred and are not mere artifacts specific to the dataset. Such robustness is essential for use in the real world, where changes in imaging protocols, patient demographics, and institutional practices are to be expected. Additionally, explainability metrics showed that the attributions were stable and accurate even when the input was changed, thus, the explanations are not simply post-hoc visualizations but they correspond to the most important factors for the model’s decision.

The proposed framework is intended to function as a clinical decision-support and second-opinion system, rather than as a fully autonomous diagnostic tool. Its primary objective is to assist clinicians by enhancing transparency, improving diagnostic confidence, and supporting interpretability through coherent multi-modal explanations that combine radiological evidence with clinical risk factors. The system is designed to complement clinical expertise by facilitating case prioritization, supporting confirmation of diagnostic hypotheses, and improving understanding of complex or ambiguous cases.

Clinically, the new model is most appropriately a support tool for decision-making, not a fully independent diagnostic system. A physician’s confirmation of a diagnostic hypothesis, a prioritization of the cases, and a recognition of the risk factors could all be facilitated by the combination of the correct predictions and the clear explanations. Moreover, the evaluation from the viewpoint of a doctor strongly indicates this function, as the experts said that their diagnostic confidence was raised when multi-modal explanations were provided. Such machines may become priceless especially in complicated or unclear situations where medical imaging cannot provide complete information and thus has to be combined with patient history and clinical biomarkers.

Though the study yielded promising results, it is still burdened with numerous limitations. First of all, the evaluation was based on retrospective, publicly available datasets, which might not account for the full extent of variations in real-world clinical scenarios. Secondly, the clinician assessment was qualitative and of a small scale, involving a limited number of experts. Although the feedback indicates strong clinical relevance and improved diagnostic confidence, larger multi-center studies with structured quantitative usability metrics are required to comprehensively validate the framework’s clinical effectiveness and real-world deployment readiness. Thirdly, even though Grad-CAM++ and SHAP offer understandable explanations, they are still post-hoc methods and might not necessarily identify the true causal relationships in the model.

Moreover, the present setup is based on the assumption that complete multi-modal data are available. The issue of missing or partially observed modalities is still unresolved and represents a significant avenue for future research.

Further studies aim to develop the framework for clinical trials and deployment in real-time scenarios. The use of uncertainty estimation, causal explainability methods, and temporal clinical data might, in fact, improve confidence and trust even more. Investigating approaches for resilient multi-modal learning when data is missing and broadening clinician-in-the-loop assessment will, likewise, be essential to the translational effect.

Overall, this research shows that an explainable multi-modal deep learning model can deliver excellent diagnostic results as well as provide clinically logical and reliable explanations. The innovative system that the authors present, which combines attention-based fusion with dual-level explainability, seems to be an effective way of tackling the main issues articulated in the field of medical AI regarding precision, openness and use by the medical community. Thus, it represents a potential solution for making cancer diagnosis systems not only more reliable but also interpret-able, which is essential for their integration in clinical practice.