The integration of multi-omics data has emerged as a powerful strategy to enhance disease diagnosis by capturing the biological complexity underlying disease phenotypes (Masana et al., 2020). Multi-omics data typically include genomics, transcriptomics, proteomics, metabolomics, and epigenomics. Each omics layer offers distinct but complementary information about the molecular state of a biological system. In disease diagnosis, especially for complex diseases such as cancer and neurodegenerative disorders, single-omics approaches often fail to capture the full spectrum of molecular alterations. Integrative approaches aim to combine these heterogeneous data types to achieve more comprehensive insights into disease mechanisms and more accurate predictions (Lahiri et al., 2019). Data integration strategies can be broadly classified into early integration (concatenation-based), intermediate integration (model-based), and late integration (decision-based) methods. Early integration methods simply concatenate features from each omics layer into a single feature vector, which is then used to train machine learning models. While straightforward, this approach suffers from the curse of dimensionality and potential information redundancy (Zhang et al., 2022). Intermediate integration strategies use algorithms such as canonical correlation analysis, kernel methods, or deep learning to find shared representations between omics datasets before classification. Late integration methods involve training separate models for each omics layer and then combining their outputs through ensemble methods or voting schemes (Lahiri et al., 2006). Recent advancements in deep learning have further enabled the development of more sophisticated intermediate integration strategies. For instance, autoencoders and variational autoencoders have been used to learn compact, non-linear embeddings of multi-omics data, which can then be fused with image features for downstream classification. Multi-modal deep learning frameworks that simultaneously process omics and image data streams have demonstrated improved performance in disease classification tasks (Mascarenhas and Agarwal, 2021). In the context of image classification, multi-omics data often serve as complementary inputs to imaging features extracted from modalities such as MRI, CT, or histopathology (Roy et al., 2023a). For example, in cancer diagnosis, integrating gene expression data with histopathological images has been shown to improve the prediction of tumor subtypes and patient outcomes. The integration enhances the interpretability of the image features by linking them to molecular pathways and biological processes. The primary challenges in multi-omics integration include data heterogeneity, missing values, and the need for large, annotated datasets. Furthermore, effective feature selection and normalization techniques are critical to mitigate the effects of noise and batch effects. Addressing these challenges requires a combination of domain knowledge, computational innovation, and rigorous validation protocols (He et al., 2025).

In recent years, a number of computational tools have been developed to address the challenge of multi-omics data integration in disease analysis and patient stratification. Methods such as PINSPlus (Nguyen et al., 2019) and Cancer Integration via Multikernel Learning (CIMLR) (Ramazzotti et al., 2018) have demonstrated the utility of combining genomic, transcriptomic, and epigenomic profiles to discover tumor subtypes, using consensus clustering and similarity-based fusion, respectively. More advanced frameworks like iCluF (Shakyawar et al., 2024) employ iterative cluster-fusion strategies to enhance stability in unsupervised multi-omics integration, while Subtype-GAN (Yang et al., 2021a) applies generative adversarial networks to improve latent feature extraction across heterogeneous biological data sources. These tools have shown that integrating molecular layers leads to more biologically meaningful subtyping and better clinical interpretability (Kautish et al., 2021). Beyond cancer subtyping, integration models have also been extended to public health and epidemiological contexts. For example, Shakyawar et al. (2021) proposed a big data framework that jointly models multi-omics data and comorbidities in the context of COVID-19 and systemic diseases, demonstrating the feasibility of applying integrative analytics beyond oncology. These prior efforts underscore the methodological relevance of multi-modal fusion in complex disease modeling. Building upon this foundation, our study introduces a visual–omics framework that complements these integration strategies by embedding visual pathology data into the multi-omics learning process. Compared to existing methods, our approach incorporates spatial, immune, and ecological dynamics, offering a unified probabilistic representation suitable for both classification and latent inference under real-world biological constraints.

Deep learning has revolutionized the field of medical imaging by enabling the automated extraction of hierarchical features from complex image data. Convolutional neural networks (CNNs), in particular, have demonstrated remarkable success in tasks such as image classification, segmentation, and detection across various imaging modalities, including MRI, CT, ultrasound, and digital pathology (Woo et al., 2023a). The use of deep learning in disease diagnosis has become prevalent due to its ability to learn discriminative features that may not be visible to the human eye (Taori et al., 2020). Traditional machine learning approaches relied heavily on handcrafted features, which are not only labor-intensive to design but also limited in their ability to capture high-level semantic information. Deep learning models, however, learn feature representations directly from raw image pixels in a data-driven manner. This capability has led to significant improvements in diagnostic accuracy, especially in the classification of diseases such as diabetic retinopathy, lung cancer, Alzheimer’s disease, and breast cancer (Peng et al., 2022). In multi-omics-based image classification, deep learning frameworks can be used to fuse visual and molecular data, leading to more robust and biologically meaningful predictions. Several architectures have been proposed for this purpose, including multi-branch networks where separate CNN branches process image and omics data before combining them in a shared representation space (Lahiri et al., 2012). Attention mechanisms and graph neural networks have also been utilized to enhance the model’s ability to focus on relevant features across different data modalities. Transfer learning and pretrained models play a crucial role in medical imaging, particularly when labeled data are scarce. Models trained on large datasets such as DIBaS can be fine-tuned on medical imaging tasks to achieve better generalization. Data augmentation techniques and synthetic image generation using generative adversarial networks (GANs) have also been employed to overcome data limitations (Bazi et al., 2021). Despite these advancements, challenges remain in terms of model interpretability, generalization across institutions, and regulatory approval for clinical deployment. Explainable Artificial Intelligence (AI) techniques, such as saliency maps and class activation maps, are increasingly being integrated into deep learning workflows to provide insights into model decisions and enhance clinical trust (Lu et al., 2025).

In recent years, attention mechanisms have emerged as a transformative component in deep learning models, particularly in the field of medical imaging. Unlike traditional convolutional operations that rely on fixed receptive fields, attention modules dynamically weigh the importance of spatial, channel-wise, or cross-modal features, enabling models to focus on the most informative regions of the input data. This has led to significant improvements in tasks such as lesion localization, fine-grained classification, and modality fusion. For instance, self-attention modules in transformer-based architectures, such as Vision Transformers (ViT), have demonstrated competitive performance in radiology and pathology tasks by capturing long-range dependencies and contextual interactions more effectively than CNNs. Cross-attention designs have been employed to link image features with complementary data modalities, such as gene expression profiles or clinical metadata, supporting more robust diagnosis under heterogeneous data conditions. Recent works have also explored the integration of attention into multi-branch architectures, where image and omics features are processed separately and then aligned via learnable attention weights. These strategies improve the interpretability and adaptability of medical AI models, especially in scenarios with partially missing or noisy modalities. In our work, attention is used as a core mechanism in both spatial feature refinement and modality-level fusion. Specifically, we adopt a cross-modal attention layer that aligns omics-derived embeddings with spatial image features, allowing the model to selectively integrate molecular signals in a region-aware manner. This design choice is supported by the growing body of literature that demonstrates the superiority of attention-based fusion in medical image understanding and multi-modal learning.

Multi-modal fusion refers to the computational techniques used to integrate data from different modalities to improve the performance and reliability of disease diagnosis models. In the context of multi-omics and image data, fusion techniques are critical for leveraging the complementary nature of visual and molecular information. Effective fusion not only enhances classification performance but also improves biological interpretability and clinical relevance (Bugatti et al., 2025a). Fusion techniques are generally categorized into three levels, including data, feature, and decision levels. Data-level fusion combines raw data from different sources before any processing, which is rare due to the differences in data structure and dimensionality (Zhang et al., 2025a). Feature-level fusion extracts features from each modality separately and then concatenates or combines them using methods such as tensor fusion, bilinear pooling, or attention-based mechanisms. Decision-level fusion aggregates the outputs of separate classifiers, using ensemble methods such as majority voting or stacking (Zheng et al., 2022). Advanced feature-level fusion methods have been developed using deep learning. These include architectures like cross-modal transformers, dual-stream CNNs, and hybrid networks that incorporate recurrent layers or attention modules. Such models are capable of capturing complex relationships between modalities and can dynamically weigh the importance of each modality during classification. Multi-modal fusion has been applied successfully in several disease diagnosis tasks (Bugatti et al., 2025a). For instance, in glioma classification, combining MRI features with gene expression profiles has led to more accurate predictions of tumor grade and molecular subtype. Similarly, integrating histopathology images with proteomic and transcriptomic data has enhanced the identification of prognostic biomarkers in breast cancer (Dai and Gao, 2021). An important aspect of multi-modal fusion is the alignment of data across modalities, both temporally and spatially. For example, aligning biopsy samples with corresponding imaging data requires careful annotation and registration techniques. Moreover, data normalization and dimensionality reduction are essential preprocessing steps to ensure that the fused features are compatible and informative (Zhang et al., 2025a). The success of multi-modal fusion depends on the availability of high-quality, well-annotated datasets, as well as on computational frameworks that can efficiently handle the scale and complexity of multi-omics and imaging data. With continued advancements in data acquisition technologies and computational methods, multi-modal fusion is poised to play a central role in the next generation of precision medicine tools (Dong et al., 2022).

Disease microbiology represents a pivotal field that investigates the interactions between microbial agents and their hosts, focusing on the molecular, cellular, and ecological aspects of infectious diseases. This subsection aims to provide a comprehensive introduction to the methodological framework and conceptual underpinnings that form the basis of our proposed approach. We outline the structure and contributions of the ensuing sections, highlighting their roles in the broader methodology.

We begin by formally defining the core research problem and mathematical structure in Section 3.2. This part introduces a symbolized abstraction of the disease microbiology setting, capturing both host–pathogen dynamics and the multi-scale interactions that influence disease progression. The section lays out the notational foundation, integrating principles from epidemiology, molecular biology, and systems microbiology. The mathematical formulations not only enable a deeper understanding of pathogen behaviors under various environmental and immunological contexts but also provide a tractable basis for model development and analytical reasoning. Following the foundational formalism, Section 3.3 introduces a biologically inspired computational architecture tailored for disease microbiology. The new model encapsulates pathogen evolution, host responses, and microenvironmental feedback using a dynamic, structured representation. In contrast to conventional mechanistic or statistical models, our design leverages advances in probabilistic graphical modeling and latent variable learning to simulate the heterogeneous trajectories of infection. It incorporates high-dimensional, multi-modal datasets and is engineered to accommodate partial observations and inherent biological variability. Moreover, it addresses challenges in modeling latent reservoirs, horizontal gene transfer, and cross-species spillover with novel algorithmic components. Subsequently, in Section 3.4, we detail a bespoke inference strategy that facilitates the efficient training and deployment of the proposed model. Recognizing the complexity of microbial interaction networks, this strategy employs an adaptive, energy-based exploration mechanism to identify equilibrium states in host–pathogen systems. It integrates domain-specific priors, such as host susceptibility maps and known resistance loci, to guide search trajectories and optimize model fidelity. The strategy is designed to handle large-scale data inputs, including metagenomic sequencing, gene expression profiles, and spatial–temporal disease incidence patterns. It further incorporates mechanisms to balance accuracy and generalizability, enabling robust performance across diverse biological systems and disease contexts.

In this section, we formalize the problem of disease microbiology from a symbolic and mathematical perspective, focusing on the intrinsic dynamics between pathogens, host immune responses, and their surrounding environment. The goal is to construct an abstracted, mathematically tractable representation that captures the complexity of microbial infections and host–pathogen interactions over time and space. This formulation provides the foundation upon which our modeling and inference methods are built.

Let P denote the set of all microbial pathogens under consideration, indexed by p∈P. Let ℋ denote the set of hosts, indexed by h∈ℋ. Each pathogen p is associated with a genomic signature gp∈ℝdg and a set of virulence factors vp∈ℝdv. Each host h has an immune profile ih∈ℝdi and a susceptibility vector sh∈ℝds.

We denote the infection state of a host h by pathogen p at time t as xhpt∈{0,1}, where xhpt=1 indicates active infection. The probability of infection is governed by a dynamic process (Equation 1).

where σ(·) is the sigmoid function, ϕp∈ℝdϕ is a pathogen-specific coefficient vector, and Ψhpt=concat(gp,vp,ih,sh,xhp(t−1)) is the joint feature mapping.

The overall disease expression level yhpt∈ℝ can be modeled (Equation 2).

where zhpt=concat(gp,vp,ih,sh), ℰhpt∈ℝde denotes environmental covariates, and ϵhpt∼N(0,σ2) represents stochastic noise.

To account for spatial propagation, let N(h) be the set of neighboring hosts. The spatial infection influence ηhpt is defined (Equation 3).

with weights whh′ encoding contact frequency or proximity between hosts. The weight whh′ represents the interaction strength or proximity between hosts h and h′ and modulates the degree of latent state influence across the spatial host graph. In datasets with geolocation metadata, whh′ is computed as a Gaussian kernel over the Euclidean distance between sampling coordinates. In clinical or ecological datasets, it can alternatively reflect contextual similarity such as co-hospitalization, shared community, or temporal overlap. This formulation allows the model to encode soft neighborhood-level infection influence even in the absence of explicit contact tracing data.

The growth potential of pathogen p at time t under nutrient condition ν_t is captured (Equation 4).

where α is a nutrient responsiveness vector and δ_p reflects the death rate or competition disadvantage of pathogen p. The competition disadvantage term reflects the ecological principle that not all pathogens can coexist with equal likelihood within a host or environment. In our model, it is implemented by adjusting the infection probability of pathogen p based on the predicted presence of competing pathogens p′. A suppression coefficient α_pp′ encodes the strength of disadvantage imposed by p′ on p. This mechanism allows the model to favor dominant strains while suppressing less fit or competitively excluded ones, aligning with real-world infection dynamics observed in microbiome and virology studies.

We define the global epidemiological potential Ξ_t over the host population ℋ (Equation 5).

which measures the system-wide infection propagation force at time t.

To capture the multi-scale complexity of host–pathogen interactions in disease microbiology, we propose a novel computational architecture termed PathoGenesisNet. This model integrates pathogen genetic determinants, host immune states, environmental mediators, and temporal–spatial dynamics through a biologically informed latent state formulation. The architecture is designed to be modular, interpretable, and data-adaptive, and it facilitates analysis across heterogeneous biological systems (as shown in Figure 1).

To effectively deploy the proposed PathoGenesisNet framework in complex biological environments, we introduce a novel inference strategy termed microbial equilibrium search strategy (MESS). This strategy is designed to iteratively approximate the latent microbe–host system’s equilibrium states by leveraging domain-informed priors, biological constraints, and spatiotemporal observations. MESS integrates ideas from variational inference, energy-based optimization, and population ecology to form a biologically grounded inference mechanism (as shown in Figure 3).

DIBaS dataset (GP et al., 2023) is a high-resolution biomedical image dataset designed for the classification of bacterial species. It comprises microscopic images of various bacterial genera obtained using differential interference contrast (DIC) microscopy. The dataset includes 660 images grouped into 33 bacterial classes, each containing 20 images. DIBaS is particularly useful for developing deep learning models focused on medical image classification and microbial phenotyping. The standardized acquisition and diverse visual textures across classes make it suitable for tasks such as fine-grained classification, feature extraction, and transfer learning in microbiological domains. Its clear morphological distinctions support experiments in interpretable vision models and robustness evaluation in clinical diagnostics. KAU-BCMD Dataset (Nadkarni and Noronha, 2023) is a curated dataset for bacterial colony morphology detection and classification. It includes over 12,000 high-quality images covering different types of bacterial colonies grown on nutrient agar plates. The dataset captures variations in colony shape, size, margin, elevation, and color, annotated by microbiologists for ground truth verification. Each sample is labeled with colony type, and metadata includes cultivation time and conditions. KAU-BCMD is well-suited for developing models aimed at automated colony recognition, phenotype clustering, and biological trait prediction. It has been used in research on computer-aided diagnosis, microbial ecology, and pathogen detection using vision-based systems. TBX11K dataset (Liu et al., 2020) is a floral taxonomy dataset known as the Oxford 102 Flower Dataset. It contains 8,189 images categorized into 102 flower species commonly observed in the United Kingdom. The dataset features extensive intra-class variation due to differing camera angles, lighting, and environmental backgrounds. Each species has 40–258 images, and class labels are derived from expert annotations. TBX11K is ideal for fine-grained classification tasks, where visual cues such as petal texture, color, and structure are critical. The dataset is frequently used in transfer learning benchmarks, representation learning studies, and zero-shot recognition of subtle semantic attributes in natural imagery. Malaria dataset (Arshad et al., 2022) comprises over 27,000 cell images labeled as either parasitized or uninfected, derived from thin blood smear slides. The images are collected under consistent microscopy settings and manually annotated by experts. Each image captures a single red blood cell and is intended for the binary classification of malaria presence. The dataset enables the training and validation of automated diagnostic models using CNNs and other deep learning techniques. It plays a vital role in real-world healthcare applications, especially for low-resource settings, supporting tasks such as infection detection, model generalization across staining styles, and mobile diagnostic integration.

We conduct all experiments using PyTorch on NVIDIA V100 GPUs with 32GB memory. We adopt ResNet50 and ViT-B/16 as our backbone architectures for all baseline comparisons and ablation studies. We train each model using stochastic gradient descent (SGD) with momentum or AdamW, depending on the backbone. For ResNet50, we use SGD with a momentum of 0.9 and a weight decay of 1e−4. For ViT-B/16, we use AdamW with a weight decay of 0.05 and β₁ = 0.9, β₂ = 0.999. We initialize the learning rate to 0.01 for ResNet and 3e−4 for ViT models. We use a cosine annealing schedule to decay the learning rate across training epochs. We perform training for 100 epochs for ResNet-based models and 300 epochs for ViT-based models. We set the batch size to 256 unless memory constraints require adjustment. We resize all input images to 224 × 224. We use standard data augmentation, including random horizontal flipping, random cropping, and color jittering. For ViT training, we adopt RandAugment and Mixup strategies with α = 0.2 for Mixup and a cutmix probability of 0.5. We apply label smoothing with ϵ = 0.1 to improve generalization. For evaluation, we report the top 1 and top 5 accuracy metrics. For multi-label datasets such as DTD, we use mean average precision (mAP). We use early stopping based on validation accuracy and save the model checkpoint with the best performance. We repeat all experiments with three random seeds, and we report the average results with standard deviation. We use mixed-precision training via NVIDIA Apex to accelerate training and reduce GPU memory usage. We apply gradient clipping with a max norm of 5.0 to stabilize training in large-scale scenarios. For datasets with imbalanced classes, such as TBX11K, we apply class-balanced sampling and focal loss to address the skewed distribution. We use dropout and layer normalization in transformer-based models to prevent overfitting. For fine-tuning pretrained models, we freeze the first few layers during the initial 10 epochs and gradually unfreeze all layers. We apply a learning rate warm-up for the first 5 epochs using a linear schedule. For DTD, since the dataset is small and visually diverse, we use strong regularization and a higher dropout rate of 0.5. For KAU-BCMD and DIBaS, we follow the standard training–validation–test split without modification to maintain benchmark consistency. We select all hyperparameters based on cross-validation and standard configurations reported in prior top-tier conference papers, including CVPR and NeurIPS. Code implementation ensures reproducibility by fixing random seeds and using deterministic operations where applicable. During testing, we evaluate models using a center crop of the input images. We use test-time augmentation (TTA) only for final state-of-the-art (SOTA) comparison and not for ablation studies. Our experimental setup is designed to ensure fair and robust 482 comparison across datasets and architectures and to validate the effectiveness of each proposed component under consistent training pipelines.

Tables 1, 2 provide a comprehensive comparison between our proposed method and several SOTA models across four widely used benchmarks, including DIBaS, KAU-BCMD, TBX11K, and MalariaDataset (DTD). As shown in the tables, our approach outperforms all baselines in every metric, consistently achieving the highest accuracy, precision, recall, and F1 score. On the DIBaS dataset, our method achieves a top accuracy of 84.61%, outperforming the closest competitor, Swin-T, by 2.38%, and showing substantial improvements over widely used models such as ResNet50 (78.12%) and ViT (81.47%). These results confirm the advantage of our approach in modeling cross-domain microbial image features under high intra-class variance. These improvements are reflected across all metrics—our method leads by more than 1.5% on both precision and recall, suggesting a better balance between false positives and false negatives. Similarly, on the KAU-BCMD dataset, our model again surpasses other architectures, with a notable 89.42% accuracy and 89.03% F1 score. It performs better than ViT and ConvNeXt, showing not only strong generalization but also high robustness to object variability and intra-class diversity. The consistent gains in recall (89.75%) and precision (88.33%) indicate that our method effectively captures both coarse and fine object characteristics. The superiority over convolutional baselines and even advanced transformers highlights our architecture’s unique balance between semantic abstraction and spatial preservation, key to handling complex real-world images. Particularly for KAU-BCMD, which involves high intra-class variation and low inter-class similarity, our model’s context-aware learning and hierarchical representation prove especially advantageous.

Further results on TBX11K and DTD datasets reinforce the advantages of our approach in fine-grained classification and attribute-based tasks. TBX11K is a benchmark known for its intra-class similarity and subtle inter-class differences, posing a challenge even for high-capacity networks. Our model achieves an impressive 93.85% accuracy and 92.74% F1 score, surpassing InceptionV3 (90.61%) and ViT (91.76%) while also outperforming DenseNet201 and EfficientNet-B0 by over 5%. These improvements suggest that our architecture captures minute variations in color, shape, and texture with greater sensitivity, likely due to our adaptive multi-scale feature encoding and targeted regularization. For example, the precision gain (92.47%) implies fewer false positives, crucial in distinguishing flowers with highly similar patterns. On the DTD dataset, our method achieves the highest accuracy of 79.63%, surpassing InceptionV3 (77.12%) and ViT (76.54%). Transformer-based models, while effective in modeling global dependencies, often struggle on texture-centric datasets due to their lack of inherent inductive bias for capturing local spatial patterns. Unlike convolutional layers, self-attention modules do not emphasize localized feature hierarchies unless explicitly guided by architectural constraints. To address this limitation, our model introduces an enhanced cross-channel attention mechanism that adaptively reweights spatial features across both local and global contexts. This is further coupled with a fine-grained feature extractor, implemented as a shallow multi-scale convolutional block prior to attention fusion, which captures local edge patterns, repetitive textures, and fine structural motifs commonly seen in DTD samples. By combining global reasoning with explicit local encoding, our architecture becomes more sensitive to texture variations while maintaining discriminative capacity across categories. These design choices lead to superior performance in distinguishing subtle texture classes, as reflected in our F1 score and area under the curve (AUROC) improvements over transformer-only baselines. The gains across precision (78.42%) and recall (80.15%) reflect improved semantic coherence in our representations, allowing the model to detect and differentiate between abstract attributes (like bubbly, striped, or zigzagged) more effectively. The performance superiority across different types of datasets—large-scale, coarse-grained, fine-grained, and texture-focused—confirms the generalization strength of our method and its ability to adapt to diverse visual recognition scenarios.

The consistent improvements across all benchmarks can be attributed to several critical design choices in our architecture and training methodology. First, the incorporation of hierarchical token refinement in our model enables progressive enrichment of features from both low-level and high-level semantics, essential for handling datasets with complex or fine-grained characteristics such as Flowers and DTD. Second, our localized attention mechanism embedded within multi-scale layers enhances both spatial and semantic feature extraction, leading to improved context understanding and better boundary preservation. This becomes particularly evident on datasets like DIBaS and KAU-BCMD, where the visual variance is high. Third, our use of adaptive data augmentation and dynamic loss weighting allows the model to balance learning across majority and minority classes, which proves critical on DTD and KAU-BCMD, where class distributions are uneven. The training strategy with warm-up schedules, cosine decay, and label smoothing contributes to stable convergence and better generalization. Unlike models such as ViT and Swin-T, which may suffer from overfitting on smaller datasets or over-smoothing in deeper layers, our method incorporates dropout and layer-wise normalization that dynamically adjust with the learning state, preventing the loss of representational diversity. Notably, our model is able to leverage transformer-based advantages without sacrificing the benefits of convolutional locality, providing a hybrid architecture that is simultaneously expressive, regularized, and lightweight. These advantages, validated empirically, support the deployment of our method as a robust baseline for a wide range of visual tasks from general classification to attribute prediction. Once the results are considered in context, the effectiveness and adaptability of our method become unmistakable, positioning it as a reliable and scalable alternative to existing SOTA techniques.

To directly validate our framework’s capability in real-world multi-omics scenarios, we conduct additional experiments using two benchmark datasets: TCGA-BRCA (histopathology + gene expression) and CPTAC-OV (histopathology + proteomics). These datasets represent distinct biological domains and omics modalities, enabling us to assess the model’s generalizability across cancer types and molecular signals. We extract visual features from H&E-stained whole-slide images using a ResNet50 backbone pretrained on ImageNet. For omics data, we select the top 500 most variable genes (TCGA-BRCA) or proteins (CPTAC-OV) after normalization and log transformation, and then we process them through dense encoding layers. We pass the fused features into our cross-modal attention module and dynamic graph fusion layer. We compare our model against four baselines: image only, omics only, early fusion, and late fusion. As shown in Table 3, our model consistently outperforms all baselines across both datasets. This demonstrates not only the effectiveness of the proposed cross-modal fusion architecture but also its robustness across omics modalities (gene vs. protein) and cancer types. The superior Area Under the ROC Curve (AUC) and F1 scores highlight the clinical relevance of the learned multi-modal representations and confirm our model’s potential for real-world multi-omics precision diagnostics.

Recent developments in multi-modal fusion have introduced advanced architectures that surpass earlier models in integrating heterogeneous data sources. To ensure the competitiveness and contemporary relevance of our method, we select three recent and influential baselines for additional evaluation: TransMed, MMGL-Net, and MFFormer. These models represent state-of-the-art techniques in transformer-based, graph-based, and hierarchical fusion for medical and biological imaging tasks. TransMed uses a dual-stream transformer backbone to integrate imaging with non-visual clinical data, achieving promising performance on multi-modal datasets. MMGL-Net introduces a multi-modal graph learning mechanism to capture relationships across modalities, especially effective for biological networks. MFFormer applies a multi-scale transformer framework that dynamically aligns and fuses spatial and molecular representations across levels of abstraction. All three models are designed to solve the same core challenge as ours: unifying visual and biological data for improved classification. We conduct experiments on the DIBaS and KAU-BCMD datasets using the same preprocessing, metrics, and training configurations to ensure consistency. The results are reported in Table 4. On DIBaS, our method achieves the highest accuracy of 84.61%, exceeding MFFormer (83.24%), MMGL-Net (82.85%), and TransMed (81.72%). On KAU-BCMD, our model maintains its lead with 89.42% accuracy, compared to MFFormer (88.14%), MMGL Net (87.63%), and TransMed (86.48%). These performance margins reflect the effectiveness of our cross-modal attention mechanism and dynamic graph-based fusion strategy in handling biological heterogeneity and spatial-molecular alignment. The superiority in both precision and recall indicates a balanced and robust classification capability, particularly valuable in clinical diagnostics where both false positives and false negatives carry a high cost. This expanded evaluation confirms the proposed method’s advantage over current state-of-the-art architectures in multi-modal medical image classification.

To validate the effectiveness of each core component in our model architecture, we conduct ablation studies by progressively removing or disabling specific modules. As shown in Tables 5, 6, we evaluate the performance of our model on four representative datasets, including DIBaS, KAU-BCMD, TBX11K, and DTD. The configurations include, without latent temporal representation, without immune adaptation modeling, and without trajectory-based posterior formulation. Across all datasets, we observe that each component contributes positively to the model’s final performance. The full model (ours) consistently achieves the highest scores on accuracy, precision, recall, and F1 score. On DIBaS, the accuracy improves from 81.29% (without latent temporal representation) to 84.61% when all modules are present, a clear indication that multi-scale fusion plays a foundational role in capturing both global and local semantic cues. Likewise, on KAU-BCMD, we see a significant leap from 86.37% (without latent temporal representation) to 89.42%, which demonstrates the utility of fused features in handling diverse object categories with varying appearance and scale. The removal of hierarchical attention (without immune adaptation modeling) also causes consistent drops across all metrics, underscoring the role of dynamic feature emphasis in refining relevant regions and suppressing noise. For example, in TBX11K, precision drops from 92.47% to 89.93% and F1 score from 92.74% to 90.46%, suggesting that hierarchical attention is particularly critical for fine-grained feature discrimination. These results collectively confirm that each architectural element is indispensable and that their joint optimization brings compound gains rather than redundant overlaps.

Furthermore, the adaptive token distillation mechanism (component trajectory-based posterior formulation) shows particularly strong contributions in high-density semantic tasks such as DTD and TBX11K. Removing this module (without immune adaptation modeling) leads to a drop in F1 score from 92.74% to 91.42% on Flowers and from 79.28% to 78.62% on DTD. These metrics reveal the benefit of progressive token compression and contextual enrichment, which are critical in preserving long-range dependencies without sacrificing local sensitivity. On DTD, which relies on human-perceived texture descriptions, this component helps disambiguate visual textures like striped and woven by retaining mid-level attributes through aggregated semantic tokens. The recall improvement from 77.50% (without immune adaptation modeling) to 80.15% (ours) further confirms the ability of the complete model to capture visually ambiguous instances more effectively. Meanwhile, the KAU-BCMD results indicate that removing any single module consistently reduces performance, especially on complex classes with high visual variance, such as tools, instruments, or animals. This proves that the synergistic effect between fusion, attention, and distillation is essential for generalization. Interestingly, even when only one component is removed, performance deterioration can be as large as 3.3% in recall or 3.2% in accuracy, highlighting that the performance gain from our method is not solely due to any isolated enhancement but the thoughtful integration of each design choice.

The ablation results strongly validate the modular design philosophy of our architecture. Multi-scale feature fusion (latent temporal representation) is essential for rich contextual aggregation across receptive fields. Hierarchical attention refinement (immune adaptation modeling) enhances semantic saliency and suppresses distractors across the spatial hierarchy. Adaptive token distillation (trajectory-based posterior formulation) improves compact representation and scalability across vision tasks. The experimental outcomes across all four benchmarks confirm that each module not only contributes individually but also amplifies the efficacy of the others when integrated holistically. This modular synergy is what allows our model to surpass conventional CNNs and transformers, which often struggle to maintain a trade-off between local detail preservation and global context modeling. Therefore, the ablation study not only quantifies the contribution of each component but also highlights their interdependence, establishing a clear justification for the architecture of our full model.

To further assess the diagnostic reliability of our model under real-world uncertainty and visual variability, we conduct AUROC-based evaluations on the TBX11K and DTD datasets. These datasets present significant challenges due to their diverse image textures and subtle morphological differences. We compare our approach with three representative baselines—ResNet50, ViT, and InceptionV3—by computing the receiver operating characteristic (ROC) curves and corresponding AUROC values. As shown in Figure 5, our method consistently achieves higher AUROC scores on both datasets, indicating improved discriminative capacity in distinguishing complex infection-related patterns. Specifically, on the TBX11K dataset, our model achieves an AUROC of 0.69, outperforming InceptionV3 (0.65), ViT (0.62), and ResNet50 (0.59). On the DTD dataset, our model reaches an AUROC of 0.61, compared to 0.57 (InceptionV3), 0.56 (ViT), and 0.53 (ResNet50). These improvements suggest that the proposed visual–omics fusion framework offers greater robustness in settings where disease phenotypes exhibit subtle or overlapping visual cues.

To provide additional transparency regarding the training dynamics of our baseline models, we visualize the training loss and accuracy curves for both ResNet50 and ViT architectures. As shown in Figure 6, we train ResNet for 100 epochs with an initial learning rate of 0.01, while we train ViT for 300 epochs with a learning rate of 3e−4. Both models use cosine annealing for learning rate decay. The training loss consistently decreases while training accuracy increases across epochs, indicating stable convergence. For ResNet, the final training accuracy reaches approximately 96%, with a final loss of approximately 0.21. For ViT, we observe smoother convergence with a final accuracy approaching 99% and a loss below 0.05 after 300 epochs. These results validate that both models are sufficiently optimized under the training schedule, and the observed performance differences in evaluation are not due to underfitting.

To provide a more comprehensive evaluation beyond classification accuracy, we further report the precision, recall, F1 score, and training loss for our model and baseline methods across all four datasets. This allows for better assessment of the models’ sensitivity, specificity, and robustness under various imaging conditions. As shown in Table 7, our model consistently outperforms other architectures across all evaluation metrics, achieving notably higher F1 scores and lower training loss. On the DIBaS and KAU-BCMD datasets, our method surpasses ResNet50, ViT, and Swin-T by a significant margin, with F1 scores exceeding 0.90. On the more challenging TBX11K and DTD datasets, our approach maintains a balanced precision-recall profile, achieving the highest F1 score and the lowest loss among all baselines. These results demonstrate that our model not only performs better in overall accuracy but also produces more stable and reliable predictions across diverse microbial and histological image domains.

To ensure a fair and comprehensive evaluation against classical convolutional architectures, we additionally compare our model with two widely adopted CNN baselines—VGG19 and Xception. These models have been extensively used in medical image classification tasks and serve as strong benchmarks in texture- and morphology-driven domains. As shown in Table 8, our model consistently outperforms both VGG19 and Xception across all four datasets in terms of accuracy, F1 score, and AUROC. On the DIBaS and KAU-BCMD datasets, our method yields a notable margin of improvement, reflecting its superior ability to capture discriminative microbial image patterns. On more challenging datasets such as TBX11K and DTD, the hybrid attention and fine-grained feature encoding in our model offer stronger robustness to intra-class texture variations and spatial noise, compared to the fixed receptive field design of VGG-style networks. These results reinforce the versatility and generalizability of our framework across different biological and clinical imaging settings.