High-level vision task-driven fusion methods typically incorporate a semantic segmentation [24–27] or object detection network [23, 28] after the fusion network, using the loss functions from these downstream tasks to constrain the fusion results and improve the quality of the fused image. However, introducing high-level vision tasks at the fused image level only provides indirect guidance for the feature extraction network to learn features relevant to the downstream tasks.

To provide direct task-level guidance at the feature level and further enhance fusion performance, PSFusion [29] injects semantic features extracted from a segmentation task directly into the fusion network. SegMiF [25] feeds the fused result into a semantic segmentation network to extract semantic features, which are then interacted with the multimodal image features from the encoder to enhance the fusion result. MRFS [26] interacts and fuses the source image features before feeding them into a semantic segmentation head to enforce semantic supervision, thereby improving the global scene perception of the fusion network. MetaFusion [28] sends the fused result into an object detection network to extract features, which are then combined with the source image features and passed into a meta-feature generator to guide feature extraction in the fusion branch.

Although these methods improve fusion performance to some extent by leveraging downstream high-level tasks, they all assume that the input images are perfectly aligned in spatial position—a condition rarely met in real-world applications. In practice, such methods rely on additional image registration algorithms to achieve accurate alignment before performing fusion. This not only makes the fusion quality highly dependent on the registration accuracy but also significantly increases the complexity of the overall network design.

To address the problem of unregistered infrared and visible image fusion, most existing approaches combine registration and fusion algorithms, i.e., first aligning the input misaligned image pairs and then performing fusion. However, due to the large modality gap between infrared and visible images, ignoring the adverse impact of modality discrepancy on registration can greatly degrade fusion quality. For instance, ReCoNet [30] adopts this strategy but produces suboptimal fusion results due to this issue. UMF-CMGR [31] and IMF [32] consider the effect of modality differences on registration results. They propose to convert visible images into pseudo-infrared images via an image generation network and then perform mono-modal registration between the pseudo-infrared and misaligned infrared images. However, the quality of the generated image has a direct impact on the final performance of these methods. Moreover, these methods treat registration and fusion as two independent tasks, failing to establish a unified framework where both tasks can benefit each other.

To address this, RFNet [33] and MURF [34] treat image fusion as a downstream task of registration and improve registration performance by enhancing the sparsity of the gradient in the fused result. However, to tackle the modality discrepancy issue during registration, both methods aim to transform the multimodal registration into a mono-modal one. Specifically, RFNet uses an image generation model to produce a pseudo-image with the same modality as the misaligned one before performing mono-modal registration, while MURF leverages contrastive learning to extract modality-invariant features from the input image pair for registration. Similarly, Super-Fusion [35] extracts modality-invariant features using shared-parameter encoders and consistency constraints on the fused result for registration.

Nevertheless, the information carried by modality-invariant features in infrared-visible pairs is often far less rich than the complementary information present in the image pair. As a result, it is difficult to achieve satisfactory cross-modal registration using only modality-invariant features. In addition, the above methods all follow a two-stage approach (registration + fusion). This two-stage strategy greatly limits deployment in practical applications due to computational constraints. Although RFVIF [36], IVFWSR [37] and MulFS-CAP [38] attempt to achieve registration and fusion within a single-stage framework, the types of deformations they can handle remain limited. Unlike the methods mentioned above, our approach considers multiple challenges simultaneously: the impact of modality discrepancy on cross-modal registration, the deployment limitations of two-stage processing, and the feature requirements of downstream high-level vision tasks for both registration and fusion.

As shown in Figure 2, the proposed method consists of three core components: feature extraction, feature alignment and fusion, and dual-task reconstruction. The feature extraction component is designed to obtain both modality-specific and modality-common features from the source images. The feature alignment and fusion component is used to predict a deformation field, which is then used to spatially align the infrared-specific and common features. These aligned features are then fused with the corresponding visible image’s specific and common features. In the dual-task reconstruction stage, the fused features are fed into the object detection head and the image reconstruction head, respectively, to generate both the object detection result map and the fused image.

The main objective of feature extraction is to extract both the common and specific features of infrared and visible images, in order to facilitate subsequent cross-modal registration and feature fusion. This process consists of four modules: the IR-Specific Feature Extraction (IR-SFE) module, the VI-Specific Feature Extraction (VI-SFE) module, the IR-Common Feature Extraction (IR-CFE) module, and the VI-Common Feature Extraction (VI-CFE) module. Among them, the IR/VI-SFE modules are used to extract modality-specific features from the infrared/visible images, while the IR/VI-CFE modules are used to extract their common features. Assume that each sample in the training dataset contains three images: a pixel-wise strictly aligned infrared image I i , a visible image I v , and a deformed infrared image I i d . We feed I i and I i d into the IR-CFE and IR-SFE, respectively, to obtain the infrared common feature F i , the deformed infrared common feature F i d , the infrared specific feature F ̂ i , and the deformed infrared specific feature F ̂ i d . At the same time, we feed I v into the VI-CFE and VI-SFE to obtain the visible common feature F v and the visible specific feature F ̂ v .

In the cross-modal registration process, it is usually necessary to rely on the common information between cross-modal images to establish pixel-wise correspondences. To reduce the modality gap between infrared and visible images and thus establish more accurate pixel-wise correspondences, we introduce a modality consistency loss ℓ c : ℓ c = 1 H W C F i − F v 1 , (1) Here, H , W , and C denote the height, width, and number of channels of the feature maps, respectively, and ⋅ 1 represents the l1-norm. In addition, considering that the goal of image fusion is to integrate as much complementary information as possible from cross-modal source images into a single image, we introduce the modality complementary information loss ℓ s to further enrich the complementary information from the source images in the fused image: ℓ s = − 1 H W C F ̂ i − F ̂ v 1 . (2)

Feature alignment corrects the deformation in infrared features by predicting a deformation field, thereby achieving spatial alignment between infrared and visible features. This process is mainly implemented by the registration network. Subsequently, the aligned infrared features are fused with the visible features to obtain the fused features. As shown in Figure 3, the registration network is composed of a Channel and Spatial Enhancement Block (CSEB) and a Multi-Scale Registration Block (MSRB). The CSEB is mainly used to enhance the information beneficial to registration at both the channel and spatial levels, thereby improving the accuracy of the predicted deformation field. The CSEB consists of six feature extraction layers and a Global Average Pooling (GAP) layer. Each feature extraction layer is composed of a convolutional layer with a kernel size of 3 × 3 , stride 1, followed by Batch Normalization (BatchNorm) and a LeakyReLU activation function. The MSRB is used to predict the deformation field to correct the deformed infrared features and ensure spatial alignment between the infrared and visible features. The MSRB adopts a U-Net-like architecture.

We input the deformed infrared common feature F i d and the visible common feature F v into two CSEBs with unshared parameters, obtaining the enhanced features F ̃ i d and F ̃ v , respectively. Taking the enhancement process of F v as an example, F v is fed into three feature extraction layers to generate the spatial enhancement weights W v s . To enhance registration-relevant information at the spatial level, we perform element-wise multiplication between W v s and F v : F v s = W v s ⊙ F v , (3) Here, F v s denotes the feature enhanced at the spatial level, and ⊙ represents the element-wise multiplication operation. We feed F v s into three feature extraction layers and a global average pooling (GAP) layer to obtain feature W v c for channel-level enhancement. Then, W v c is element-wise multiplied with F v s to produce the enhanced feature F ̃ v , which has been refined at both the spatial and channel levels: F ̃ v = W v c ⊙ F v s , (4) Similarly, we obtain the deformed infrared common feature F ̃ i d enhanced at both the spatial and channel levels. We concatenate F ̃ i d and F ̃ v along the channel dimension and feed the resulting feature into the MSRB to predict the deformation field ϕ . To ensure the accuracy of the predicted deformation field, we introduce a registration loss ℓ reg : ℓ reg = 1 2 H W ϕ − ϕ g t 1 , (5) Here, ϕ g t is the label of ϕ .

We use ϕ to correct F i d and F ̂ i d respectively, resulting in the corrected infrared common feature F i r and infrared-specific feature F ̂ i r : F i r = ϕ    ◦    F i d , F ̂ i r = ϕ    ◦    F ̂ i d , (6) Here, ◦ denotes the Warp operation, which resamples the deformed feature maps based on ϕ to correct the deformations within them. During the fusion process, to minimize information loss, we concatenate F i r , F ̂ i r , F v , and F ̂ v along the channel dimension to obtain the fused feature F f : F f = F i r , F ̂ i r , F v , F ̂ v , (7) Here, [ ⋅ ] represents the operation of concatenation along the channel dimension.

In the dual-task reconstruction, the fused feature is fed into both the object detection head and the image reconstruction head to respectively generate the object detection result map and the fused image. The dual-task reconstruction primarily consists of the object detection head and the image reconstruction head. We adopt YOLOv5 [39] as the object detection head. The image reconstruction head is composed of three feature extraction layers, where the LeakyReLU activation function in the final layer is replaced with a Tanh activation function. The fused feature F f is input into both the object detection head and the image reconstruction head to obtain the object detection result map y ̂ and the fused image I f , respectively. To ensure high-quality object detection results, we introduce the object detection loss ℓ o b to constrain the network: ℓ o b = c yolov5 y , y g t , (8) Here, c yolov5 ( ⋅ ) refers to the loss function used during the training of YOLOv5. In addition, to encourage the fused image to retain as much shared and complementary information from both infrared and visible images as possible, we introduce luminance loss ℓ b and gradient loss ℓ g , and construct the fusion loss ℓ f accordingly: ℓ f = ℓ b + γ ℓ g , (9) Here, γ denotes the balancing hyperparameter. The gradient loss ℓ g is defined as: ℓ g = 1 H W ∇ I f − max ∇ I i , ∇ I v 1 , (10) Here, ∇ denotes the Sobel operator. The luminance loss ℓ b is defined as: ℓ b = 1 H W I f − max I i , I v 1 . (11) Finally, we define the total loss ℓ t as follows: ℓ t = ℓ c + ℓ s + λ 1 ℓ reg + λ 2 ℓ f + λ 3 ℓ o b , (12) Here, λ n ( n = 1,2,3 ) denotes the balancing hyperparameter.

In our experiments, we first compare the proposed method with two categories of fusion approaches for unaligned infrared and visible images based on their fusion results. We then compare the subsequent object detection results obtained using these two categories of methods. The first category involves registering the images to be fused, followed by image fusion and then object detection. We refer to this category as Registration + Fusion + Object Detection. The second category performs joint training of registration and fusion to directly handle unaligned images, followed by object detection. We refer to this as Joint Registration and Fusion + Object Detection.

The core of the proposed method lies in the losses designed to eliminate modality differences, namely, losses ℓ c and ℓ s . In this section, we conduct ablation studies on these key components to verify their effectiveness. All experiments are conducted on the M 3 FD dataset. From the ablation results, it can be observed that removing losses ℓ c and ℓ s leads to a decline in the model’s ability to correct local deformations, as shown in Figure 7. In addition, when the shared information is excluded during fusion and only complementary information is used for concatenation, the visual quality of the fused image does not deteriorate significantly, but the objective evaluation results in Table 4 show a noticeable drop in performance.

In our proposed method, four main hyperparameters are defined: λ 1 , λ 2 , λ 3 , which balances different losses, i.e., ℓ reg , ℓ f , and ℓ o b , and γ , which balances luminance loss ℓ b and gradient loss ℓ g . During model training, λ 1 , λ 2 , λ 3 , γ are set to 10, 5, 5, two respectively.

Next, we analyze the impact of variations in these hyperparameters on model performance. To analyze the impact of λ 1 , λ 2 , λ 3 on fusion performance, we perform a search over λ 1 , λ 2 , λ 3 values in the ranges of 1–20, 1 to 10, and 1 to 10. The quantitative evaluation results for both fusion and downstream object detection are presented in Table 5. As shown in Table 5, the model achieves optimal performance on fusion when λ 1 = 10 , λ 2 = 5 , and λ 3 = 5 .

To verify the effectiveness of the hyperparameter γ , we fix λ 1 , λ 2 and λ 3 to 10, 5, 5 and analyze the model performance as γ varies from 1 to 5. As shown in Table 5, the model achieves the best fusion performance when γ is set to 2. Therefore, we set the hyperparameter γ to 2.

As shown in Table 6, a complexity evaluation is introduced to evaluate the efficiency of our method from three aspects, i.e., FLOPs, training parameters and runtime. Wherein, for FLOPs calculation, the size of the input images is standardized to 512 × 512 pixels. The inference time is calculated as the average time taken to process 18 scene images from RoadScene’s test dataset. From Table 6, our model performs the best in FLOPs, implying that our method has fast calculation speed and is application-friendly. The average inference time for our model to fuse two source images is 0.40 s, only a bit longer than the SOTA method, demonstrating that our model’s inference speed is relatively fast and acceptable. Besides, the parameter size of our model is only 0.97M, which can be easily deployed in practical applications. This indicates the efficiency of our method, which can serve practical vision tasks well with better visual performance.

To validate the generalization ability of our method, we conduct experiments under other scenarios. Fusion results are shown in Figure 8. From the qualitative results we can see that our proposed model performs perfectly under other scenarios.

The proposed method enables mutual enhancement between infrared-visible image fusion and object detection, specifically designed to handle misaligned source images, achieving better experimental results compared to other methods. However, our approach still has certain limitations. Specifically, since our model is trained on the generated unaligned dataset, where the deformations in real-world images cannot be fully included, failure cases appear under real-world scenarios. As shown in Figure 9, our method fails to handle deformations under real-world scenarios. Improving the robustness of our method is vital for future research.

To validate the effectiveness of the proposed method in the field of medical imaging, we conduct a comparative study on the publicly available BraTS2020 Menze et al. [55] dataset. Specifically, we first employ the state-of-the-art medical image registration method CorrMLP Meng et al. [56] to align the deformed MRI-T2 images to the reference MRI-T1 images, and subsequently apply several advanced fusion methods (including MATR Tang et al. [57], ALMFnet Mu et al. [58], EMMA Zhao et al. [54], BSAFus Li et al. [47], and RMRFus Zhang et al. [59]) for image fusion. As shown in Figure 10, the fusion images generated by the proposed method exhibit superior image quality and effectively correct artifacts and spatial deformations. In contrast, existing ”registration + fusion” methods often introduce noticeable artifacts when handling unregistered medical images, significantly degrading the visual quality of the fused images. Furthermore, as reported in Table 7, the quantitative analysis results further demonstrate the significant advantages of the proposed method in terms of fusion performance.