With the advent of the multimodal data fusion era, the capability of unimodal systems is no longer sufficient to handle complex real-world tasks. As a result, multimodal large language models (MLLMs) have been proposed to integrate information from multiple data sources, enabling more comprehensive and accurate representations. These models have demonstrated significant practical value across various domains, including natural language processing, vision tasks, and audio tasks. In the visual domain, MLLMs enhance the performance of tasks such as image classification, object detection, and image captioning by combining textual descriptions with visual instructions. For example, GPT-4V [31] and Gemini [32] integrate image content with natural language descriptions to produce more vivid and accurate annotations. NExT-GPT [33] and Sora [34] are at the forefront of multimodal video generation, producing rich and realistic content by learning from multimodal data. Moreover, VideoChat [35] and Video-LLaVA [36] demonstrate excellent capabilities in analyzing and understanding video content in intelligent video understanding scenarios.

In the field of image fusion, Text-IF [29] and MGFusion [37] uses CLIP [38] to encode user requirement texts, guiding the model to fuse images. TeRF [30] utilizes LLaMA [39] to encode user instruction texts and generate prompts for guiding image fusion across different tasks. Although these methods employ MLLMs to tackle some challenges in image fusion, they do not consider the specific requirements of high-level downstream visual tasks for image fusion quality, which limits the application of infrared and visible image fusion in such tasks.

Conventional infrared and visible image fusion methods mainly focus on designing sophisticated feature extraction networks and fusion strategies to ensure the quality of the fused results. From the perspective of network design, these methods can be broadly categorized into CNN-based methods, CNN-Transformer hybrid methods, and GAN-based methods. CNN-based methods [40–45] typically apply convolution, activation, and pooling operations to extract features from the input images, then fuse and reconstruct the final result using the extracted features. However, since CNNs can only perceive local features within a limited receptive field, they struggle to capture long-range contextual information, limiting their representational capacity. In contrast, Transformers [46] are better at modeling long-range dependencies and are more suited for capturing global features in images. ViT [47] was the first to introduce Transformer architectures into computer vision, achieving promising results. Subsequently, to combine the respective strengths of CNNs and Transformers, hybrid methods have gained increasing attention in the image fusion domain. For instance, CGTF [48], SwinFusion [16], YDTR [17], and DATFuse [49] insert Transformer layers after CNN layers to jointly leverage local and global feature extraction. CDDFuse [50] and EMMA [51] adopt dual-branch architectures combining CNNs and Transformers to simultaneously extract features from the input images and integrate them for fusion.

GAN-based methods enhance the model’s feature extraction capabilities by introducing adversarial learning between generators and discriminators. Depending on the number of discriminators used, these methods can be classified into single-discriminator and dual-discriminator approaches. Single-discriminator methods [2, 52] tend to favor one modality over the other, potentially leading to information loss and reduced visual quality of the fusion results. To address this, dual-discriminator methods [53–56] are proposed to preserve important features from both source images simultaneously.

However, all of these methods primarily focus on designing effective feature extraction networks to produce high-quality fusion features and images. They overlook how fusion quality impacts downstream task performance, and fail to consider the potential feedback from downstream tasks that could help guide fusion more effectively.

Pedestrian detection is a fundamental problem in computer vision with a wide range of applications. Cascade R-CNN [57] extends R-CNN [58] into a multi-stage framework, improving the ability to filter hard negative samples. Faster R-CNN [59] introduces a Region Proposal Network (RPN) that shares convolutional features with the detection network, making region proposals nearly cost-free. YOLO [60] reformulates object detection as a regression problem, allowing real-time inference directly on images through a convolutional neural network. SSD [61] uses multi-scale feature maps and predefined anchors for pedestrian detection, addressing YOLO’s limitations in detecting small objects. DETR [62] adopts a Transformer-based encoder-decoder architecture for object detection. BAS Wu et al. [63] learns to represent the whole foreground region by leveraging foreground guidance and domain constraints. CREAM [64] proposes a clustering-based method to enhance activation within target regions. Group R-CNN [65] builds instance groups to perform pedestrian detection from point annotations.

However, most pedestrian detection methods are designed for unimodal images, which often leads to degraded detection performance due to incomplete scene information. In this work, we perform pedestrian detection on fused infrared and visible images, and incorporate task-specific prompts generated by large language models. This not only improves the quality of the fused images but also enhances pedestrian detection performance.

As shown in Figure 2, the proposed method consists of two training stages. The first stage is dedicated to training the Fusion Network, enabling it to perform basic infrared and visible image fusion. In the second stage, the parameters of the pretrained fusion network are frozen, and a Text-Driven Feature Harmonization (Text-DFH) module is trained to refine the fusion results to better align with the requirements of pedestrian detection.The fusion network is composed of three main components: an Infrared Image Feature Encoder (IR-Encoder), a Visible Image Feature Encoder (VI-Encoder), and a Fusion Feature Decoder (F-Decoder). The IR/VI-Encoders are responsible for extracting features from the input infrared and visible images, respectively, while the F-Decoder reconstructs the fused image based on the combined features.The Text-DFH module adjusts the features extracted by the IR/VI-Encoders based on responses from a Multimodal Large Language Model (MLLM), ensuring that the resulting fused image better satisfies the needs of pedestrian detection. In this work, we adopt LLaVA [66] as the MLLM. LLaVA analyzes the unmodulated fused image and generates suggestions in response to user queries related to pedestrian detection tasks (e.g., To improve the accuracy of pedestrian detection, how can the quality of this image be enhanced?). More text examples of LLaVA answers are shown in Figure 3.

In the first training stage, we train the fusion network to perform the basic task of infrared and visible image fusion. The fusion network primarily consists of three components: the IR-Encoder, VI-Encoder, and F-Decoder. Each of the IR-Encoder, VI-Encoder, and F-Decoder is composed of three feature extraction layers. Each layer is constructed by stacking a convolutional layer (kernel size = 3 × 3 , stride = 1), a Batch Normalization layer, and a LeakyReLU activation function. It is worth noting that the LeakyReLU activation function in the final feature extraction layer of the F-Decoder is replaced with a Tanh activation function to facilitate image reconstruction. We input the infrared image I i and the visible image I v into the IR-Encoder and VI-Encoder, respectively, to extract features F i and F v . To reconstruct the fused image, we concatenate F i and F v along the channel dimension and feed the result into the F-Decoder, which generates the final fused image I f .

To encourage the fused image to retain as much scene information from the source images as possible, we introduce an intensity loss ℓ i n and an edge loss ℓ e d , which together form the fusion loss ℓ f : ℓ f = ℓ i n + ε ℓ e d , (1) Here, ε denotes a hyperparameter used to balance the contribution of each sub-loss term. The intensity loss ℓ i n is defined as: ℓ i n = 1 H W I f − I i 1 + I f − I v 1 , (2) The edge loss ℓ e d is defined as: ℓ e d = 1 H W ∇ I f − ∇ I i 1 + ∇ I f − ∇ I v 1 , (3) Here, H and W denote the height and width of the fused image, respectively; ⋅ 1 represents the l1-norm, and ∇ denotes the Sobel edge extraction operator.

In the second training stage, we freeze the parameters of the pretrained fusion network and focus on training the Text-DFH module to ensure that the fusion results meet the requirements of the pedestrian detection task. Text-DFH refines the features output by the IR/VI-Encoders in the fusion network based on the responses from the multimodal large language model, enabling the fused image to better align with the needs of pedestrian detection. As shown in Figure 4, Text-DFH mainly consists of a dual-branch Cross Attention (CA) module and three feature extraction layers. The dual-branch cross attention computes the cross-attention between the features extracted by the IR/VI-Encoders and the textual features, allowing the model to extract useful information from the text that can help improve pedestrian detection accuracy. Subsequently, the three feature extraction layers integrate this textual information with the image scene features to generate refined features. The structure of the CA module is similar to the Multi-Scale Attention (MSA) module used in DATFuse.

We input the infrared image I i and visible image I v into the pretrained fusion network with frozen parameters to obtain the fused image I f . To obtain effective textual feedback that helps ensure the fused image meets the requirements of the pedestrian detection task, we input both I f and the text prompt “To improve the accuracy of pedestrian detection, how can the quality of this image be enhanced?” into LLaVA, resulting in the textual feature T . We then input the outputs F i / v from the IR/VI-Encoders and the textual feature T into Text-DFH to harmonize the information in F i / v . To comprehensively extract the task-relevant information from the textual features, we design a dual-branch processing strategy. In the first branch, we take F i / v as the Query (Q) and T as the Key (K) and Value (V) for cross-attention computation: F i / v 1 = s o f t m a x Q i / v 1 K i / v 1 T d 1 V i / v 1 , (4) Here, F i / v 1 represents the features injected with textual information in the first branch, d 1 denotes the dimensionality of Q i / v 1 , Q i / v 1 = W i / v Q , 1 F i / v , K i / v 1 = W i / v K , 1 T , V i / v 1 = W i / v V , 1 T . In the second branch, we use T as the Query (Q) and F i / v as the Key (K) and Value (V) for cross-attention computation: F i / v 2 = s o f t m a x Q i / v 2 K i / v 2 T d 2 V i / v 2 , (5) Here, F i / v 2 represents the features injected with textual information in the second branch, d 2 denotes the dimensionality of Q i / v 2 , and Q i / v 2 = W i / v Q , 2 T , K i / v 2 = W i / v K , 2 F i / v , V i / v 2 = W i / v V , 2 F i / v . To comprehensively aggregate the textual information, we concatenate F i / v 1 and F i / v 2 along the channel dimension and feed the result into three feature extraction layers to obtain the harmonized features F ̂ i / v . We then concatenate F ̂ i and F ̂ v along the channel dimension and input the result into the F-Decoder to reconstruct the refined fused image I ′ f .

To ensure that the refined fused image I ′ f meets the requirements of the pedestrian detection task, we introduce a pretrained pedestrian detection network with frozen parameters to supervise the fused image. We input I ′ f into the detection network and obtain the pedestrian detection result y ̂ . To make y ̂ as close as possible to its ground truth y g t , we constrain the Text-DFH module using the loss function ℓ p d , which is the same as the one used during the training of YOLOv5.

The proposed method consists of two training stages. In both the first and second training stages, we train the fusion network and the text-driven feature harmonization module on the publicly available LLVIP dataset [67], respectively, in accordance with standard practices in the field [68–70]. Specifically, we randomly select 2,000 pairs of infrared and visible images from the LLVIP dataset as the training set. To enhance the diversity of training samples, we apply random flipping, random rotation, and random cropping as data augmentation techniques. For evaluation, we randomly select 200 pairs of infrared and visible images from each of the LLVIP, M 3 FD [71], and MSRS [3] datasets to form the test set, in order to assess both the fusion performance and pedestrian detection performance of the proposed method. Among them, LLVIP, M 3 FD, and MSRS are used to evaluate fusion performance, while LLVIP is specifically used to evaluate pedestrian detection performance.

The proposed method involves two training stages. In the first stage, the fusion network is trained. In the second stage, the parameters of the fusion network are frozen, and the text-driven feature harmonization module is trained. Both training stages use the Adam optimizer to update the network parameters, with a batch size of 16 and a learning rate of 1 × 1 0 − 3 . The total number of training epochs is set to 100 for the first stage and 200 for the second stage. In addition, the hyperparameter ε is set to 0.2. The proposed method is implemented based on the PyTorch framework and is trained on a single NVIDIA RTX A6000 GPU.

We adopt five commonly used objective evaluation metrics to quantitatively assess the fusion performance of the proposed method. These metrics include Edge Preservation Index ( Q A B / F ) [72, 73], Chen-Varshney Index ( Q C V ) [74], Structural Similarity Index ( Q SSIM ) [75], Average Gradient ( Q A G ) [76], and Sum of Correlations of Differences ( Q SCD ) [77]. Q A B / F measures how well edge information from the source images is preserved in the fused image. Q A B / F higher value indicates less loss of texture details in the fused image. Q C V evaluates fusion quality from the perspective of human visual perception; a lower value means the fused image aligns better with human visual preferences. Q SSIM quantifies the similarity between the fused image and the source images in terms of luminance, contrast, and structure. A higher value indicates less information difference between the fused and source images. Q A G measures the richness of gradient information in the fused image. A higher value means the fused image contains more detailed gradient content. Q SCD assesses information loss during the fusion process by computing difference maps between the fused image and source images. A higher value indicates less distortion in the fused image. Among these, Q A B / F , Q SSIM , Q A G and Q SCD are positive indicators, meaning a higher value indicates better fusion performance. Q C V is a negative indicator, meaning a lower value represents better fusion performance. In addition, to objectively evaluate the effectiveness of the fused images in the pedestrian detection task, we adopt three widely used metrics in the pedestrian detection domain for quantitative analysis: Mean Average Precision (mAP) at IoU threshold of 0.5 ( m A P 50 ) , mAP at IoU threshold of 0.75 ( m A P 75 ) , and the averaged mAP at IoU threshold from 0.5 to 0.95 ( m A P 50 → 95 ) .

In this study, we conduct a series of qualitative and quantitative comparisons between the proposed method and eight state-of-the-art (SOTA) methods to verify its superiority in both fusion performance and pedestrian detection performance. These methods include AUIF [78], DATFuse [49], IVFWSR [79], LRRNet [80], MLFusion [81], TIMFusion [82], SwinFusion [16], and TextIF [29]. The comparative experiments are divided into two distinct groups: In the first group, we compare the fusion performance of our method with that of the SOTA methods. In the second group, we freeze the fusion networks of the compared methods and retrain their pedestrian detection networks using the corresponding fused results. The retrained detection networks are then used to perform pedestrian detection on the fused images. This setup is designed to demonstrate that our proposed method can achieve strong pedestrian detection performance without requiring retraining of the detection network.

The proposed method mainly consists of two core components: the Multimodal Large Language Model (MLLM) and the Text-Driven Feature Harmonization (Text-DFH) module. Within Text-DFH, both the text-guided cross-attention and the image-guided cross-attention play key roles. To validate the effectiveness of these components, we conduct a series of ablation experiments on the LLVIP dataset.