Multispectral sensors can obtain paired Color-Thermal images to provide complementary information about pedestrian targets. A large multispectral pedestrian detection (KAIST) dataset was constructed by [8]. Meanwhile, by combining the traditional aggregated channel feature (ACF) pedestrian detector [23] with the HOG algorithm [24], an extended ACF (ACF + T + THOG) method was proposed to fuse Color and Thermal features. In 2016, [9] proposed four fusion modalities of low-layer feature, middle-layer feature, high-layer feature, and confidence fraction fusion with VGG16 as the backbone network, and the middle-layer feature fusion was proved to offer the maximum integration capability of Color and Thermal features. Inspired by this, [25] developed a multispectral region candidate network with Faster RCNN (Region with CNN features, RCNN) [26] as the architecture and replaced the original classifier in Faster RCNN with an enhanced decision tree classifier to reduce the missed and false detection of pedestrians. Recently,[27] deployed the EfficientDet as the backbone network and proposed an EfficientDet-based fusion framework for multispectral pedestrian detection to improve the detection accuracy of pedestrians in Color and Thermal images by adding and cascading the Color and Thermal features. Although the studies [8,9,25,27] fused Color and Thermal features for pedestrian detection, they mainly focused on exploring the impact of different stages of fusion on pedestrian detection, and only adopted simple feature fusion and not focusing on the case of pedestrian and background confusion.

In 2019, [28] observed a weak alignment problem of pedestrian position between Color and Thermal images, for which the KAIST dataset was re-annotated and Aligned Region CNN (AR-CNN) was proposed to handle weakly aligned multispectral pedestrian detection data in an end-to-end manner. But the deployment of this algorithm requires pairs of annotations, and the annotation of the dataset is a time-consuming and labor-intensive task, which makes the algorithm difficult to be applied in realistic scenes. [29] proposed a new single-stage multispectral pedestrian detection framework. This framework used multi-label learning to learn input state-aware features based on the state of the input image pair by assigning an individual label (if the pedestrian is visible in only one image of the image pair, the label vector is assigned as y ₁ ∈ [0, 1] or y ₂ ∈ [1, 0]; if the pedestrian is visible in both images of the image pair, the label vector is assigned as y ₃ ∈ [1, 1]) to solve the problem of weak alignment of pedestrian locations between Color and Thermal images, but the model still requires pairs of annotations during training. [19] designed illumination-aware networks to obtain illumination-measured parameters for Color and Thermal images separately and used them as the fusion weights for Color and Thermal features. [20] designed a differential modality perception fusion module to guide the features of the two modalities to become similar, and then used the illumination perception network to assign fusion weights to the Color and Thermal features. [30] reported an uncertainty-aware cross-modal guidance (UCG) module to guide the distribution of modal features with high prediction uncertainty to align with the distribution of modal features with low prediction uncertainty. The researches [19,20] noticed that the pedestrians in Color and Thermal images are easily confused with the background and used illumination-aware networks to assign fusion weights to Color and Thermal features. However, the acquisition of illumination-measured parameters relied heavily on the classification scores, whose accuracy was limited by the performance of the classifier. In contrast, the method proposed in this paper not only considers the confusion of pedestrians and background in Color and Thermal images but also effectively fuses the two modal features.

Attention mechanisms [31] utilized in computer vision are aimed to perform the processing of visual information. Currently, attention mechanisms have been widely used in semantic segmentation [32], image captioning [33], image fusion [34,35], image dehazing [36], saliency target detection [37], person re-identification [38–40], etc. [41] introduced the idea of a squeeze and excitation network (SENet) to simulate the interdependence between feature channels in order to generate channel attention to recalibrate the feature mapping of channel directions. [42] employed the use of a selective kernel unit (SKNet) to adaptively fuse branches with different kernel sizes based on input information. A work inspired by this was from [43]. They designed a multi-scale channel attention feature fusion network that used channel attention mechanisms to replace simple fusion operations such as feature cascades or summations in feature fusion to produce richer feature representations. However, this recent progress in multispectral pedestrian detection has also been limited to two main challenges the interference caused by background and the difference of fundamental characteristics in Color and Thermal images. Therefore, we propose a multispectral pedestrian detection algorithm with cascaded information enhancement and cross-modal attention feature fusion based on the attention mechanism.

Considering the confusion of pedestrians with the backgrounds in Color and Thermal images, we design a cascaded information enhancement module (CIEM) to augment the pedestrian features of both modalities to mitigate the effect of background interference on pedestrian detection. Specifically, a cascaded feature fusion block is used to fuse the Color features F _c and Thermal features F _t . The cascaded feature fusion block consists of feature cascade, 1 × 1 convolution, 3 × 3 convolution, BN layer, and ReLu activation function. The feature cascade operation splice F _c and F _t along the direction of channels. 1 × 1 convolution is conducive to cross-channel feature interaction in the channel dimension and reducing the number of channels in the splice feature map, while 3 × 3 convolution expands the field of perception and makes a more comprehensive fusion of features for generating fusion features F _ct : F c t = R e L u B N C o n v 3 C o n v 1 F c , F t (1) where BN denotes batch normalization, C o n v n ⋅ denotes a convolution kernel with kernel size n × n, [⋅, ⋅] denotes the cascade of features along the channel direction, ReLu(⋅) represents ReLu activation function. Fusion feature F _ct is used to enhance the single-modal information because F _ct combines the consistency and complementarity of the Color features F _c and Thermal features F _t . The use of F _ct for enhancing the single-modal feature can reduce the interference of the noise in the single-modal features (for example, it is difficult to distinguish between the pedestrian information and the background noise).

In order to effectively enhance pedestrian features, the fusion feature F _ct is sent into the channel attention module (CAM) and spatial attention module (PAM) [45] to make the network pay attention to pedestrian features. The network structure of CAM and PAM is shown in Figure 3. F _ct first learns the channel attention weight w _ca ∈ R^1×1×C by CAM, then uses w _ca to weight F _ct , and the spatial attention weight w _pa ∈ R ^W×H×1 is obtained from the weighted features by PAM.

The single-modal Color features F _c and Thermal features F _t are multiplied element by element with the attention weights w _ca and w _pa to enhance the single-modal features from the perspective of fused features. The whole process can be described as follows: F t ′ = F t ⊗ w c a ⊗ w p a (2) F c ′ = F c ⊗ w c a ⊗ w p a (3) where F t ′ and F c ′ denote the Color features and Thermal features obtained by the cascaded information enhancement module, respectively. ⊗ represents the element by element multiplication.

There is an essential difference between Color and Thermal images, Color images reflect the color and texture detail information of pedestrians while Thermal images contain the temperature information of pedestrians, however, they also have some complementary information. In order to explore the complementary features of different image modalities and fuse them effectively, we design a cross-modal attention feature fusion module.

Specifically, the Color features F c ′ and Thermal features F t ′ enhanced by CIEM are first mapped into feature vectors v _c ∈ R^1×1×C and v _t ∈ R^1×1×C , respectively, by using global average pooling operation. The cross-modal attention network consists of a set of symmetric 1 × 1 convolutions, ReLu activation functions, and Sigmoid activation functions. In order to obtain the complementary features of the two modalities, more pedestrian features need to be mined from the single-modal. The feature vectors v _t and v _c are learned to the respective modal attention weights w _t ∈ R^1×1×C and w _c ∈ R^1×1×C by a cross-modal attention network, and then the Color features F c ′ are multiplied element by element with the attention weights w _t of the Thermal modality, and the Thermal features F t ′ are multiplied element by element with the attention weights w _c of the Color modality to complement the features of the other modality into the present modality. The specific process can be expressed as follows. w t = S i g m o i d R e L u C o n v 1 G A P F t ′ (4) F c t ′ = w t ⊗ G A P F c ′ (5) w c = S i g m o i d R e L u C o n v 1 G A P F c ′ (6) F t c ′ = w c ⊗ G A P F t ′ (7) where F c t ′ denotes Color features after supplementation with Thermal features, F t c ′ denotes Thermal features after supplementation with Color features, GAP(⋅) denotes global average pooling operation, Conv ₁(⋅) denotes convolution with convolution kernel size 1 × 1, ReLu(⋅) denotes ReLu activation operation, and Sigmoid (⋅) denotes Sigmoid activation operation.

In order to efficiently fuse the two modal features, the features F c t ′ and F t c ′ are subjected to an element by element addition operation to obtain a global feature vector containing Color and Thermal features. Then, the features F t ′ and F c ′ are added element by element and multiplied with the attention weight w _ct of the global feature vector element by element to guide the fusion of Color and Thermal features from the perspective of global features to obtain the final fused feature F . The fused feature F is input to the detection head to obtain the pedestrian detection results. The feature fusion process can be expressed as follows: w c t = S i g m o i d R e L u C o n v 1 F c t ′ ⊕ F t c ′ (8) F = w c t ⊗ F t ′ ⊕ F c ′ (9) where ⊕ denotes element by element addition.

The loss function in this paper is consistent with the literature [26] and uses the Region Proposal Network (RPN) loss function L _RPN and Fast RCNN [46] loss function L _FR to jointly optimize the network: L = L R P N + L F R (10)

Both L _RPN and L _FR consist of classification loss L _cls and bounding box regression loss L _reg : L p i , t i = 1 N c l s ∑ i L c l s p i , p i ∗ + λ 1 N reg  ∑ i p i ∗ L r e g t i , t i ∗ (11) Where, N _cls is the number of anchors, N _reg is the sum of positive and negative sample number, p _i is the probability that the i-th anchor is predicted to be the target, p i ∗ is 1 when the anchor is a positive sample, otherwise it is 0, t _i denotes the bounding box regression parameter predicting the i-th anchor, and t i ∗ denotes the GT bounding box parameter of the i-th anchor, λ = 1.

The difference between the classification loss of RPN network and Fast RCNN network is that the RPN network focuses only on the foreground and background when classifying, so its loss is a binary cross-entropy loss, while the Fast RCNN classification is focused to the target category and is a multi-category cross-entropy loss: L c l s p i , p i ∗ = − log p i ∗ p i + 1 − p i ∗ 1 − p i (12)

The bounding box regression loss of RPN network and Fast RCNN network uses Smooth  L 1 loss: L reg  t i , t i ∗ = R t i − t i ∗ (13) Where, R denotes Smooth  L 1 function,  Smooth  L 1 x = x 2 2 σ 2  if  | x | < 1 σ 2 | x | − 0.5  otherwise  (14)

The difference between the bounding box regression loss of RPN loss and the regression loss of Fast RCNN loss is that the RPN network is trained when σ = 3 and the Fast RCNN network is trained when σ = 1.

This paper evaluates the algorithm performance on the KAIST pedestrian dataset [8], which is composed of 95,328 pairs of Color and Thermal images captured during daytime and nighttime. It is the most widely used multispectral pedestrian detection dataset at present. The dataset is labeled with four categories including person, people, person?, and cyclist. Considering the application areas of multispectral pedestrian detection (e.g., automatic driving), all four categories are treated as positive examples for detection in this paper. To address the problem of the annotation errors and missing annotations in the original annotation of the KAIST dataset, studies [9,28,47] performed data cleaning and re-annotation of the original data. Given that the annotations used in various studies are not consistent, we use 7601 pairs of Color and Thermal images from synthetic annotation (SA) [47] and 8892 pairs of Color and Thermal images from paired annotation (PA) [28] for model training. The test set consists of 2252 pairs of Color and Thermal images, of which 1455 pairs are from the daytime and 797 pairs are from the nighttime. For a fair comparison with other methods, the test experiments were performed according to the reasonable settings proposed in the literature [8].

In this paper, Log-average Miss Rate (MR) proposed by [48] is employed as an evaluation index and combined with the plotting of the Miss Rate-FPPI curve to assess the effectiveness of the algorithm. The horizontal coordinate of the Miss Rate-FPPI curve indicates the average number of False Positives Per Image (FPPI), and the vertical coordinate represents the Miss Rate (MR), which is expressed as:  MissRate  = F N T P + F N (15) F P P I = F P  Total   images  (16) where FN denotes False Negative, TP denotes True Positive, FP denotes False Positive, the sum of TP and FN is the number of all positive samples, and Total (images) denotes the total number of predicted images. It is worth noting that the lower the Miss Rate-FPPI curve trend, the better the detection performance; the smaller the MR value, the better the detection performance. In order to calculate MR, in logarithmic space, nine points are taken from the horizontal coordinate (limited value range is 1 0 − 2 , 1 0 0 ) of Miss Rate-FPPI curve, and then there are nine corresponding vertical coordinates m ₁, m ₂,…m ₉. By averaging these values, MR can be obtained as follows: M R = exp 1 n ∑ i = 1 n ln m i (17) where n is 9.

In this paper, the deep learning framework pytorch1.7 is adopted. The experimental platform is the ubuntu18.04 operating system and a single NVIDIA GeForce RTX 2080Ti GPU. Stochastic Gradient Descent (SGD) algorithm is used to optimize the network during model training, with momentum value of 0.9, weight attenuation value 5 × 10^–4, and initial learning rate is 1 × 10^–3. The model is iterated for five epochs with the batch size of 4, and the learning rate decay to 1 × 10^–4 after the 3rd epoch.