The proposed module generally includes the following four core steps: First, for an input feature map processed by DRF, we divide the relative distance corresponding to different receptive field regions into N different discrete levels and assign a weight that is learned by the neural network to each level. Second, for each feature point, we calculate the relative distance between the feature point and its surrounding feature points, divide the calculated relative distance into N different levels, and then assign the corresponding learned weight in step 1 to each level. By this operation, we obtain a weight matrix encoding the distance relation information of all other feature points related to this target feature point. Third, we multiply the weight matrix of the feature point obtained in step 2 by the feature map to obtain the excitation output of this target feature point. For an input feature map, we finally obtain the global guidance response of each feature point based on the relative positional relationship. Fourth, to preserve the local information in the excitation output, the feature map is input to a convolutional layer with a fixed convolution kernel to obtain the local response output of the feature map. Then, we add the local response output of each feature point and its global guidance response obtained in step 3 together to obtain the final output feature. The mathematical details for each step are as follows:

Taking a target feature point f(x_target, y_target) on the feature map F as an example, (x_target, y_target) represents the spatial position of the target feature point. f(x_other, y_other) represents other feature points surrounding the target feature point on the feature map F, and (x_other, y_other) represents the spatial position of other feature points. The relative spatial distance between the target feature point and the surrounding feature points is defined as follows:

K and H, respectively, represent the length and width of the feature map F, and the maximum value of the relative distance calculated according to Equation (1) is 2, due to that we calculate the normalized distance. We divide the maximum normalized relative distance into N discrete levels and assign each level a weight value ω, which will be learned by the gradient back-propagation algorithm. Hence, different learned weight values form a vector ω¯∈RN.

For the feature point at each spatial position in the feature map F, we calculate the relative distance between the target feature point and other surrounding feature points according to Equation (1) and map the calculated relative distance dist(f_target, f_other) into N different levels. The weight value assigned to each level is calculated as follows:

According to Equation (2), we assign the weight value ω_k in the corresponding weight vector ω¯∈RN to all feature points on this level, so as to obtain a weight coefficient matrix P∈R^HK×HK of each feature point on the entire feature map relative to its surrounding feature points. Several examples of visualized weight maps learned from the dataset are shown in Figure 1C. Finally, the weight coefficient matrix is normalized point-to-point by a Softmax function as follows to obtain a weight coefficient matrix S∈R^HK×HK within the values of [0, 1].

For an input feature map F^C×H×K, where C denotes the number of channels, we first pass the feature map into a convolutional layer W_v with the size of 1 × 1.

Then, for each feature point in the feature map, we calculated the product of the feature map V and the weight coefficient matrix S, so as to obtain the excitation output of the feature point after the weighted average of the calculated feature map V and the weight coefficient matrix S.

We perform the operations of the above Equations (1)–(5) on each feature point in the feature map F^C×H×K, and finally obtain the output of the entire feature map O^C×H×K. Finally, O^C×H×K is further passed into a 1 × 1 convolutional layer of W_v to get the global response output of the feature map G = O⊗W_v.

To further retain the local context information, we pass the input feature map F^C×H×K into a convolutional layer W_l, with a fixed convolution kernel with the size of 3 × 3 to obtain the local response L = F⊗W_l. The final output of the feature map after being processed by the convolutional neural network module based on the biological visual dynamic receptive field mechanism described in this study is the addition of the global response output G with the local response L. Figure 2 shows the details of biologically inspired DRF module. In practical computation, to reduce computation and memory consumption of the proposed DRF module, first, we designed our DRF model as a bottleneck-like architecture by reducing the number of feature channels through a convolution with kernel size 1, and then feeding the output into the DRF module and restoring the number of feature channels through another convolutional layer.

Inspired by multi-head self-attention, we also devised a multi-head DRF module as shown in Figure 3. Concretely, we increased the learned weight vector ω¯∈RN by one more dimension ω¯∈RN×d, and d indicates the head-dim. Hence, we get the multi-head weight coefficient matrix S∈R^d×HK×HK and the multi-head feature map VCd×d×HK. The final results of multi-head DRF are obtained by multiplying S∈R^d×HK×HK with VCd×d×HK using the regular matrix manipulation as shown in Figure 3. The proposed multi-head design allows the DRF module to extract feature information from multiple subspaces and facilitate the representation capacity of features. The total computational cost of the multi-head DRF is similar to that of a single-head DRF.

As shown in Figure 2, we adopted the Feature Pyramid Network (FPN) as the detection network and use FPN levels from 2 to 5 for feature extraction. We transformed the output of FPN levels C₂, C₃, C₄, and C₄ to the new feature maps R₂, R₃, R₄, and R₅ using one convolutional layer with kernel size 1. The feature maps R₃, R₄, and R₅ are upsampled using the nearest neighbor by a factor of 2, and then the upsampled R₃, R₄, and R₄ are merged with the corresponding original feature maps C₃, C₄, and C₄ through element-wise addition. Finally, we feed the summation result into another 1 × 1 convolutional layer to get the final detection results. To reduce the number of parameters, we set the group number as 8 for each convolutional layer.

Experiments show that the grouped convolution does not reduce the final detection effect while reducing the number of parameters sufficiently. This process is described by Figure 2. The output of each feature map point is a one-dimensional vector whose dimension is A×(NC+m), A represents the number of anchors, NC refers to the number of categories of the training data, and m indicates the number of confidence value and the transformation of the width and height relative to the corresponding anchor. To evaluate the performance of the proposed DRF module, we incorporate the DRF module into each stage of the backbone as shown in Figure 2, where the structure of a specific module is amplified for better visualization.

The loss function used in our model includes three parts as follows:

(1) We adopt the conventional Binary Cross Entropy (BCE) as our first classification loss function. When calculating the classification loss function, only the loss of positive samples is considered. The positive samples are determined by the proposed anchor matching strategy, which will be described later in detail. In the following formula, y is the classification output vector, and y′ is the target one-hot label, which has been smoothed (Müller et al., 2019).

(2) Focal Loss (Lin et al., 2017b) was usually designed to down-weight easy examples in object detection, and thus focus training on hard negatives. Hence, we also use the focal loss to deal with the imbalance of difficult and easy samples.

Where p is the confidence output, and p′ is the target confidence output. We set the confidence value to 1.0 corresponding to the positive sample, and 0.0 corresponding to the negative sample. The focal factor γ was set as 2.0 in our experiment to adjust the importance of easy/hard examples.

(3) Generally, L1 or L2 norm is used to regress the four coordinate points of the predicted box and use IOU (Intersection over Union) to evaluate the accuracy of the prediction. However, both measures are not completely positively correlated. Many variants of IOU loss have been proposed (Yu et al., 2016; Rezatofighi et al., 2019; Zheng et al., 2020) to address this issue. Here, we use CIOU loss (Zheng et al., 2020) as the final regression loss function.

Where b and b^gt, respectively, indicates the predicted coordinates of the box and the corresponding ground truth. In summary, our loss function can be expressed as following.

1^obj is the positive sample mask and 1^noobj is the negative sample mask, which is determined by our matching strategy.

We benchmarked the max-IOU assigner strategy with some improvements. First, for each ground truth, we calculate the IOU with all anchors and select the top N with the largest IOU as the candidates. Then, referring to ATSS (Zhang et al., 2020), we calculate the L2 distance between the center points of the candidate anchors and the center point of the ground truth and select the first k anchors with the smallest distance as the further candidates. Finally, to make the anchors match the ground truth more, we remove the anchors whose center point is located outside the ground truth box. In particular, when the same anchor is accidentally selected by multiple ground truths, we assign it to the ground truth with the largest IOU.

In addition, we found through experiments that in a few cases, some ground truth cannot match any anchors, which is extremely unfavorable for target detection. To avoid such a situation, we additionally take out the anchor, which has the largest IOU with each ground truth and make it responsible for predicting the position offset. In this way, ground truth can match up to k anchors. In our experiment, we set N to 10 and k to 5 to achieve a relatively good test effect. The proposed matching strategy process is shown in Algorithm 1.

Figure 4 shows the comparison of the number of anchors matches between our proposed matching strategy (Figures 4A, C) and the common max-IOU matching strategy (Ren et al., 2015; Liu et al., 2016) (Figures 4B, D) on two benchmark datasets. It can be seen that the number of anchors matched by our method is more balanced. The red area indicates that the ground truth does not match any anchors at all. This part has a huge impact on the detection. Hence, the matching strategy needs to avoid the situation where the ground truth cannot completely match the anchor as much as possible. Moreover, the closer the distance between the ground truth and the matching anchor, the greater the intersection of IOU, and the better and faster return of the target location information by matching the anchor during training. Subsequent experiments are shown in Table 5 also show that our proposed matching strategy significantly improves the detection results of small targets compared with the traditional IOU-max method (Ren et al., 2015; Liu et al., 2016).

Our experimental hardware platform is NVIDIA TI 2080 GPU and Intel I7-9700 CPU with Cuda 11.1 and Cudnn 7.65. Our network is based on pytorch 1.4.0, and the weight of the convolutional layers adopts a normal distribution initialization with a mean value of 0.0 and a standard deviation of 0.1, all bias is set to be a constant of 0.0. The weight of the regularized BatchNorm (Ioffe and Szegedy, 2015; Santurkar et al., 2018) layers is designed to be constant 1.0, and the deviation is 0.0.

In this section, we build a target detection framework to verify and analyze the effectiveness of our proposed DRF/M-DRF module. The complete network architecture is shown in Figure 2. The FPN structure and loss design have been detailed in Sections 3.3, 3.4, respectively. We use the SGD optimizer with momentum of 0.9 and weight decay of 0.0005 to train our network, and the learning rate decay strategy uses a cosine decay scheduler (Takahashi et al., 2019), with a warmup of two epochs in the beginning (Goyal et al., 2017). Specifically, our total training epoch number is 30, the initial learning rate is set as 0.01, and the end learning rate is 0.0001.

We quantitatively compare our method with several SOTA methods including SSD (Liu et al., 2016), Faster RCNN (Ren et al., 2015), RetinaNet (Lin et al., 2017b), YOLO v3 (Redmon and Farhadi, 2018), and YOLO v5 (Jocher et al., 2020). The main information involved in the comparison includes the input size of image, the size of parameters, the flops of the model, and the test results measured with mAP for each class of the experimental datasets. For a fair comparison with SOTA backbones, we also report the results of replacing only the backbone on our detector framework alone. Specifically, we compared the classic Resnet50, Resnet101 and ResneXt50, and ResneXt101 backbones of different depths. Among these, for the Resnet backbone, we directly replace the bottleneck with our proposed M-DRF. For the ResneXt backbone, we replace the 3 × 3 convolution in M-DRF for generating the local response with the same group convolution as in ResneXt.

For Resnet50 and ResneXt50, the replacement position of M-DRF is the penultimate residual module of the last stage. For Resnet101 and ResneXt101, we replaced three residual modules, which are the 5th, 11th, and 17th residual modules of stage3, respectively. Tables 6, 7 show that our proposed approach outperforms several recent SOTA approaches on two benchmark datasets under the same or even lower parameters and flops. Figures 5, 6, respectively, show several example detection results on the two datasets. For experimental comparison in Figures 5, 6, we use the proposed object detection framework that is generally similar to Yolo v3 but with some key differences. Concretely, we adopt the Feature Pyramid Network (FPN) as the feature extraction network and use FPN levels from 2 to 5 for feature extraction. We use Renet50 as the backbone and incorporate the proposed DRF into the backbone for experimental comparison. Furthermore, we also adopt the different loss function and matching strategy as shown in previous sections compared with the original Yolo v3. Our proposed model incorporated into the DRF module is very effective in detecting small targets compared with the one without the DRF module in which many small and difficult targets are missed.

We also design an experiment to compare the deformable convolutional networks (DCN) (Dai et al., 2017; Zhu et al., 2019) with the proposed DRF. DCN achieves the deformable receptive field by adding a two-dimensional position offset to each sampling point position of the standard convolution kernel, and learning this offset from the dataset. DCN enables the trained convolution kernel with variable shapes compared with the standard convolution kernel with a fixed shape. However, DCN is still a local operation because it only exploits the local information of feature points offsetting with the target feature point. As a comparison, the proposed DRF learns different weight values for different receptive field positions in a global scope, and the learned receptive field is larger and can be adjusted dynamically. We can see from Table 8 that our proposed method with DRF clearly performs better than both DCN v1 (Dai et al., 2017) and DCN v2 (Zhu et al., 2019), especially for small targets such as a person on the airport dataset.

We finally evaluate the proposed DRF on CIFAR-10 and CIFAR-100 datasets for image recognition. The classification network has an architecture consisting of a single convolutional layer, followed by three stages, which have three residual blocks for each stage. We conduct experiments using ResNeXt-29(16 × 32d), ResNeXt-29(16 × 16d), SKNet, SE-ResNeXt-29(16 × 16d),ResNeXt-29(16 × 16d)+DRF, and SE-ResNeXt-29(16 × 16d)+DRF with last three residual blocks that are replaced by the proposed M-DRF module for comparison. We train our models for 100 epochs in total, using Adam with a mini-batch size of 128 and a weight decay of 1e-5. The initial learning rate is set to 0.001 and decreased by a factor of 10 every 30 epochs. All experiments shown in Table 9 demonstrate that our proposed network building on the DRF can also significantly improve the recognition performance, which proves that our network has a strong ability to extract target features and good portability to be integrated into various tasks (e.g., object detection and image recognition).