Semantic segmentation is an image classification task at the pixel level [12]. proposed a Full Convolution Network (FCN), which is a basic paradigm of semantic segmentation methods. FCN pioneers an end-to-end approach to achieve pixel-by-pixel classifications. In the encoder of FCN, a series of convolutional layers and successive down-sampling ones are often used to extract deep features with large receptive fields, and then the decoder upsamples the extracted deep features to the same resolution as the input. To lessen the loss of spatial information caused by down-sampling, skip connections are used to fuse low-level features with high-level ones from the different scales of encoders and decoders. Two fundamental paradigms have emerged as a result of further researches, including symmetric codec architectures [7, 13] and asymmetric codec ones [8, 14–17]. These symmetric codec structures mainly focus on minimizing the information loss caused by frequent down-sampling for expanding receptive fields. Meanwhile, asymmetric codec structures revolve the problem of extracting the most abstract semantics without reducing the feature map resolutions too much. Thus, a balance of spatial and semantic information can be achieved.

Traditional smoke segmentation methods make an effort to separate smoke targets from images by extracting the color features of images in different color spaces [18]. Combined color features and shape features to present a fast smoke detection method for video surveillance [19]. Used background removal methods and color features to filter non-smoke pixels [20]. Used the rough set theory to extract candidate smoke regions for video fire detection.

With the rapid development of deep learning in recent years, there are many deep neural networks that have also been proposed for smoke semantic segmentation. Without the need for designing complex hand-crafted features, deep learning based semantic segmentation approaches combine feature extraction and classification for implementing an end-to-end manner [21]. directly used the AlexNet network [22] for smoke recognition [23]. proposed a 3D parallel FCN model to segment smoke regions from videos [24]. proposed a coding and decoding network by designing a dual-path structure to obtain detailed information and semantic information for smoke segmentation [25]. proposed a Wave-shaped deep neural Network (W-Net) for smoke density estimation, which is factually a regression over each pixel [26]. proposed a global smoke attention network that makes a full use of the modeling capabilities of attention mechanism.

Traditional methods rely on manual features, while recent networks only focus on the performance of CNN itself and cannot perform smoke segmentation well. We focus more on the correlation between smoke boundaries and pixels, and complete smoke segmentation by improving the segmentation ability of boundaries and strengthening global modeling capabilities.

We choose the ResNet50 network [27] as our backbone network because it can obtain rich information. As shown in Figure 1, the ResNet50 [27] is used as the backbone of our method to extract features, and the backbone network is divided into four stages. Atrous convolutions [30] increase the receptive field and extract the abundant features of images without significantly reducing spatial resolutions, so we adopt atrous convolutions to compensate for the reduction of spatial resolutions due to down-sampling. The outputs of Stages 2, 3, and 4 greatly improve the semantic representation of deep features after feature fusion and multi-scale context extraction. The shallow features from Stage 1 are first delivered to the proposed Boundary Enhancement Module (BEM) for pixel alignment between different feature maps of different layers. Then the proposed Pixel Alignment Module (PAM) accepts the warping information of the BEM module for information fusion. Thus, we implement pixel alignment for different features. Finally, the fused structure map is upsampled to the original map size for generating the final segmentation map.

In deep neural networks, the size of receptive fields can roughly represent the degree of using contextual information. High-level contextual information can be captured by explicitly fusing features of objects at different scales, and it can effectively solve the problem of pixel inconsistency within the objects in the segmentation task and reinforce deep semantics.

To obtain more contextual information, we use the Pyramid Pooling Module (PPM) [11] to obtain the context information of objects, as shown in Figure 2. The PPM module contains global and local information using the adaptive global pooling with the different output sizes of 1, 2, 3, and 6. Then, the pooled feature maps are filtered using a 1 × 1 convolution. Next, bilinear interpolation is used to up-sample these filtered feature maps to the same size as the original input of PPM. The resized feature maps are concatenated together. Three layers of convolution (Conv), batch normalization (BN), and activation (ReLU) are used to obtain a feature map with a large amount of contextual information.

The embedding of spatial contexts can emphasize the most informative parts and enable the network to selectively focus on more important features. The proposal of attention mechanism offers a new direction for extracting powerful spatial and contextual information. Most spatial attention methods adopt matrix operations to capture the relationship between any two pixels in the global scope. To obtain spatial attention maps and improve object boundary localization accuracy, we propose a Boundary Enhancement Module (BEM). As shown in Figure 3, the input feature map of BEM is fed into the three branches for extracting features. H, W and C denote the height, width and channel numbers of the feature map, respectively. Average pooling captures the low frequency components of features, maximum pooling is able to extract the high frequency signals, and a 1 × 1 convolution learns the features about objects. As a result, we obtain three two-dimensional attentional feature maps, which are fused by point-wise summation. The fused feature map is activated by the sigmoid function for generating a spatial attention map. Finally, the input to our BEM is weighted with the spatial attention feature map to produce the feature map of the adaptive refinement boundary. Conv1x1 is a convolutional kernel with a kernel of 1.

To aggregate multi-scale features, feature fusion is often performed in different levels of feature maps, but pixel positions corresponding to their objects are different. To solve this problem, we propose a Pixel Alignment Module (PAM). This module first calculates the pixel offsets for pixel alignment, and the feature maps are pixel-aligned for effective fusion. Figure 4 shows the proposed PAM, which accomplishes pixel alignment by obtaining the pixel offset field. The value of each pixel in the offset field can be viewed as a moving distance for the pixel. In other words, the offset field can also be called a motion field. The relationship between a feature map and another one obtained by convolutions can be reconstructed by the pixel motion field. Thus, we actually obtain translation invariance by convolutions.

A bilinear interpolation layer is used to up-sample the feature map F ₁ to produce a feature map F ₇, which has the same size as the feature map F ₂, and feature maps F ₇ and F ₂ are concatenated to generate a feature map F ₃ for channel fusion. A set of depth-wise separable convolutions (DW-Conv) are used to establish the positional relationships between pixels on different feature maps. The pixel motion field F 4 ∈ R H × W × 2 is then generated using a 3 × 3 convolution in a similar way to DCN [28]. The field map F ₄ contains the spatial translation offsets along x and y-axes, and the feature values at each pixel position ρ l on F ₄ are used to move the pixel of F ₁ to a new position, resulting in a warped feature map F 5 ∈ R H × W × 256 , formulated as: F 5 ρ l = ∑ ρ ∈ δ ρ l ω ρ F 4 ρ l (1) where ω ρ denotes the weight of the bilinear kernel on the curved space grid, which is calculated by F ₄, and δ ρ l denotes the adjacent position of pixel position ρ l .

The warped feature map F ₅ is generated by F ₁ and F ₄, and F ₄ is produced from F ₁ and F ₂, so F ₅ certainly has a strong relationship with F ₂. Therefore, we concatenate F ₅ and F ₂ along the channel axis for further feature enhancement. Finally, the concatenated feature map is processed by a 3 × 3 convolution without BN and ReLU layers for feature fusion and dimensionality control to obtain the final output F ₆.

In this paper, the datasets used for training and testing are the same as those in [25], including a virtual synthetic smoke training set with 70,632 images and three virtual synthetic smoke test sets. The three test sets are respectively named DS01, DS02 and DS03, and each set has 1,000 images. The synthetic dataset was created from 8,162 pure smoke images [25]. adopted computer graphics to generate these pure smoke images with a variety of transparency, texture and fluid properties. Each pure smoke sample is an RGBA image with a spatial resolution of 256 × 256, containing RGB color channels S and a opacity channel α, respectively. The opacity α is limited in the range [0, 1]. According to the rules of linear composition, the combination of a pure smoke image S and a background B generates an observation image I . This procedure can be mathematically defined as: I = α ⋅ S + 1 − α ⋅ B (2)

Using the above method, we are able to construct a large number of training sets without the tedious labeling. Each virtual smoke image is randomly and linearly combined with a real background image for generating a virtual smoke training dataset. The generated virtual dataset is diverse in terms of colors, sizes, textures of smoke and background for simulating most real smoke scenes.

All the experiments were performed on a windows 10 PC with an NVIDIA RTX3090 GPU, and the programming environment is the python 3.7 and pytorch 1.7 framework. The Stochastic Gradient Descent (SGD) optimizer was used for training. The learning rate is set to 0.0001, the momentum is set to 0.9, and the learning rate decay is set to 0.95 using step decay. The mean Intersection over Union (mIoU) is used as the segmentation metric. mIoU is widely used to evaluate the overall performance of semantic segmentation algorithms and reflects the degree of overlap between the predicted results and their corresponding true labels. mIoU is formulated as: mIoU = 1 N ∑ k = 1 N P k ∩ G k P k ∪ G k (3) where P _k and G _k are the predicted result of the kth image and the corresponding true label, respectively.

To evaluate the effectiveness of the proposed network, we tested it on three synthetic test datasets and one real smoke dataset, and compared it with several state-of-the-art semantic segmentation methods based on deep learning, including FCN-8S [12], SegNet [13], SMD [29], TBFCN [9], Deeplab v1 [15], ESPNet [31], LRN [32], DSS [24],HG-Net [33], MS-Net [34], W-Net [25], and GSANet [26]. In addition, we also compared it with some Transformer structures ViT [35], Swin-Transformer [36] and SegFormer [37]. To objectively and fairly evaluate the performance of each method, we used the same dataset and experimental configurations to train all the compared methods, and the results are shown in Table 1.

The mIoU metrics achieved by our method on the three virtual smoke test datasets are 78.61%, 77.63% and 77.30%, respectively. Our method achieves the good mIoUs among all the existing methods second only to the SegFormer.

The visualized segmentation results on virtual smoke images are shown in Figure 5, where the first and second columns are the original and labeled images, respectively. According to Table 1 and Figure 5, the models with the mIoU below 70 obtain poor performance, while DSS, W-Net, and GSANet, as specially designed smoke segmentation models, have good performance. However, compared to our method, they are still slightly worse, both in terms of evaluation metrics and visual image quality. Our network has more distinct boundaries that are basically consistent with the original image. Compared with the latest transformer structures in recent years, our model also has good performance, only slightly worse than Segformer.

The segmentation results on the real smoke images are essentially as good as those on the synthetic smoke images. As shown in Figure 6, the predicted results by our BEPA-Net are visually similar to their corresponding real images. To accurately locate smoke edges, feature maps require spatial details, local and global semantic abstractions for delineating smoke. Our BEPA-Net model is proposed to solve these problems. The reasons may be that fusing multi-scale features can easily extract global and local information for better smoke representations.

As shown in Figures 6H, I, the generalization of HG-Net is poor. Although it has achieved certain results on virtual datasets, the performance on real images is poor. The reason may be the lack of skip connections to complete the fusion of deep and shallow features. The segmentation area obtained by our method is basically consistent with the real smoke area. In addition, by comparing the visualized results of virtual smoke datasets and real data, we find that Transformers obtain good results on virtual data, but the results on real data were very poor, as shown in Figures 6L–N. Hence, it may be overfitting.

Compared with DSS, W-Net, and GSANet, our method uses multi-scale fusion and skip connections, resulting in better performance than them. This is because the pixel alignment is performed during feature fusion. This technique greatly improves model performance. We also compared the fusion methods in ablation experiments.

Commonly used methods for feature fusion are pixel-wise addition (Addition) and channel concatenation (Concat). In this paper, we use the technique of pixel alignment followed by feature fusion to bridge the semantic gap between different feature maps during channel fusion. Further comparison results of feature fusion are shown in Table 2. The experiments show that performing pixel alignment first and then fusing features can achieve better results than other configurations. It proves that the proposed modules are powerful for feature representations.

To evaluate the performance of the proposed modules, several ablation experiments were performed on the data set for the different combinations of the proposed modules in our network. The results are shown in Table 3. We adopt ResNet-50 as the backbone network of all the variants of our method for ablation experiments. In Table 3, Atrous-Conv indicates the improved Atrous convolution, PPM is the Pyramid Pooling Module, BEM stands for the Boundary Enhancement Module, and PAM denotes the pixel alignment module.

According to Table 3, we find that the performance of the proposed network can be obviously enhanced by employing Atrous convolutions in the backbone network. After the pyramid pooling module (PPM) is enabled, we achieve the mIoUs of 71.83%, 67.68% and 69.47% on the test datasets of DS01, DS02, and DS03, respectively. The mIoUs are improved by about 2% after using the Boundary Enhancement Module (BEM). Although the boundary pixels often occupy a relatively small portion of the whole image, boundary information plays a key role in improving the accuracy of segmentation. The mIoUs are greatly improved by 7%–8% after the pixel alignment module (PAM) is used.

In addition, we compute the p-values of our results in Tables 2, 3 for analyzing the statistical significance of ablation experiments. Given two random sets X and Y, the t-score of the two sets is computed as follows: t = μ x − μ y n x − 1 σ x 2 + n y − 1 σ y 2 n x + n y − 2 1 n x + 1 n y (4) where μ _x , σ _x and n _x are respectively the mean, the standard deviation and the sample number of the best set X, and μ _y , σ _y and n _y are the mean, the standard deviation and the sample number of the tested set Y, respectively.

According to the t-scores and degree of freedom d _f = n _x + n _y -2, we can find the ranges of the p-values in the t-score lookup table. For the sake of convenience, we use the Excel software to automatically compute the p-values. By observing Table 2, we find that all the p-values are greater than 5%, so it is not significant in statistics. In vision fields, although existing methods achieving only 1% increase of mIoUs are not significant statistically, they are often considered as excellent algorithms, and they do not perform p-value analyses. In Table 3, we find that all the p-values are far less than 5%, so the best variant is statistically significant.

The proposed method also achieves good results for real fires in wild scenes, as shown in Figure 7. The proposed method can accurately segment smoke regions. Although there are smoke-like objects, such as clouds, our model easily discriminates between clouds and smokes.

In order to ensure the safety of electric power transmission lines, it is necessary to check fire safety around the electric lines. We tested the proposed method on several images captured from iron towers of electric power transmission lines, as shown in Figure 8. From the experimental results, we can see that the proposed method not only detects smoke successfully, but also obtain a relatively accurate smoke contours. Segmented smoke contours can allow the relevant personnel in the electric power department to have a more accurate judgment of the possible fire spread trend, and take corresponding countermeasures in advance to determine the safety of power lines.