An improved color dissimilarity feature based on fast multi-exposure image fusion with a median filter and recursive filter is proposed by Li and Kang (2012). Unlike the method proposed by Li and Kang (2012), static background images I^S of the scene are processed by a hybrid median filter (mHMF) (Kim et al., 2018) as in Equation (5).

where I^S represents the static background of the scene and mhmf (·) denotes an operator. The hybrid median filter (mHMF) (Kim et al., 2018) was applied to the worst image IminHE(x,y) in the histogram equalized exposure sequence IiHE(x,y), which is more beneficial to preserving the image edges in regions such as mutation than the median filter. Besides, the color dissimilarity feature D_i(x,y) of moving objects is calculated between the static background image I^S and histogram equalized image IiHE(x,y) in Li and Kang (2012) and Hayat and Imran (2019).

Comparisons of the color dissimilarity feature by Li and Kang (2012) and the proposed method have been conducted, as shown in Figure 3. The fused image in Figure 3B generated by the method of Li and Kang (2012) has ghosting artifacts at the ellipsoid. The proposed algorithm is validated by adopting underexposure, exposure normal, and overexposure source images, as shown in Figures 3C–E. According to Figure 3E, it can be seen that the results generated by underexposure images have a good effect on deghosting and are better than in Figure 3B. Therefore, in the following algorithm, we utilized the mHMF to handle underexposure images for related experiments.

Because of the correlation between the three channels in RGB color space, which affects the final multi-scale pyramid decomposition and fusion, the input source image is converted from RGB to YUV color space. The exposure feature weight item E_i(x,y) of the input image is measured in the Y channel as in Equation (6).

where Y_i(x,y) is the standardized value of the Y channel, Yi is the mean value of Y_i(x,y), and σ is a Gaussian kernel parameter taken as σ = 0.2. Besides, to increase the SNR of the input image sequence and retain the detailed information of the brightest/darkest regions, this method uses the brightness quality metric Bi=Yi2 in Kou et al. (2018).

The local contrast is measured using Equation (7), which is extracted by non-standardized dense filtering in dense SIFT descriptor (Liu et al., 2010).

where DSIFT(.) represents the operator that computes the non-normalized dense SIFT source image mapping, C_i(x,y) represents a simple indicator vector for local contrast measurement, and Iigary(x,y) denotes the grayscale image corresponding to the input image sequence I_i(x,y). At each pixel, the Iigary(x,y) mapping is regarded as the l₁ norm of C_i(x,y). Besides, this study selects a winner-take-all weight allocation strategy (Liu and Wang, 2015; Hayat and Imran, 2019) to obtain the final local contrast weight term Cifinal(x,y).

First, the following four weight items of the input image sequence are calculated: color dissimilarity feature, exposure feature, brightness feature, and local contrast. Second, weight items are multiplied to generate a weighted mapping, as in Equation (8).

Using WGIF (Li et al., 2014) directly refines and filters the weight map obtained by Equation (8), which is different from the refinement of the weight map in Liu and Wang (2015) and Hayat and Imran (2019). In the process of filter refinement, both the source image and the guide image are used W_i(x, y). Then, normalizing refined weight maps makes weight maps sum to 1 at every pixel. The final weight map is shown in Equation (9).

where ŴiWF(x,y) denotes the weight map after WGIF refinement, Wi(x,y) denotes the final normalized weight map, and ε = 10⁻⁵ is a small positive value, avoiding a zero denominator in the calculation process.

Utilizing the original multi-scale pyramid model (Mertens et al., 2007) may produce fusion images with a loss of details and the halo phenomenon. Therefore, an improved pyramid fusion model is used. In this pyramid model, the Laplacian and Gaussian pyramids are disaggregated into n levels, as shown in Figure 4. The total number of levels n is defined by Equation (10).

where r_o and c_o are the number of rows and columns of input image pixels, respectively.

It is considered that, at the highest level of the Gaussian pyramid, improper smoothing of edges is the main reason for producing halos. On the lower levels of the Gaussian pyramid, the improper smoothing of the edges is not evident for the generation of halos. Therefore, on the n-th level of the RGB color space pyramid, using the single-scale fusion algorithm in Ancuti et al. (2016) adds the Laplacian pyramid information of the source image to the Gaussian pyramid weighted mapping as in Equation (11).

where I_i is the input image of LDR, Rni is the result of fusing the i-th image and the i-th image weight on the n-th level, and Gn{Wi(x,y)} is the n-th Gaussian pyramid of Wi(x,y). In Ancuti et al. (2016), n is the maximum number of levels of the Gaussian pyramid, L₁{I_i(x, y)} is the first level of the input image I_i(x,y) Laplacian pyramid, and λ is the coefficient of L₁{I_i(x, y)}, which controls the amplitude of the high-frequency signal L₁{I_i(x, y) }.

To retain detailed information on overexposed/underexposed areas, on the n-th level, the improved multi-scale exposure fusion algorithm proposed by Wang et al. (2019) is used as in Equation (12).

For underexposure source images, |L₁{L_n{I_i(x, y)}}| in Equation (12) is introduced at the n-th level to correct the incorrect weights introduced by the weighted mapping smoothed by the Gaussian smoothing filter. It also reasonably enhances the weight of the well-exposure areas in the underexposure image, which retains the details of the underexposure areas. For overexposure images, the weight map of the n-th level adopts the primary Gaussian smoothing filter to smooth, which retains the details of the overexposure area.

For other scales, the improved pyramid fusion is the same as the original pyramid fusion (Mertens et al., 2007). Finally, reconstructing the Laplacian pyramid composed of R_l(x,y) in Equation (13) generates the fused image R.

where l represents the level number of the pyramid. The image details and brightness enhancement method proposed by Li et al. (2017) is adopted to enhance fusion image detail information, which obtains the final multi-scale exposure fusion image.

Comparisons of the original and improved pyramid models have been conducted, as shown in Figure 5. Compared with the original pyramid model (see Figure 5B), the generated image in Figure 5C by the improved pyramid model performs well in contrast and detail processing aspects, especially in pedestrian and white cloud areas. It is considered that multi-scale pyramid decomposition and fusion, loss of details, and the halo phenomenon are complex problems in pyramid decomposition and fusion. Therefore, this study selects the improved pyramid model to decompose and fuse the input image.

Figure 7 shows the experimental results of different methods in the dynamic Brunswick sequence. In terms of ghost removal, the methods (see Figures 7B–E) presented by Mertens et al. (2007), Li and Kang (2012), Liu and Wang (2015), and Lee et al. (2018) have poor effects and cannot effectively remove ghosts in pedestrian areas. The pixel oversaturation distortion in Figure 7A significantly reduces the visual quality. The proposed method can produce a good result (see Figure 7F). No ghosting artifact phenomenon exists in the image, and human visual perception is natural.

Figure 8 shows the fusion results of different methods in the dynamic Cliff sequence. The images in Figures 8A, B generated by the methods of Mertens et al. (2007) and Hayat and Imran (2019) are dark in color, the local contrast is not apparent, and the ghosting phenomenon exists in the water waves, which reduces the visual observation effect to a certain extent. Although the methods (see Figures 8C–E) of Li and Kang (2012), Liu and Wang (2015), and Lee et al. (2018) increase the contrast of the image, there are still darker colors and ghost phenomena. Figure 8F is the method proposed in this study. In contrast, the ghost removal performance significantly improved. On the waves and beaches, detailed information, local contrast, and naturalness are maintained, consistent with human visual observation.

Figure 9 shows the performance comparison of different methods in the dynamic Llandudno sequence. The results (see Figures 9B–D) acquired by Mertens et al. (2007), Li and Kang (2012), and Liu and Wang (2015) show that there are apparent ghosting artifacts in the area of characters and that there is a loss of detail information and color distortion. In Figure 9A, the overall image deghosting effect is good, but the color above the house is dark. The image in Figure 9E is unclear, and there is a color distortion phenomenon. The proposed method can produce a good result (see Figure 9F). The characters in the image have no noticeable ghosting artifacts, details are well preserved, and the exposure level is consistent with human visual observation.

Experimental results on the static Venice sequence using different methods are shown in Figure 10. In terms of image sharpness and detail processing, the proposed method (see Figure 10F) is superior to the methods (see Figures 10A–E) proposed by Mertens et al. (2007), Li and Kang (2012), Liu and Wang (2015), Lee et al. (2018), and Hayat and Imran (2019). Especially in Figures 10B–D, in the sky and church areas of the image, exposure and sharpness are poor, local contrast is not apparent, and fused image details are lost. In the results of the method proposed by Lee et al. (2018) and Hayat and Imran (2019), the sharpness of the fused image has improved, but there is still local contrast that is not obvious, and details are lost (see Figures 10A, E).

The fusion results of six MEF methods on static scene landscape sequences are shown in Figure 11. In Figures 11B–E, in the sky area (white cloud parts), the sharpness is not good enough. In the method (see Figure 11A) proposed by Hayat and Imran (2019), although the sharpness and naturalness of the image are enhanced in the sky area, the fused image details are seriously lost. Compared with the methods (see Figures 11A–E) presented by Mertens et al. (2007), Li and Kang (2012), Liu and Wang (2015), Lee et al. (2018), and Hayat and Imran (2019), the proposed method in this study (see Figure 11F) has good saturation and contrast in the sky area, and the detailed information is retained better.

The structural similarity index (MEF-SSIMd) (Fang et al., 2019) is applied to measure structural similarity between input image sequences and fused images in dynamic ranges. The overall MEF-SSIMd is defined in Equation (14).

where q_d represents MEF-SSIMd of dynamic scenes and q_s represents MEF-SSIMd of static scenes.

The data range of MEF-SSIMd is [0,1]. The greater the value, the better the deghosting efficiency, and the stronger the robustness of the dynamic scene. The smaller the value is, the opposite is true. As shown in Table 1, using MEF-SSIMd objectively evaluates six MEF methods for the quality of generating fused images. Overall, the proposed method is superior to the other five existing MEF methods in the performance evaluation of MEF-SSIMd.

In multi-exposure image fusion, the fused image should meet the requirements of the human visual system to observe the scene. Since the general purpose does not reference the IQA (image quality assessment), the algorithm requires much training to meet the IQA. Thus, a non-reference quality metric, NIQE (Mittal et al., 2012) was proposed. The smaller the NIQE value is, the better the image quality is, and the image more closely accords with the requirements of the visible human system to observe the scene. On the contrary, the greater the NIQE value is, the fewer images conform requirements of the human visual system observation scene. As shown in Table 2, NIQE is used to evaluate the quality of fusion images produced by different MEF methods. Overall, the proposed method can acquire images with better naturalness.

In multi-exposure image fusion, sharpness is a critical factor in the visual evaluation of image quality. The sharpness of the image to achieve the human visual system can effortlessly detect blur and observe visual images. Therefore, Hassen et al. (2013) used sharpness in the complex wavelet transform domain to evaluate the local solid phase coherence (LPC) of the image features. Then, the overall sharpness index of LPC (LPC-SI) is proposed. A more considerable LPC-SI value of the fused image represents a clearer image, which conforms to the evaluation of human visual observation. A smaller LPC-SI value of the fused image represents a blurred image. The value range of LPC-SI is [0,100]. Table 3 shows the comparison results of LPC-SI values between the other five MEF methods and the presented method. A comprehensive comparison shows that the proposed method in this study outperforms the other five existing MEF methods.

As shown in Figure 12, the proposed method in this study ranks first in the line graph of the mean values of the entire reference objective evaluation index MEF-SSIMd and non-reference objective evaluation index NIQE, LPC, and average gradient (AG). The proposed MEF method without ghosting based on the exposure fusion framework and color dissimilarity feature can effectively remove ghosting in dynamic scene MEF. It also improves the sharpness and naturalness of the fused image and retains many details.