Generally, we only know the image I_c under an unknown light source E_c that must be estimated. The goal of color constancy is to estimate E_c from I_c and then compute it as E_c = I_c/R_c. How do we estimate E_c from I_c? To address this problem, we formulate color constancy as a regression problem.

First, we obtain the semantic context of the image, and for this, we use PSPNet (Zhao et al., 2016), defined as I_s, and then convert I_c from RGB space to log − uv space to obtain (I_u, I_v). Combining these three channels into a new three-channel image I_n, the aim is to find a mapping f_θ such that f_θ(I_n) = P_uv, where P_uv represents the light value in log − uv space.

As mentioned earlier, the objects in the scene have a great effect on the color constancy of human vision (Van De Weijer et al., 2007; Gao et al., 2019). Therefore, the designed color-constancy algorithm should imitate the human visual system, i.e., the mapping f_θ should be able to be based on semantic information and is used to support the larger contribution area and suppress the smaller contribution area in the image. Therefore, two aspects must be considered in the model: First, one must find a way to estimate the illumination of each area in the image, and, second, one must use an adaptive algorithm to integrate the illumination of these multiple areas into a global illumination. Supposing that R = R₁, R₂, ..., R_n represents n non-overlapping regions in the image I_c, Euvi represents the estimated scene illumination of the ith area R_i. Therefore, the mapping f_θ can be expressed as follows:

where w(R_i) represents the contribution of each area to the illumination estimation, i.e., the weight. In other words, if R_i contains the semantic context information, the corresponding w(R_i) has a higher weight value.

It can be seen from Equation (4) that it is necessary to design a network structure f_θ to be able to calculate w(R_i) and Euvi in each area. The network structure is shown in Figure 1.

We learn P_uv at different scales from the intermediate features with scales of 1/16, 1/32, and 1/64. Defining the superscript j to represent the scale, Puvj then represents the estimated illumination in log − uv space under the jth scale, which is converted back to RGB space according to Equation (3) to obtain Pcj, where c = R, G, B. Finally, the final illumination P_c in RGB color space is obtained by simple calculation of the obtained illumination on different scales, and the formula is:

where C_j represents the weight of the illumination obtained at each scale. In this article, j = 1, 2, 3. In the final illumination calculation, it is assumed that the estimated illumination at different scales contributes the same to the scene illumination, namely, C_j = 1/3, j = 1, 2, 3.

In most of the previous methods, the extracted features are directly calculated through several fully connected layers to obtain a global illumination. However, it can be seen from the results of the earlier literature that the effect of illumination estimation is not significantly improved. Referring to Hu et al. (2017), we used a custom network layer, called a WPL, the main function of which is to converge the regional illumination into a global illumination, and at the same time, learn the weight w(R_i) of each area; R_i represents the ith area. The WPL on each scale is expressed as follows:

where P^j represents the output of the WPL layer in the jth scale, wij the contribution of each region that must be learned on the jth scale, Eij the illumination of the i area on the jth scale that must be learned, and n represents the number of areas. In this study, each scale is n=W16×H16,W32×H32,W64×H64.

We made a simple attempt to determine how to select illumination at multiple scales, and we used several samples to train the 3 classification problems, hoping to obtain the probability of illumination at different scales. However, the training process model is difficult to converge and the effect is not ideal. Finally, for the sake of simplicity, it is assumed that each scale has the same contribution to the illumination estimation, so the average value of 3-scale illumination is taken as the final illumination in this section³, and the results also show that the average value is higher than that of a single scale on a variety of datasets.

At the time of training optimization, Euclidian loss is utilized for the network, defined

where Loss_j is the loss function of the jth scale, E^e is the illumination estimated by the network, E^t is the ground truth illumination, and N represents the batch of the training samples. The loss is minimized using stochastic gradient descent with standard back-propagation.

Either shallower (i.e., Shi et al. 2016) or deeper networks (i.e., VGG-net Simonyan and Zisserman 2014) could replace the pre-feature extraction in the proposed system. However, due to the color-constancy problem, the best network for feature extraction should have enough capacity to distinguish ambiguities and should be sensitive to different illuminants. We tried several common networks, such as AlexNet (Krizhevsky et al., 2012), VggNet16 (Simonyan and Zisserman, 2014), and VggNet19 (Simonyan and Zisserman, 2014), and they all achieved good results. Finally, to improve the computational efficiency of the network, we simplified AlexNet (Krizhevsky et al., 2012) and removed all of its pooling layers. The test results showed that the efficiency increased by approximately 4 and the accuracy by an average of 5.2%.

Because semantic information is needed in this study, we mainly used a semantic segmentation dataset, namely, the ADE20k dataset (Zhou et al., 2016). Meanwhile, we used PSPNET (Zhao et al., 2016; Zhou et al., 2017)⁴ to segment the semantic information for Color Checker (Gehler et al., 2008; Zhou et al., 2017) and NUS 8-Camera datasets (Cheng et al., 2014).

During the training process, 100 images with accurate semantic segmentation were manually selected from the Color Checker dataset (Gehler et al., 2008), and 200 images were extracted from the NUS 8-Camera dataset (Cheng et al., 2014).

In addition, since the ADE20k dataset (Zhou et al., 2017) does not provide illumination information, it is assumed that the images in the ADE20k dataset are corrected white-balanced images, and we, therefore, visually selected 500 images with normal color. Different lights were then rendered according to the following equations:

where (r_i, g_i, b_i) represents the simulated scene illumination. After simulation, more than 2,000 training images with illumination labels and accurate semantic information were obtained from the ADE20k (Zhou et al., 2017) dataset. These 2,000 images were cut, mirrored horizontally and vertically, and rotated from (−30^o, 30^o), 90^o, 180^o, and a total of approximately 20,000 pieces of data were obtained. Similarly, the images selected from the Color Checker (Gehler et al., 2008) and NUS 8-Camera datasets (Cheng et al., 2014) were processed in the same way to obtain approximately 8,000 images. All 28,000 images were randomly cropped and normalized to 512 × 512 as network input. As in the previous study, 3-fold cross-validation was used for all of the datasets, and each one run was used for training, one for validation, and one for testing.

Color-constancy algorithms are often evaluated using a distance measure, such as Euclidean distance (Land, 1977; Buchsbaum, 1980), perceptual distances (Gijsenij et al., 2009), reproduction angular error (Finlayson et al., 2016), and angular error (Hordley and Finlayson, 2004; Cheng et al., 2014; Bianco et al., 2015; Shi et al., 2016). Within these metrics, the angular error is the most widely used in this field, and most of the existing works (Cheng et al., 2014; Bianco et al., 2015; Shi et al., 2016) that tested using the Color Checker (Gehler et al., 2008), NUS 8-Camera (Cheng et al., 2014), and ADE20k datasetd (Zhou et al., 2017) reported their performance in terms of the angular error in five indexes: Mean, Median, and TriMean of all errors; mean of the lowest 25% of errors (Best 25%); and mean of the highest 25% of errors (Worst 25%). Hence, in the present study, we also used these indexes. In particular, although we aimed to estimate the single illuminant in this study, we also tested performance on the popular outdoor multi-illuminant dataset (Arjan et al., 2012). The angular error between the estimated illuminant E_e and ground-truth illuminant Ee* is computed for each image as follows:

The less the value of e is the better performance of the method.

In this subsection, the parameter settings for training our final model are given.

Feature-extraction-network selection: Different network structures, such as AlexNet (Krizhevsky et al., 2012), VGGNet-19 (Simonyan and Zisserman, 2014), and SqueezeNet (Iandola et al., 2016), were used to test performance. The comparison diagram is shown in Figure 2B, from which it can be seen that, although VGGNet-16 (Simonyan and Zisserman, 2014) and VGGNet-19 (Simonyan and Zisserman, 2014) network structures have a better effect than other networks, they take more time. Finally, considering effect and efficiency, the structure in Figure 1 is used in this study.

Network input and output: We compared the performance of the model trained with (I_u, I_v, I_t) three-channel image input and (I_u, I_v) two-channel input, and we tested the performance of different resolution images as network input. The comparison results are shown in Table 1. Considering effect and efficiency, the effect is the best when the network input image resolution is 512 and semantic information is used at the same time.

For output, we tested the effects of 1–5 scales outputs on illumination estimation, where 1 scale uses W64×H64, 2 scales uses W32×H32,W64×H64, 3 scales use W16×H16,W32×H32,W64×H64, 4 scales use W8×H8,W16×H16,W32×H32,W64×H64, and 5 scales use W8×H8,W16×H16,W32×H32,W64×H64,W128×H128. The curves are shown in Figures 2C,D. It can be seen that the effect is best when the scale is 4, but the time consumption is more than doubled when the scale is 3. Considering comprehensive effect and efficiency, we used 3 scales in the network. Figure 3 shows the intermediate results of estimating illumination at different 3 scales.

Batch size and learning: For optimization, Adam (Kingma and Ba, 2014) was employed with a batch size of 64, and a basic learning rate of 0.0001 was set for training. We trained all of the experiments over 4,000 epochs (2,50,000 iterations with batch size 64). The average angular error is calculated in the Color Checker dataset for every 20 epochs. The curve is shown in Figure 2A. Figure 4 shows the resulting image and local illumination after different numbers of epochs.

To evaluate the performance of the proposed method and the influence of the WPL layer on the method. We trained two models. One used the mean value when the local region converges to the global, which is defined as MSRWNS-AVG, the other model used the WPL layer, which is defined as MSRWNS. In addition, we also estimated the illumination effect at each scale, defined as MSRWNS-1, MSRWNS-2, and MSRWNS-3. Several visualizations of processing outputs obtained using the proposed method are presented in Figure 5.

The quantitative performance comparison on the Color Checker dataset is presented in Table 2 and the results on the NUS 8-Camera dataset in Table 3. The performance comparisons on the ADE20k (Zhou et al., 2017), SFU Lab dataset (Barnard et al., 2010), and SFU Gray-Ball dataset (200, 2003) are shown in Tables 4–6, respectively. Most CNN-based methods compare the effects on only two datasets, Color Checker dataset and NUS 8-Camera dataset. In order to make the comparison results consistent, the data in Tables 2, 3 are from Shi et al. (2016), while others were trained by us with the same training samples mentioned in the datasets section.

In addition, we compared our method with most of the existing works; typical works include the Deep Specialized Network (DS-Net) (Shi et al., 2016) and Fully Convolutional Color Constancy with confidence-weighted pooling (FC4) (Hu et al., 2017). Several visualizations of testing outputs obtained using the proposed method are presented in Figures 6, 7.

From Tables 2, 3, it can be seen that the mean error of the proposed method is reduced by 12.3% on the Color Checker dataset and by 5.8% compared to DS-Net (Shi et al., 2016), and reduced by 7.3% on the Color Checker dataset and reduced by 0.5% compared to FC4 (Hu et al., 2017). In addition, using the WPL layer under a single scale gives a better result than that obtained using the mean. The effect of using the mean of three scales on most indicators is better than the result of using a single scale.

In particular, the proposed method shows the best performance on the ADE20k (Zhou et al., 2017) dataset. The mean angular error is lower than that of DS-Net (Shi et al., 2016) by 29.7% (from 1.68 to 1.18), and the mean error of the worst 25% was reduced by 25.6% (from 3.86 to 2.87) compared to DS-Net (Shi et al., 2016). Other indicators have also been reduced to a certain extent. The reason for these results is that the semantic segmentation model used is trained based on the ADE20k (Zhou et al., 2017) dataset. The accuracy of the semantics on this dataset is high, and there are parts of training data in the ADE20k dataset (Zhou et al., 2017).

In addition, we also provided several natural examples captured by a Canon D7100 without auto-white-balance ⁵, as shown in Figure 7, and we obtained several images of natural scenes with more accurate colors from the Internet, and then performed some random color casting. Results are shown in Figure 8.

It can be seen From Figure 7 that on the image taken indoors (the first line in Figure 7) the image corrected by a CNN (Bianco et al., 2015) is obviously yellowish, and that corrected by DS-Net is slightly greenish. The white-balance effect of the camera and our result is similar and look relatively natural. In outdoor scenes, it can be seen from the second line that the image corrected by a CNN (Bianco et al., 2015) has a very obvious color cast. The other types are visually more natural. Our results are seen in the sky part of the image, in which the clouds are more realistic. The scene in the third row exhibits little difference in visual effects. This may be due to sufficient sunlight in the shooting scene, and the objects in the scene receive very uniform illumination. In this way, any method can obtain better results more accurately. From Figure 8, it can be found from the first row that, although there is a phenomenon of highlight overflow in some areas of the sky (due to random color casts leading to pixel overflow in some areas), the overall color is close to that of the real image; the second row is due to the random color cast. The resulting color cast is small and the corrected image is basically the same as the original image. In the third row, it can be seen that, although the corrected image is different from the original image, the visual perception is better.

The code used to test the efficiency of the proposed method is based on Tensorflow (Rampasek and Goldenberg, 2016) and training took approximately 5 h, after which the loss tended to stabilize. In the testing phase, we converted the model to that of Caffe (Jia et al., 2014), implemented the WPL layer with C++ under Caffe, and finally used C++ code for testing. An average image took 34 ms on a CPU and only 12 ms on a GPU (the time does not include semantic segmentation) ⁶.

As mentioned in this article, the proposed method aims to solve the color constancy under a single illuminant, and we only compare our algorithm with existing single illuminant based methods. In addition, after the WPL layer, we can get the local illumination of the regions, it can estimate the multi-illuminant sources in different local regions, shown in Figure 9, where the images are taken from the popular outdoor multi-illuminant dataset (Arjan et al., 2012). However, there has a large deviation between the estimated illumination and the real multi illumination, we have analyzed the reasons and found that there are big errors in semantics. We will solve this problem in future research.