In our study, two global daily soil moisture products, namely the climate change initiative soil moisture prodcut (CCI_ori) (Dorigo, et al., 2017; Gruber et al., 2019; Preimesberger et al., 2020) and the correlation merging soil moisture product (CM_ori) (Hagan et al., 2020), are used. Both datasets are daily products merged from multiple satellites and gridded at 0.25° horizontal resolution. The applied datasets have been widely validated using in-situ observations, and they are indicated to be amongst the best satellite soil moisture products with good data promising (Dorigo, et al., 2017; Gruber et al., 2019; Hagan et al., 2020; Preimesberger et al., 2020). The CCI data is combined with active and passive microwave sensors from multiple satellites. The passive microwave soil moisture data include the Scanning Multichannel Microwave Radiometer (SMMR), the Special Sensor Microwave Imager (SSM/I), the Tropical Rainfall Measuring Mission Microwave Imager (TRMM/TMI) (Owe et al., 2008), the Advanced Microwave Scanning Radiometer for the Earth Observing System (AMSR-E) (Njoku et al., 2003), the WindSat (Parinussa et al., 2011), the Soil Moisture Ocean Salinity (SMOS) (Kerr et al., 2001), the Advanced Microwave Scanning Radiometer 2 (AMSR2) (Parinussa et al., 2015), the Soil Moisture Active Passive (SMAP) (Entekhabi et al., 2010), the Global Precipitation Measurement Microwave Imager (GMP/GMI) (Hou et al., 2014) and the Fengyun-3 (FY-3) (Yang et al., 2012) respectively. We used the most recent CCI version 7.1 combined soil moisture data. The CM data is merged from six passive microwave products, including SSM/I, TRMM/TMI, AMSR-E, WindSat, AMSR2 and FY-3B satellites. The CM data is also merged using the CCI scheme with however different merging criteria. CCI merges data by minimizing the error while CM merges data by maximizing the correlation (Hagan et al., 2020). In this study, the CCI ranges from 1989 to 2021 and the CM ranges from 2006 to 2019 respectively. Spatially, we have limited both data to 60°S–60°N, owing to the frozen soil (Ran et al., 2021) and ice cover issues at high latitudes.

The percentages of missing data in CCI_ori and CM_ori are shown in Figure 1. There are high percentages of missing data in highland mountains and mid-high latitudes, mainly due to frozen soil in winter. Apparently, there are more missing values in the CM_ori than the CCI_ori data in most regions of the globe.

Due to different meteorological and terrestial conditions, the spatio-temporal soil moisture patterns are different among continents (Liu et al., 2018). We therefore partition the global land into six regions as shown in Figure 2, considering two key aspects: 1) The oceans have to be eliminated from the model training so as to minimize its impact on terrestrial soil moisture; 2) these partitions have some overlapping regions, so that the artifacts at edges can be reduced during post-processing (Barth et al., 2020).

Generally, traditional reconstruction methods can only consider temporal information for time series reconstruction or only spatial information for image reconstruction in the reconstruction process, while cannot consider both spatial and temporal information of data in the reconstruction process, which leads to a great uncertainty (Zhang et al., 2021c). It has been shown that DL methods generally outperform traditional data reconstruction methods with stronger non-linear modeling capabilities (Nogueira et al., 2018; Ma et al., 2019; Wu et al., 2019), and the CNN method is able to consider both temporal and spatial information of the data in the reconstruction process (Barth et al., 2020; Han et al., 2020; Luo et al., 2022). For the above reasons, this study does not compare the performance of different methods, but focuses on obtaining reconstructed data.

The used model here is a CNN structure with autoencoding approach. The convolutional auto-encoder replaces the full connection layer in the traditional auto-encoder with a convolutional layer and a pooling layer, thus effectively reducing data transfer loss (Yuan et al., 2019). The model contains five encoding and decoding layers, and each layer has a convolutional kernel size of 3 × 3. The pooling layer is used to reduce the size of the model, enhance computational speed, and optimize model robustness during features extraction. After each encoding layer, there is an average pooling layer with a filter size of 2 × 2; they together constitute the encoder which can convert the input feature into a latent-space representation. The decoder, consisting of convolutional layers and interpolation layers, reconstructs the input data according to the potential spatial representation obtained by the encoder. Meanwhile, skip connection structures are added to the model to increase feature utilization. The DL model obtains soil moisture values, longitude, latitude and time information from CM_ori and CCI_ori data respectively, from which the input feature is generated. The input feature includes not only soil moisture at date T, but also the values at date T − 1 and T + 1. Moreover, The seasonality calculated at date T is also added to the feature, so that the DL model can consider the seasonal variability. Longitudes and latitudes are added to the input feature, so that the DL model can consider the influence of spatial positions on soil moisture. During the network training process, the convolution operation enables the model to consider the influence of the surrounding pixels on the center pixel soil moisture. In summary, the used DL model can consider information from both spatial and temporal dimensions, which is the main advantage of the model. The model diagram is shown in Figure 3. We chose RMSE as the loss function of the model to assess the error between the predicted and true values, which also provides support for the parameter adjustment of the model (LeCun et al., 2015). The formula is as follows: R M S E = 1 n ∑ i = 1 n   y ^ i − y i 2 (1) where y ^ i represents the predicted value, y i represents the true value, and n represents the number of data involved in the calculation of the loss function. Adam optimizer is used to optimize the parameters. The learning rate is set to 0.001, the exponential decay rate is set to 0.9 for the first moment and 0.999 for the second moment. The regularization parameter is set to 10⁻⁸.

The DL model is trained for the six partitioned regions respectively, from which the reconstructed data are then spatially merged with edge artifacts removed. Soil moisture is considered ineffective in densely vegetated areas; therefore we have eliminated the reconstructed values where the multiple-year averaged NDVI (Normalized Difference Vegetation Index) is higher than 0.8 using the Advanced Very High-Resolution Radiometer (AVHRR) product (doi.org/10.7289/V5PZ56R6). The reconstructed values are also eliminated where soil temperature is below 0°C in cold seasons using ERA5-Land data a reference (doi.org/10.24381/cds.e2161bac).

We use cross-validation and in-situ observations to evaluate the quality of reconstructed data. Several error metrics including the correlation coefficient (R), bias, root mean square error (RMSE) and unbiased root mean square error (ubRMSE) are used. The formula is as follows: R = ∑ i = 1 n y i − y ¯ ( y ^ i − y ^ ¯ ) ∑ i = 1 n y i − y ¯ 2 ∑ i = 1 n y ^ i − y ^ ¯ 2 (2) b i a s = 1 n ∑ i = 1 n y ^ i − y i (3) u b R M S E = R M S E 2 − b i a s 2 (4) where y ^ i represents the original or reconstructed soil moisture, y i represents the reference soil moisture, y ^ ¯ represents the mean value of original or reconstructed soil moisture, y ¯ represents the mean value of reference soil moisture, and the n represents the number of data involved in the calculation.

For the cross-validation, we add 20% artificial missing values in the CCI_ori and CM_ori data respectively, and also continous missing values in rectangular regions. Then the convolutional auto-encoder is used for reconstruction. Finally, the reconstructed artificial missing values and their corresponding original values are used to evaluate the reconstruction performance. For the in-situ validation, we use the measured daily data obtained from multiple soil moisture stations in the International Soil Moisture Network (ISMN) as the in-situ data for the evaluation (Dorigo et al., 2013). Totally eleven networks are selected from ISMN, which are distributed in North America and Europe, Africa, Asia, and Oceania. An additional soil moisture network from Jiangsu province, China is also used. Finally we have twelve soil moisture networks which consist in-situ soil moisture observations, spanning 2012–2017, from 485 stations for data validation. The soil moisture networks are shown as point in Figure 2 with different colors.

We design an experiment of artifical missing values to prove the feasibility and effectiveness of deep learning model in reconstructing soil moisture, using data of the last year of CCI_ori (2021) and CM_ori (2019). In the experiment, we add 20% missing values in random and spatially continuous manners to the CCI_ori and CM_ori data respectively. The processed data is put into the trained deep learning model to obtain the reconstructions, denoted as CCI_rec and CM_rec, respectively. Figure 4 shows the CCI_ori and CM_ori with artificial missing values on 31 August 2021 and 31 August 2019 as examples, and their reconstructions. It can be seen from Figure 4 that the reconstructed artificial missing values shows quite similar spatial patterns as the original data; particularly, the artificial missing values in the rectangular regions are very well reconstructed. This indicates the used DL model is capable of reconstructing not only random missing values but also those of large regions.

To better understand the reconstruction performance, we further select several small regions with artificial random and rectangular missing values to show the results as in Figure 5. Clearly, the reconstructed missing values in CCI_rec and CM_rec are highly consistent with those in the CCI_ori and CM_ori data. Particularly, it is found that the artificial missing values in all the rectangular regions are very well constructed with very fine spatial patterns. However, this is not surprising because the used DL model considers soil moisture variability in both spatial and temporal dimensions.

In addition, the scatter plots of the reconstructed artificial missing values against the original ones are shown in Figure 6. It appears clearly that the fited line (white dotted line in Figure 6) is close to 1:1 (white solid line in Figure 6), indicating the consistent density distribution between the reconstructions and original values. Meanwhile, they have high correlation coefficients and low RMSE. R for CCI_ori and CCI_rec is 0.987 and RMSE for them is only 0.015 m³/m³. For CM_ori and CM_rec, R is 0.974 and RMSE is 0.032 m³/m³. Obviously, R is higher for the CCI data than the CM data, and RMSE is vise versa. This is reasonable because there are apparently more missing values in the CM data which can largely affect the training of DL model. However, their differences of R and RMSE is relatively limited; this implies that the used DL model can still fulfill the reconstruction task even if there is a large portion of missing values in the original data.

The above results of cross-validation indicates that the used DL method is highly capable of reconstructing the missing values in the original soil moisture data, which proves that the soil moisture data reconstructed by this method has good credibility.

To optimize parameters of the DL model, the loss function of each iteration is computed using RMSE, which is shown in Figure 7. Since the global land is partitioned into six regions (Figure 2) for the reconstruction, their loss curves are shown in Figure 7 respectively. The loss values of CCI and CM data in different regions are initially all above 0.05, which gradually decrease with increasing iterations and finally stabilize after 200 epochs. Although all the loss functions are stabilized below 0.05, they are different for the partitioned regions owing to different data properties.

The spatial patterns of CCI_ori and CM_ori as well as their reconstructions on 1 August 2012 is used as an example to better illustrate the results. Clearly, although there are large regions of missing values in CCI_ori (Figure 8A) and CM_ori (Figure 8C), their spatial patterns are very well reconstructed as shown in Figures 8B, D respectively. It is worth noting that, values are masked out in CCI_rec and CM_rec where either soil temperature is below 0°C or Normalized Difference Vegetation Index (NDVI) is larger than 0.8. Clearly, both CCI_rec and CM_rec are rather consistent in their spatial patterns, indicating such a reconstruction with DL method can effectively resolve the data gaps in large space while preserve the spatial features of the original data.

To further verify the soil moisture reconstruction in temporal dimension, two pixels are selected to superimpose the original time series onto the reconstructed ones as shown in Figure 9. The time series spans from 1 January 2012 to 31 December 2017. While time series in Figures 9A, C are derived from the same pixel, those in Figures 9B, D are derived from a second one. It appears that, the temporal changes of original data are well reconstructed considering the variations at seasonal and finer time scales, although they are slightly different in CCI and CM data. It can be seen that, CCI data has smaller day-to-day variations than those in CM data. It is thus indicated the used DL method can reliably reconstruct the temporal variations of both CCI and CM data in presence of considerable portion of missing values.

The above analysis indicates the reconstructed data retains the features of the original data in both spatial and temporal dimensions (Figures 8, 9). To understand the general performance of data reconstruction, we show the scatter plots of the global reconstructions against the original data and in-situ observations in Figure 10. The fitted data is quite close to 1:1 line (white solid line in Figure 10), and the points are almost evenly distributed on both sides of the fitted line. It appears R between CCI_rec and CCI_ori is 0.97, and RMSE is 0.019 m³/m³ (Figure 10A); those metrics for CM_rec and CM_ori is 0.96 and 0.031 m³/m³ (Figure 10B) respectively. It is indicated the CCI reconstruction performs better than CM data. The validations of CCI_rec and CM_rec with in-situ observations (Figures 10C, D) are similar to in-situ observation verification results for CCI_ori and CM_ori (Figures 10E, F). The data in Figures 10C, D are all distributed above the 1:1 line (white solid line in Figure 10), which may indicate that the CCI_rec and CM_rec data overestimate soil moisture to some extent. The data distribution of CCI_rec is more concentrated than CM_rec. R between CM_rec and in-situ observations is 0.45 and 0.140 m³/m³ (Figure 10D), and those metrics for CCI_rec and in-situ observations is 0.61 and 0.099 m³/m³ (Figure 10C) respectively, indicating better reconstruction results of CCI than CM.

Four metrics of R, RMSE, ubRMSE and bias are used to further evaluate CCI_rec and CM_rec using global in-situ observations (Figure 11). Generally speaking, R between CCI_rec and in-situ observations appears to be 0.65 on average, and that for CM_rec is 0.42 on average. Apparently, CCI_rec shows much higher correlations with in-situ observations with smaller range among the stations. Conversely, RMSE, ubRMSE and bias of CCI_rec is much smaller than those of CM_rec, and their ranges are much smaller indicating less uncertainties of these results. It is thus convincing that the derived CCI_rec is more consistent with the in-situ observations globally.

In Figure 12, we further show the boxplots of R, RMSE, ubRMSE and bias of CCI_rec and CM_rec against in-situ observations from eleven regional soil moisture from ISMN and the JIANGSU network in addition. Metrics of satellite data show considerable variations in different regions due to the factors such as satellite sensors, retrieval algorithms and merging algorithms (Fu et al., 2019). Apparently, R coefficients between CCI_rec and in-situ observations in all the twelve regions appears to be much higher than those for CM_rec; on the contrary, metrics of RMSE, ubRMSE and bias for CCI_rec are largely smaller than those for CM_rec. It is thus indicated that CCI_rec notably outperforms CM_rec in the studied soil moisture networks across the globe.

It is of interest to understand the spatio-temporal consistency of the reconstructed CCI_rec and CM_rec data, for which we computed their pixel-wise correlation coefficients as shown in Figure 13. Generally, CCI_rec and CM_rec are highly and positively correlated across most of the global land, and their correlation coefficients can reach >0.8 in transitional regions between wet and dry climate. However, in some regions such as southwest China, they show slightly negative correlation coefficients, indicating large discrepancy in both data and we have to use them with caution. In densely vegetated regions such as rainforest, the CCI_rec and CM_rec values are masked out since microwave remote sensing can not penetrate dense vegetation.