For the experimental scenarios of different types of oil products, the Cubert S185 airborne hyperspectral imager and DJI “Yu” 2 airborne thermal infrared camera were used to obtain approximately synchronous hyperspectral images and thermal infrared images of different types of oil products ( Figures 3A, B ). For the experimental scenes of crude oil and its emulsions in different states, the Cubert S185 airborne hyperspectral imager and Zemmuse H20T airborne thermal infrared imager were used to obtain approximately synchronous hyperspectral and thermal infrared images of crude oil and its emulsions in different states ( Figures 3C, D ). The main parameters of the Cubert S185 airborne hyperspectral imager and Zemmuse H20T airborne thermal infrared imager are shown in Table 3 . Except for the camera focus and stability, the main parameters of Zemmuse H20T airborne thermal infrared imager and DJI “Yu” 2 airborne thermal infrared camera are the same.

The data acquired by the airborne hyperspectral imager include the original measured spectral data, reference plate and dark current calibration data. The data to be converted was imported to the Cubert Utils Touch software. The spectral reflectance data of different oil spill pollution types was exported. Import the thermal infrared image acquired by the airborne thermal infrared imager into the DJI Thermal Analysis Tool to export the brightness temperature data of different oil spill pollution types. Due to the different coverage of airborne hyperspectral image and airborne thermal infrared image, two images need to be registered and cropped. The image size of different oil products as well as crude oil and its emulsions in different states used for identification experiments is 638×630 and 551×765 respectively.

Field data includes Analytical Spectral Devices (ASD) data and site photos used to assist in the production of ground truth image. The ASD FieldSpec4 spectrometer (350-2500 nm) was used to measure Lambertian standard plate, different oil spill pollution types, seawater and skylight to obtain their radiance, and then convert the radiance into the spectral reflectance of different oil spill pollution types and seawater according to the formula in literature (Yang et al., 2020). The detailed index parameters of ASD FieldSpec4 spectrometer are shown in literature (Yang et al., 2020). The ground truth image ( Figure 4 ) obtained through human-computer interaction interpretation is used as the benchmark for evaluating the classification and recognition results, and the performance of the classification and recognition model is tested at the same time.

SVM is a traditional machine learning method based on statistical learning theory. It can automatically find the support vector that has a greater ability to distinguish classification and then construct the classifier, which can maximize the interval between classes to achieve good statistics when the number of samples is small. This method has high convergence efficiency, training speed, and classification accuracy, and has been widely used in many fields of research in recent years (Wang et al., 2011; Hu et al., 2019a; Hu et al., 2019b). The kernel function selected in this paper is the radial basis function.

Random Forest is an algorithm that effectively uses multiple decision trees to train and predict samples and optimize decisions through the idea of ensemble Learning. The basic unit is the decision tree. The randomness of RF is mainly reflected in two aspects: one is the random selection of data, and the other is the random selection of features to be selected. This can make the decision trees in the RF different from each other, and further enhance the generalization ability of the model.

The RF algorithm uses multiple CART decision trees as weak classifiers. In CART tree, the criterion of impure measure used to select variables is Gini coefficient. The minimum Gini coefficient criterion is used for feature selection. Here, the number of decision trees in the RF is 100.

CNN is a significant achievement in the field of deep learning, and has been widely used in hyperspectral remote sensing classification in recent years. CNN has two main characteristics. One is local receptive fields, the other is weight sharing, which effectively reduces the number of parameters in the network and makes CNN have displacement invariance.

The CNN model structure used in this paper consists of seven information layers, including one input layer, two convolutional layers, two pooling layers, one full connection layer, and one output layer. The number of convolutional kernel in the first convolution layer is 10, and the size of convolutional kernel is 5×5. The number of convolutional kernel in the second convolution layer is 8, and the size of convolutional kernel is 3×3. The size of the subsampling filter in the first pooling layer and the second pooling layer is 1×1 and 1×1 respectively. The maximum pooling is adopted for the pooling layer. The number of batch training is 2, the number of iterations is 80, and the learning rate is 0.7.

According to the classification results of different types of oil products ( Figure 5 ), it can be seen that 2D-CNN, SVM and RF classifiers have different abilities in data feature mining, and the performance of classification is also different. Intuitively, SVM has relatively poor oil identification effect based on the HSI. The mixing of different oils, especially light oils (palm oil, diesel oil, gasoline) is serious. The recognition effect of heavy oils (crude oil, fuel oil) is good. The classification results are quite different from the ground truth image ( Figure 4A ). The recognition results of RF model and 2D-CNN model ( Figures 5A–C ) can better maintain the continuity of oil film, which is consistent with the ground truth image, and shows the powerful data mining ability and feature extraction ability of ensemble learning model and deep learning model.

For the HTI, the recognition results ( Figures 5D–F ) of the three algorithms are obviously better than those based on the HSI. With the addition of thermal infrared features, the oil type recognition results can better maintain the continuity of oil spill on the sea, with clearer boundaries, and the mixing phenomenon between light oils is greatly reduced, which is consistent with the ground truth image. However, gasoline and seawater still have a misclassification phenomenon, which is due to the volatility of gasoline, resulting in that the selected gasoline training samples are not completely pure gasoline pixels.

It can be seen from the recognition results of crude oil and its emulsions in different states ( Figure 6 ) that, for the HSI, the recognition results of 2D-CNN model on WO emulsion, OW emulsion and seawater ( Figure 6C ) are good, which are consistent with the ground truth image ( Figure 4B ), but the crude oil is partially divided into WO emulsion, showing the strong data mining ability and feature extraction ability of the deep learning model. The recognition results of SVM model and RF model for crude oil and its emulsions in different states ( Figures 6A, B ) are similar. In addition to the fact that crude oil is partially divided into WO emulsion, and seawater is also partially divided into WO emulsion.

Different from the recognition results of different oil products in Section 3.1.1, the recognition results of the three algorithms for crude oil and its emulsions in different states based on the HSI are significantly better than those of the HTI. The addition of thermal infrared features enhances the recognition effect between seawater and WO emulsion, but aggravates the mixing phenomenon between crude oil and WO emulsion, making most crude oil be wrongly divided into WO emulsion ( Figures 6D–F ).

The overall accuracy and Kappa coefficient in the confusion matrix are used in this paper to evaluate the overall identification accuracy of oil spill pollution types based on HSI and HTI, as shown in Table 5 . For different oils, the overall recognition accuracy of SVM, RF and 2D-CNN models based on the HSI is 72.66%, 78.82% and 80.09% respectively. After adding thermal infrared features, the overall recognition accuracy of SVM, RF and 2D-CNN models reached 83.24%, 85.08% and 82.9% respectively, and Kappa coefficients were 0.80, 0.82 and 0.80 respectively, indicating that the prediction results of the HTI were in good agreement with the actual results. For crude oil and its emulsions in different states, the overall recognition accuracy of SVM, RF and 2D-CNN models based on the HSI reaches 94.52%, 94.62% and 95.2% respectively. After adding thermal infrared features, the overall recognition accuracy of SVM, RF and 2D-CNN models is 92.06%, 91.67% and 91.45% respectively.

The F1 score is used to evaluate the identification accuracy of single oil spill pollution type based on HSI and HTI, as shown in Figure 7 . In general, for different oils, the recognition accuracy of the three algorithms based on HTI has been improved to varying degrees compared with that based on HSI. Taking the recognition results of RF model as an example, the F1 scores of gasoline, palm oil and diesel oil increased by 0.07, 0.21 and 0.20 respectively. The F1 score of crude oil and fuel oil increased by 0.11 and 0.03 respectively. The F1 score of seawater increased by 0.09. The experimental results show that the thermal infrared features can increase the identification ability of hyperspectral for different oils, especially light oils, and can effectively improve the oil identification accuracy. For crude oil and its emulsions in different states, the recognition accuracy of the three algorithms based on HTI is lower than that based on HSI to varying degrees, but the recognition accuracy of seawater is improved. Taking the recognition results of RF model as an example, the F1 scores of crude oil, WO emulsion and OW emulsion decreased by 0.37, 0.09 and 0.04 respectively, while the F1 scores of seawater increased by 0.10.

In general, the reflectivity of the five oil products and seawater is very small ( Figure 8 ), which is related to the absorption properties of oil products and seawater. In the visible light band, the reflectance spectra of light oils such as diesel, gasoline and palm oil are generally consistent with those of seawater, and there is an obvious reflection peak at about 480 nm; The reflectivity of heavy oil such as crude oil and fuel oil are obviously lower than those of light oils and seawater. In the near-infrared and shortwave infrared bands, the reflectivity of the five oils are higher than that of seawater. This is because the pure natural water is nearly a “black body” in the near-infrared band. Therefore, in the spectral range of 850~2500 nm, the reflectivity of the purer natural water is very low, almost zero.

Within the spectral range (450~950 nm) of the airborne hyperspectral image, the spectral curves of the three light oils and seawater are very similar. Therefore, based on the single dimensional characteristics of hyperspectral, the recognition results of the three algorithms for light oils (palm oil, diesel oil, gasoline) and seawater are poor, and the mixing phenomenon is serious. At the same time, the spectra of heavy oils and light oils are quite different, so except at the edge of the enclosure, the recognition accuracy of heavy oils such as crude oil and fuel oil based on the HSI is good, and there is little mixing with light oils and seawater.

In general, the spectral response of crude oil and its emulsions in different states and seawater in different spectral ranges is different ( Figure 9 ), which is related to its absorption and scattering properties. WO emulsion contains seawater droplets. When incident light enters seawater droplets from oil film, it will produce strong backscattering. In the near infrared and short wave infrared spectra, the WO emulsion has higher reflectivity. In addition, due to the absorption of -C-H in WO emulsion, obvious reflection valleys are formed at ~1725 nm, ~1760 nm and ~2170 nm. In the OW emulsion, dispersed small oil droplets exist in continuous water, which makes it have high reflectivity in the spectral range of 360~1400 nm. At the same time, due to the absorption of -O-H, obvious reflection valleys are formed at ~975 nm and ~1200 nm. This is very close to the results obtained from the indoor experiment carried out by Lu et al. (2019). Due to the strong absorption and low reflection of incident light, the spectral reflectance of crude oil is low, especially in the near infrared band, the spectral reflectivity of crude oil is nearly zero.

In the spectral range of airborne hyperspectral images (450~950 nm), there are large differences in the spectral response of crude oil and its emulsion different states and seawater. Therefore, based on the HSI, three algorithms are used to identify WO emulsion, WO emulsion and seawater with good results. However, due to the high oil concentration of the stable WO emulsion, the spectral curve of the WO emulsion is very similar to that of the crude oil. Therefore, the WO emulsion is mixed with the crude oil in the classification and recognition results based on the HSI.

Because the heat capacity and infrared reflectance of oil film and seawater are different, there exists temperature difference between oil film and background seawater, which is the basis using infrared image to detect marine oil spill. It can be seen from Figure 3B that the heavy oils (crude oil and fuel oil) show a “bright” hue on the thermal infrared image, which is the most obvious difference from the seawater background, in which the crude oil shows “brightest”, and the fuel oil shows “brighter”. Light oils (palm oil, diesel oil and gasoline) is displayed as “dark” hue on the thermal infrared image. Diesel oil is “darker”, palm oil is “dark”, and gasoline is “darkest”. This is mainly related to the characteristics of the oil products themselves, the ability to absorption and radiation and other factors. Heavy oils can absorb more solar radiation and emit it in the form of thermal radiation.

There is a certain conversion relationship between intensity of thermal infrared image and brightness temperature. Here, intensity is used to replace brightness temperature to analyze the thermal response characteristics of different oils. It can be seen from Figure 10 that the thermal infrared radiation intensity of the five oil products and seawater from high to low is crude oil, fuel oil, diesel oil, palm oil, seawater and gasoline. The thermal infrared intensity of gasoline is lower than that of seawater, because gasoline volatilization will cause the surface temperature to drop. The thermal infrared intensity of crude oil and fuel oil are obviously higher than those of diesel oil, palm oil and gasoline. The thermal infrared intensity of the three kinds of light oils is quite different. It is precisely because of this difference that the mixing phenomenon between light oils in the recognition results of the HTI is greatly weakened. However, there is still a certain mixing phenomenon between gasoline and seawater, which is related to the strong volatility of gasoline.

It can be seen from Figure 3D that the crude oil and WO emulsions show “bright” hue on the thermal infrared image, which is the most obvious difference from the seawater background. The OW emulsion and seawater show a “dark” hue on the thermal infrared image, among which, the oil in water emulsion is relatively “dark”, and the seawater is the “darkest”, which is mainly related to the concentration of oil. The higher the oil concentration, the more solar radiation absorbed and emitted in the form of thermal radiation.

The thermal infrared image of crude oil and its emulsion in different states was obtained using the Zemmuse H20T airborne thermal infrared imager. The brightness temperature analysis was carried out by selecting pixels of the same size for crude oil, WO emulsion, OW emulsion and seawater from the thermal infrared image. It can be seen from Figure 11 that the brightness temperature of crude oil and its emulsions in different states and seawater from high to low are crude oil, 90% WO emulsion, 75% WO emulsion, 60% WO emulsion, 0.1% OW emulsion and seawater. The brightness temperatures of crude oil and WO emulsions with different concentrations are close. The brightness temperature of OW emulsion and seawater is similar, but the brightness temperature of crude oil and WO emulsion is significantly different from that of OW emulsion and seawater, with an average difference of 20 °C. It is precisely because the brightness temperature difference between crude oil and WO emulsions of different concentrations is not obvious, which makes the mixing phenomenon of crude oil and WO emulsions enhanced in the recognition results of the HTI, so the vast majority of crude oil is wrongly divided into WO emulsions. This shows that thermal infrared remote sensing technology cannot be used to distinguish crude oil and WO emulsion, which is consistent with the conclusion reached by Jiao et al. (2021).

In order to explain the identification results of different oil spill pollution types using the HTI, the spectral and brightness temperature characteristics of different oils, crude oil and its emulsions in different states have been analyzed and discussed in the previous section. Here we further analyze and discuss the spectral and brightness temperature characteristics of oil films with different thicknesses, which can be used to explain the possible results of oil film thickness classification using the HTI.

It can be seen from Figure 12 that the spectral curves of oil films with different thicknesses are generally the same. With the increase of wavelength, the spectral reflectance of crude oil decreases gradually. Similarly, with the increase of oil film thickness, the spectral reflectance of crude oil decreases gradually. In the spectral range of airborne hyperspectral images (450~950 nm), the spectral responses of oil films with different thicknesses exist difference. Therefore, theoretically, the classification results of oil films with different thicknesses should be good based on the HSI. It can be seen from Figure 13 that the brightness temperature of oil film with different thickness is different, which increases with the increase of oil film thickness, and the brightness temperature difference is greater than 1°C. In theory, the addition of thermal infrared features will enhance the recognition ability of oil films with different thicknesses. The combination of hyperspectral and thermal infrared remote sensing can be used to classify and identify oil film with different thickness on the sea surface.

Here, in order to explain the possible classification results of WO with different thicknesses at the same concentration using hyperspectral and thermal infrared remote sensing, the spectral and brightness temperature characteristics of WO emulsions with different thicknesses at the same concentration are analyzed and discussed. It can be seen from Figure 12 that, taking WO emulsions with a concentration of 60% as an example, the spectral curves of WO emulsions with different thickness at the same concentration are generally consistent. In the spectral range before 1200 nm, the spectral reflectance of WO emulsions decreases gradually with the increase of oil film thickness; In the near-infrared range after 1200 nm, the spectral reflectance of WO emulsions increases gradually with the increase of oil film thickness.

In the spectral range of airborne hyperspectral images (450~950 nm), the spectral responses of WO emulsions with different thicknesses at the same concentration are different. Therefore, theoretically, the classification results of oil films with different thicknesses should be good based on the HSI. It can be seen from Figure 13 that the brightness temperature of WO emulsions with different thicknesses at the same concentration is different, which increases with the increase of oil film thickness. However, the brightness temperature difference is not all greater than 1°C, and there may be mixing between oil films with different thicknesses. In conclusion, theoretically, the addition of thermal infrared features may enhance the recognition ability of WO emulsions with different thicknesses at the same concentration.