The SMS-PCNN model consists of three parallel CNN branches with different receptive fields. The three CNN branches have similar structures, so the first branch in the single-branch benchmark architecture network is taken as an example to illustrate the data flow after the image is inputted to the network. The first branch uses a 3 × 3 convolutional kernel with a relatively small receptive field. The schematic structure of this branch is shown as Figure 1.

The single-branch network design is inspired by the Resnet network (Xie et al., 2017) and consists of three parts. The first part is the CNN Head, which includes the convolutional layer, (group normalization (GN) layer, ReLu activation layer, and max pooling layer sequentially. The second and third parts are the two layers with similar structure, each containing two blocks. Through two layers, the feature maps will be inputted to the global average pooling layer and then be flattened into a one-dimensional feature vector.

The convolutional layer of CNN Head uses 64 convolution kernels of size 3 × 3 with depth 3 to filter the image of size 512 × 512 × 3. The stride is 2 pixels and the padding is 3 pixels. Then, the image goes through the GN layer and ReLu activation layer. Then, max pooling is performed on the feature maps. The size of pooling is 3 × 3, the stride is 2 pixels, and the padding is 1 pixel.

The basic architecture of the block in Layer is shown as follows.

A formula can express the output of the block (1)y = ReLu(F(x) + x′) where x and y are, respectively, the input and output of the block. F(x) is the residual mapping which the network needs to learn, and x′ is the output from the side road. As shown in Figure 2, the block contains two roads, the main road and the side road. After entering the block, x will be inputted to the main road and the side road for data processing, respectively. In the main road, feature map x goes through the first convolution layer, the first batch normalization (BN) layer, the first ReLu activation layer, the second convolution layer, and the second BN layer and then becomes feature map F(x). To complete the summation in Eqn (1), the dimensions of x and F(x) must be equal. In the s side road, if the dimensions of F(x) are the same of x, no further data processing will be performed, which is called x′ = x. However, if the dimensions of the feature map output by the main road decrease, x needs to be processed by 1 × 1 convolution layer. The stride of 1 × 1 convolution kernel in Figure 2 is 2 pixels, and the padding is 1 pixel. The number of channels of the 1 × 1 convolutional layer depends on the number of channels outputting F(x) from the main road in the block. F(x) from the main road and x′ from the side road are activated by ReLu function after the summation of the pixels at the corresponding positions. The feature map y is the final output of this block.

There are two cascaded blocks in each layer with the same structure. The difference between the two blocks is that the resolution of the output F(x) in Block1 decreases and the number of channels changes. In Block2, the stride of the first convolution layer is 1 pixel. Therefore, the size of feature map and the number of channels do not change, and a 1 × 1 convolution operation is not required in Block2. Block1 uses a convolution kernel with the stride of 2 pixels instead of a pooling layer to reduce the resolution of the feature map. Springenberg et al. (2014) show that max pooling layer can be replaced by a convolution layer with a stride bigger than 1 pixel without loss of accuracy. Moreover, this can reduce the error to some degree in vast majority of networks. Therefore, we use stride of 2 pixels to compress the size of the feature map to ensure that the network not only reduces the dimension and removes redundant information, but also improves the discriminative accuracy.

Layer2 has nearly the same design as of Layer1, except that Layer1 has 128 convolutional channels and Layer2 has 64 convolutional channels. Moreover, when the image is put into the single-branch benchmark network, it passes through the CNN Head and 2 layers one by one. Then, the network uses global average pooling to further reduce the resolution of the image and flatten the high-dimensional feature map into a one-dimensional feature vector, pending further classification processing.

Based on the above single-branch architecture network of Branch1, other two branches with similar structures can be designed, which are Branch2 and Branch3. The main difference is that the size of the convolutional kernel used in Branch2 is 5 × 5, and the size of the convolutional kernel used in Branch3 is 7 × 7. Moreover, to make the feature map of each branch consistent in size for comparison and calculation, the padding of different branches is slightly different. Branch2 uses 5 × 5 convolutional kernels, and the padding of the first block in each layer is 2 pixels. Branch3 uses 7 × 7 convolutional kernels, the padding of the head convolutional layer is 4, and the padding of the first block in each layer is 3 pixels.

Generally, ship images are natural scene images, so the input of SMS-PCNN model is RGB images. Analyzing the ship dataset, we found that the difference in appearance between different categories of ships mainly lies in the hull structure of ships, while the difference in appearance between ships of the same category lies in the superstructures and equipment of ships.

Therefore, for different categories of ships, we should pay more attention to discriminating architectural differences and use convolutional kernels with large receptive fields, which are more conducive to extracting features such as hull structures that occupy large pixel areas. For light differences between ships of the same category, we should use convolutional kernels with small receptive fields to extract features such as ship equipment and superstructures that occupy small pixel areas. For example, there are huge differences between the hull of the Ariake class destroyer and landing ship, which are two different categories of ships, so the convolution kernel with large receptive field is more advantageous. While aircraft carriers contain many categories of ships such as Queen Elizabeth class, the Nimitz class, the main difference between them is the bridge and other special equipment that occupy small pixel areas. In this case, the convolution kernel with smaller receptive field is more advantageous.

Combining the above two considerations, the SMS-PCNN model uses different sizes of receptive fields to extract features of ship images at different levels, so as to achieve a high classification accuracy. The SMS-PCNN model has three parallel branches with different sizes of convolutional kernels that is 3 × 3, 5 × 5, and 7 × 7. The receptive fields of the branches increase, so that the features of the ship images can be extracted and classified in a multi-scale manner. The network structure of SMS-PCNN model is shown in the Figure 3.

The number of channels in the SMS-PCNN model has been adjusted two times. In the CNN Head, the convolution layer contains 64 channels. In Layer1, this number is expanded to 128. The reason is that the feature map shrinks and part of information loses during the process of extracting features from low-dimensional to high-dimensional features. To expand the amount of data flow between layers as much as possible, we expand the number of channels. However, the number of channels is reduced to 64 in Layer2. That is because the redundant information increases in the process of further abstraction of high-dimensional features. So, as to further improve the accuracy and reduce the redundant feature information, we decrease the number of channels.

After generated from three parallel branches with different sizes of receptive fields, feature vectors are linearly concatenated in the SMS-PCNN model. (2)Fused (featurevector)1 × p = [featurevector(branch1)1×q ,featurevector(branch2)1×q , feature vector(branch3)1 × q ], where p and q denote the dimensions of feature vectors. p is 192 and q is 64. Branch1 denotes the branch which uses 3 × 3 convolution, Branch2 denotes the branch which uses 5 × 5 convolution, and Branch3 denotes the branch which uses 7 × 7 convolution.

These fused feature vectors are processed by multi-layer perceptron neural networks (MLP neural networks), mapping feature vectors with different sizes of receptive fields into different classifications of ship images by decreasing the dimension layer by layer.

Suppose after flatten layer, MLP with L layer is used for feature classification, then the layer l has m^l neurons and the input vector is (3)al - 1 =  [a1l - 1, a2l - 1,…,amll - 1]T

a^l−1 is called the feature vector from layer (l − 1). The matrix of weight connected between neurons in layer (l − 1) and l is: (4)Wl = [w11lw21lw12l⋯w1ml−1lw22l⋯w2ml−1l⋮wml1l⋮wml2l⋱⋯⋮wmlml−1l] where wjkl denotes the weights connected between neuron k in layer (l − 1) and neuron j in layer l.

The bias vector of neurons in layer l is (5)bl =  [b1l,b2l,…,bmll]T then the feature vector a^l from neurons in layer l is (6)al = fl(Wl•al - 1 + bl) where the f^l( ) is the activation function used by neurons in layer l. After the signal enters the feedforward neuron network, it passes layer by layer and gets the output a^L in the end. The whole process in the feedforward neural network is actually inputting the feature vectors from PCNN into the composite function F(.; W, b). Finally, the output of the network a^L is as follows. (7)aL = F(x;W,b)= fL(WL•fL-1(…W2•f1(W1•x + b1) + b2…)+ bL) After the multilayer perceptron neural network, the SMS-PCNN model uses softmax function to normalize the vector a^L. Let the i-th element of the vector a^L be aiL, then the result obtained after aiL being processed by softmax function is (8)Si = softmax(aiL) = exp(aiL)∑jexp(ajL) After softmax function, aiL is mapped to the S_i.S_i satisfies the properties of the probability distribution, S_i ∈ (0, 1) and ∑iSi=1. Therefore, S_i can represent the prediction probability. By selecting the node with the largest prediction probability, we can obtain the final classification category.

The predicted probability S_i will also be inputted to loss function. In the SMS-PCNN model, we use cross-entropy loss function used. It is (9)Loss = -∑i = 1KpilogSi where K is the number of categories, and p_i denotes the indicator variable (takes the value of 0 or 1). If the predicted category of the sample i is the same as its true category, p_i takes the value of 1 and 0 otherwise. Cross-entropy loss function can measure the prediction performance of the model.

To address the problem of too many parameters in the SMS-PCNN model, the SMS-PCNN model uses L2 regularization to mitigate overfitting. Regularization is a method to reduce the complexity of the model by introducing penalty terms. The L2 regularization is to add the L2 norm of the parameters after the loss function: (10)L^oss = -∑i = 1KpilogSi + αΩ(w) where L^oss is cross-entropy loss function with L2 regularization, and w is the weight coefficient matrix, and the parameter α controls the strength of regularization. Ω function is the L2 norm, namely, the sum of squares of the weight coefficients: (11)Ω(w) = ||w||22

Generally, the distinguishing features of some ships are mainly concentrated in parts that account for a small proportion of the images, such as the bridge of aircraft carriers and the radar of frigates. Therefore, to improve the accuracy of the model, we need to retain as much information of images as possible and use images with large size to contain more information as much as possible. For images with large size, the batch size is reduced to fit the training. However, relatively low batch size will lead to bad performance of the batch normalization (BN) used in the SMS-PCNN model. In this case, Wu and Kaiming (2020) proposed group normalization (GN) to improve the dependence of BN on batch size and proved that when the batch size is <8, the error of using GN is smaller than BN.

The GN layer calculates the mean and variance based on the input data and then uses these two values to normalize the input data. Specifically, GN divides the channels of the layer into groups and uses the mean and variance of the data within the groups. (12)xi^ = 1σi(xi - μi)

xi^ is the feature map of the layer, and i = (i_N, i_C, i_H, i_w). N is the batch axis, C is the channel axis, H and W are the spatial height and width axis. μ and σ here can be calculated by the following equations (13)μi = 1m∑k∈Sixk (14)σi=1m∑k∈Si(xk - μi)2 + ϵ where ε is a small constant. S_i is a feature collection with a size of m used to calculate μ and σ. The solution of S_i in GN is the following equation. (15)Si = {k|kN = iN,⌊kCCG⌋ = ⌊iCCG⌋}

G is the number of groups. In the SMS-PCNN model, we pre-set G = 2. C is the number of channels, and C/G is the number of channels in each group. “⌊kCCG⌋=⌊iCCG⌋” indicates that i and k are in the same channel group.

When we used GN in all the normalization layers of the SMS-PCNN models, gradient explosion appeared. Therefore, we only use GN to normalize the output of the head convolution layer of each branch whereas BN is used in other layers, which achieved good results.

The model uses gradient descent to minimize the value of loss function and thus converge to the optimal solution. During the iterations performed by the gradient descent, the learning rate controls the learning progress of the model. To optimize the model learning rate, we use cosine annealing and Warmup (He et al., 2016) technique. Warmup is used to train with a small learning rate at the beginning of to make the network familiar with the data. As the training continues, the learning rate slowly increases and reaches the initial learning rate within a pre-set iteration. Then, we use cosine annealing to adjust learning rate.

The equation of decreasing learning rate is as follows: (16)ηt = ηmin + 12(ηmax - ηmin)(1 + cos(TcurTπ)) where η_t is the learning rate of the t-th iteration. η_min is the termination learning rate. η_max is the initial learning rate. T_cur is the current iteration. T is the total iteration. The key of the formula is cos(TcurTπ). As iteration increases, the value of cos(TcurTπ) decreases from 1 to −1, so that the learning rate η_max decreases to η_min.

The learning rate adjustment for the whole model training is shown in the Figure 4.

The learning rate is adjusted with iterations. In the first 2,709 iterations (3 epochs), warmup is used. As iteration gradually increases, the learning rate grows to 0.1, and then, this number decreases to 10⁻⁴ using cosine annealing.

On the Internet, ship images are limited in amounts and in low quality, so the ship dataset we compiled from open-source data is limited in size, which leads to overfitting to some degrees when we use the SMS-PCNN model for training. For this reason, we use two methods, label smoothing and dropout, to mitigate overfitting in training.

Label smoothing is a technique proposed by Sergey and Christian (2015) in the model of inception-V2 to mitigate overfitting. The traditional one-hot coding suffers from overconfidence, which leads to overfitting. To address overconfidence, label smoothing decays the item with probability 1 in one-hot, which reduces the weight of category of the ground-truth label in the calculation of loss function. The confidence of the decayed items in one-hot is equally divided into other items, so that each category has a certain amount of confidence, which can suppress overfitting. In the SMS-PCNN model, we add the label smoothing to cross-entropy loss function, and loss function is changed as follows. (17)Loss(p,q) = -∑i = 1Kpilogqi→L^oss(p′,q) = -∑i = 1Kpi′logqi where K denotes the number of categories, and p_i denotes the indicator variable (takes the value of 0 or 1). q_i is the predicted probability value. (18)pi′ = (1 - ε)pi + εu(k) where ε is a smoothing parameter, which is a pre-set hyperparameter. u(k) is a probability distribution and uniform distribution is used here, so u(k)=1K. The result of cross-entropy loss function through label smoothing L^oss(p′,q) is as follows. (19)L^oss(p′,q) = (1 - ε Loss(p,q) + ε Loss(u,q) In addition to label smoothing, we also use dropout to prevent the model from overfitting. In dropout (Hou et al., 2019), the training process randomly drops out some neurons in hidden layers and maintains the number of neurons in input and output layers. Then, we inputted data through the modified network to propagate forward, and the loss which is calculated by the network is propagated back. Suppose the multilayer perceptron neural network has L layers, and l ∈ {1 …, L} denotes the neuron in layer l. Let z^l denotes the input of the neuron in layer l, y^l denotes the output of the neuron in layer l, and W^l and b^l denote the weight coefficients and bias of the neuron in layer l. Then, the feedforward neural network can be expressed by the following equation. (20)z(l + 1) = W(l + 1)yl + b(l + 1) (21)y(l + 1) = f(z(l+1)) where f denotes the activation function. When we use dropout during the model training, the feedforward neural network becomes of the following form. (22)ril ~ Bernoulli(p) (23)ỹ(l) = r(l) * y(l) (24)z(l+1) = W(l + 1)ỹl + b(l + 1) (25)y(l + 1) = f(z(l + 1)) where the probability p is a pre-set hyperparameter. r^l is a random variable that obeys the Bernoulli distribution and is designed to randomly generate a vector of 0 or 1 with probability p. Probability p denotes the proportion of the elements of this vector taking the value of 0. r^l is to sample each layer and multiply y^(l) which is the output of that layer element by element. y^(l+1) denotes the final result of Dropout.

During the prediction of the model, the weight parameter of each neural unit is multiplied by the probability p, wtest(l) = pW(l).

In the SMS-PCNN model, we used the dropout before two fully connected layers and set the probability p to be 0.3, which means that 30% of the neurons were dropped out randomly, and the experiment achieved a good accuracy improvement.

We use the SMS-PCNN model optimized by three tricks to conduct experiments. Samples of 12 categories were trained iteratively using 175 epochs with the batch size of 4. The model was trained iteratively using the stochastic gradient descent (Sebastian, 2016). Cosine annealing and warmup optimization techniques were used on the learning rate. After warmup, the learning rate is initially set to 0.1 and finally decreased to 10⁻⁴. The weight of label smoothing is set to 10⁻³. The SMS-PCNN model uses L2 regularization and its weight is set to 10⁻³. MLP uses two dropouts, both set to discarding 30% of the neurons. The batch normalization, group normalization, and MLP in the model all use the correlation functions that come with PyTorch. After being preprocessed, the natural scene images with the adjusted resolution of 512 × 512 are inputted to the SMS-PCNN model for training.

Table 2 shows the architecture of SMS-PCNN model for a ship dataset with resolution of 512 × 512. Among them, the images are processed by Branch1, Branch2, and Branch3 in parallel and then jointly processed by average pooling. Finally, the feature vectors from three branches are concatenated together. The concatenated feature vectors with 192 dimensions sequentially go through the input layer, the hidden layer with 64 neurons and the output layer with 12 neurons of the MLP neural network. Then, these vectors pass through the softmax function, and the results of classification are outputted.

The “64” in Conv1 denotes the number of convolutional kernels. The “128” in Layer1 and 64 in Layer2 both denote the number of convolutional kernels. The “×2” of each Branch in Layer1 and Layer2 denotes that the two cascaded blocks using the same parameters. Take the parameters of fully connected layer1 (192, 64) as an example, “192” indicates the dimension of the input vector and “64” the dimension of the output vector.

Generally, the indicator to evaluate the classification effectiveness of a classifier is its accuracy. The classification accuracy is the proportion of correctly predicted results to the total sample.

As seen in Figure 6, the SMS-PCNN model, which combines all tricks, converges faster in training and has nearly reached the highest accuracy rate at the 100th epochs. With the increase of epochs, the learning rate gradually decreases and the training gradually tends to be stable, and the final model optimal results reach 84.79% accuracy.

The loss function in the SMS-PCNN model uses cross entropy loss function with label smoothing and is shown in the Figure 7. The improved loss function converges faster and more stably in the training set. At the end of the training, the model training finally converges. The decrease of the loss function on the valid set tends to be stable as the learning rate decreases and reaches the optimal result around 150th epochs. The confusion matrix of the SMS-PCNN model is shown in the following figure.

From the confusion matrix of the SMS-PCNN model on the valid set as shown in Figure 8, three points can be concluded. (1) The darkest areas of the confusion matrix are concentrated in the diagonal line, showing great classification performance. (2) Among all categories, landing ships, type 055 destroyers, and oil tankers have the highest classification accuracy. (3) Sovremenny class destroyers and Nimitz class aircraft carriers cannot be classified well. The reason why these two categories have a slightly worse classification performance than others is that there are diverse ships of these two categories with minor differences, resulting in the lack of obvious different features between categories.

T-sne (Laurens, 2014) clustering analysis of the SMS-PCNN model for the valid data is plotted in the Figure 9.

From the t-sne clustering analysis graph, we can find that the samples of each category are basically concentrated together, and the SMS-PCNN model can classify the samples well.

This section is divided into two parts for performance comparison. The first part is the performance comparison of the SMS-PCNN model with classical model frameworks and the model with single branch. The second part is the validation of the tricks of the SMS-PCNN model.

We compared our SMS-PCNN model with 4 state-of-the-art approaches: Resnet-50 (He et al., 2016), Alexnet (Alex et al., 2017), VGG-16 (Simonyan and Zisserman, 2014), and Resnet-18 (He et al., 2016). All the methods were compared on the valid set of our ship dataset. The accuracy comparison is presented in Table 3. The results in Table 3 show that the proposed SMS-PCNN model achieved the best results among all algorithms. All the state-of-the-art comparisons on the ship dataset are fine-tuned well. These comparisons use the same hyperparameters as the SMS-PCNN model we proposed.

In addition, we conducted experiments on network architecture with single branch to analyze the classification of ships. The experiments showed that the Branch1-CNN with small receptive field of 3 × 3 convolutional kernel achieved an accuracy of 75.53%, which is lower than the accuracy of the SMS-PCNN model. This demonstrates the necessity of designing a multi-branch structure.

Comparing with the confusion matrix and t-sne clustering analysis graph in Figures 10, 11, it can be clearly concluded that compared with the SMS-PCNN model, other models do not have well classification effect and have limitations in generalization performance. The confusion matrix shows that other models have low accuracy in discriminating the Queen Elizabeth-class aircraft carrier and the Chinese aircraft carrier Liaoning. The t-sne clustering analysis shows that the point representing Queen Elizabeth-class aircraft carrier and the Chinese aircraft carrier Liaoning are scattered and disorganized, showing a bad performance on clustering. The results of confusion matrix and t-sne clustering analysis are consistent and unified.

Basic SMS-PCNN means the SMS-PCNN model we used to perform an experiment without trick1, trick2, and trick3. These tricks are used for optimizing the network, including enhancing specific ship dataset applications, improving training learning rate and avoiding overfitting, respectively.

We designed the ablation experiments. The result is shown in Table 4.

It demonstrates that the accuracy of Basic SMS-PCNN model without any tricks is 77.28%, which is still higher than the other state-of-the-art approaches and Branch1-CNN in valid dataset mentioned above. This demonstrates the superiority of the model design principles and the basic framework structure. Meanwhile, the ablation experiments examined that the accuracy of the Basic SMS-PCNN model with trick1 increases by 2.16–79.44%. For the Basic SMS-PCNN (add trick1 and trick2), the accuracy increases by 5.96–83.79%. When using trick1, trick2, and trick3, the model accuracy increases by 7.51–84.79%. It can be seen that the use of three tricks did improve the accuracy of SMS-PCNN model and enhance the classification effect.

To further analyze the mechanism of the SMS-PCNN model and verify the effectiveness of parallelizing multi-scale convolutional branch and adjusting the number of channels, we visualize the convolutional kernels and the feature maps of tanker image through each convolutional module. The head convolution layer of each Branch in the SMS-PCNN model is a convolutional kernel with depth 3 designed for three channels of RGB. When visualizing, we combine these convolutional kernels into single convolutional kernel with color. The visualization of the 64 combined convolutional kernels of the head convolutional layer of each branch of the SMS-PCNN model is shown in the following figure.

From the visualization of the convolution kernels in Figure 12, we can find that the head convolution layer of Branch3 with 7 × 7 size convolution kernels mainly focuses on the edge features of images, so it can better extract features such as the hull of ships. The head convolution layers of Branch2 and Branch1 with 5 × 5 and 3 × 3 size convolution kernels focus on the fine-grained features of images, so it can better extract features such as ship superstructures and equipment.

We take the tanker image as an example to visualize feature maps that pass through each branch convolution module. In each branch, the tanker image passes through the CNN Head, Layer1, and Layer2, and the feature maps are shown in the following figure.

From Figures 13A–C, we can see that after the head convolution layer with different convolution kernels in three branches, the feature maps of the tanker image have different attention. The feature map from Branch3_Head with a larger receptive field is obviously coarser than that of Branch2_Head and Branch1_Head, which have smaller receptive fields. It shows that the feature map from Branch3_Head is more concerned with coarse-grained feature extraction, whereas Branch1_Head with the smallest receptive field is clearer in detail and is obviously more concerned with the fine-grained feature extraction.

Figures 13D–F show the feature maps of the tanker image after processing by the layer1 of each branch. Branch1_layer1 obviously retains more details and pays more attention to the fine-grained features of the ship. Branch1_layer2 has fewer details than Branch1_layer1 and focuses more on the moderate scale features such as the superstructures of the tanker. Branch1_layer3 focuses on the contours of the ship and the coarse-grained features of the hull structure of ships.

Figures 13G–I shows the feature maps of the tanker image after the layer2 of each branch. It can be clearly seen that there are huge differences between the feature maps after all the convolution layers of each branch. The feature map of Branch1 is clear and focuses on fine-grained features such as ship equipment. The feature map of Branch2 focuses on the superstructures of the tanker. The feature map of Branch3 pays attention to contours and the hull structure of the tanker.

By comparing the feature maps of different convolution of the three branches, we demonstrate the effectiveness of using multi-scale convolutional branch parallelization for ship images.

We also compared the feature maps of each convolutional layer of single branch. Taking Figures 13A,D,G as an example, after the processing of CNN Head, Layer1, and Layer2, feature maps are gradually abstracted from specific low dimension to high dimension, which is effective for image feature extraction. Besides, from Figures 13D–F, we observe that there is redundant clutter in part of the output from Layer1 of each branch. Therefore, the reduced numbers of channels in Layer2 can filter redundant feature maps from the previous layer. This verified the effectiveness of changing numbers of channels in the network.