In this paper, we utilize the pyramid semantic parsing network (PSPNet) to extract features and then refine those features from different scales via the pyramid pooling module. To boost the segmentation performance, we get an auxiliary prediction output, which is mainly the optimization of auxiliary objective function in the process of network training. The network not only includes effective global prior representations but also achieves good results in tissue micro-array (TMA) image segmentation.

For different segmentation tasks, the expert segmentation is achieved independently, but with the same real segmentation goal. The STAPLE algorithm is used to compare the differences of the final segmentation results based on a rapid interactive level set and hand contours for tumor segmentation (28). The algorithm segments images and calculates the probability of true segmentation simultaneously. The STAPLE algorithm not only takes the systematic deviation caused by the difference of different experts’ annotations into account but also evaluates the annotation quality of each expert. The algorithm can balance the two aspects well and then generate a fuzzy real annotation. We can extract more critical information with different weights of each part of the input, which makes the model to have more accurate judgments and reduces the calculation and storage of the model. The flowchart of the detailed annotation of TMA core images provided by different pathologists is shown in Figure 1 .

The output of the STAPLE algorithm is a picture with floating point values from 0 to 1, which represents the probability that the pixel point belongs to a specific segmentation target. The picture has the same size as the original image. Further, we want to extend it to multi-category situations. If each pixel has the maximum probability of belonging to a different category, the category label of the point is uncertain. At this point, we take the label of the most experienced expert as the true category label of the point. After preprocessing, the merged labels can be used for subsequent network training and verification.

Feature extraction is a key step in the field of computer vision and image processing, which can be used to extract image information and determine whether the points of each image belong to the useful features. The purpose of feature extraction is to divide the points on the image into different subsets, which are often represented as isolated points, continuous curves, or continuous regions. The quality of extracted features has a crucial impact on the performance of the training network (29).

In recent years, the way of feature extraction has developed from manual design to automatic extraction by the convolution neural network. Since AlexNet was proposed, researchers have devised a variety of convolutional neural network architectures to achieve automatic learning and extraction of features, such as VGG network (30), residual network (ResNet) (31), and densely connected network (DenseNet) (32). Different from classic convolutional neural networks, FCN can handle original input images from an arbitrary dimension and reserve spatial information. It classifies the original image pixel by pixel from the upper sampling and ignores the adjacent information when unpooling the low-resolution feature images. However, the main problem of FCN is that it cannot make good use of category information from a global scene. For the image classification task, the increase in network depth may bring additional optimization difficulties.

The residual block can solve this problem by using the long skip connection in each block and bring good performance. In the deep residual network, the latter layer mainly learns the residual thrown from the previous layer. Influenced by the most widely residual network, our research also uses ResNet as the skeleton network for feature extraction. ResNet is mainly composed of residual modules. Unlike the direct use of convolutional stacking, the residual module introduces residual learning, and hence the network can be made deeper and have more powerful capabilities of feature extraction. The skeleton network can effectively extract the tissue structure features using the prostate cancer pathological slice images and prepare for the next step of feature learning.

Global pooling has been widely used in the classical complex digital tissue pathological sections to get global image-level features (33). The pyramid pooling module (PPM) is a comparatively good way to fully utilize global information. To reduce the impact of the loss of contextual information between different subregions in the stage of training, some researchers proposed a hierarchical global prior, which contains variations among different subregions and numerous information with different scales (34, 35). Different levels of feature mapping created by the pyramid pool module are ultimately flattened and then connected to the full connection layer as input. We can utilize the global prior technique to eliminate the negative effects of fixed size constraints for the training of the convolutional network (36). The PPM collects different scales of information, which is more typical than global pooling (37). This multi-scale pooling can intuitively maintain global context information and global information of different scales better than single pooling.

After using the residual network as the feature extractor to extract the features, we need to further learn and process the extracted features to different sizes. These feature maps can be stacked into spatial pyramid feature maps. The pyramid feature map is subjected to different convolutions for feature learning and then resampled to restore the size of the input feature map. Then, these feature maps are combined with the input feature maps in a splicing manner. PPM can overcome the problems such as many parameters, difficult training, information loss, and overfitting and integrate features from four different pyramid scales. The gray module in PPM represents the feature map obtained after network training and continues to extract features with blocks having sizes of 1 × 1, 2 × 2, 3 × 3, and 6 × 6 pixels. We put these three grids on the feature map and can get 12 different blocks, and extract a feature from each block. Thus, 12 groups of features can be obtained. Obviously, the PPM can effectively combine features of different scales, including both deep-level high-level semantic features and low-level structural features. Accordingly, it can better learn and merge features.

In Figure 2 , different levels of output in the PPM contain different sizes of feature maps. The bold red highlight represents a single bin output. When the size of the pyramid layers set by the network is N, the corresponding global size decreases to 1/N of the initial size, which is sampled to the same size as the original feature map after low-dimensional feature mapping by bilinear interpolation. To better weigh the global feature weights, we add a 1 × 1 convolution layer after the pyramidal pooling layer. Finally, different levels of features are connected at different levels as the final global feature.

This auxiliary branch network undergoes operations such as convolution and upsampling to output a prediction result. Note that the loss function of the backbone network is L ₁ and the loss function of the auxiliary branch network is L ₂, where the loss function L of the entire PSPNet network can then be expressed as:

where α is a weight coefficient that balances the 0two loss functions, and both L ₁ and L ₂ are cross-entropy loss functions. Finally, we set α = 0.5.

Unlike the traditional backpropagation loss of its auxiliary blocking relays to the shallow layer of the network, we use two different loss mechanisms that can pass through all convolution layers for network calculation. The auxiliary loss can optimize the network learning without affecting the learning of the main branch. The combination of local and global information can effectively avoid information loss and make the diagnosis and prediction of diseases more reliable. During testing, we use the main branch with better optimization for final prediction. Specifically, we obtain the feature map by feeding the pathological slice image of prostate cancer into ResNet101. Subsequently, the feature map is fed into the PPM for multi-scale feature learning, and then the learned features pass through several convolution layers. Finally, we can get the final prediction result.

By constructing a feature pyramid to extract features of different sizes, each feature has rich image information, which improves the reliability of network feature extraction. The network model reveals the tradeoff between memory and accuracy and achieves good segmentation performance. Aiming at solving the problem of digital histopathological segmentation of prostate cancer, this method has high accuracy and robustness. Our method helps us get the champion of automatic prostate Gleason grading challenge 2019 which placed 1st on the MICCAI 2019 prostate segmentation benchmark, as well as the Vancouver Prostate Centre dataset.

In Figure 3 , we can see the differences at each gleason grade groups in detail. The histopathological data in this article are provided by the Vancouver Prostate Center, which are collected from different medical institutions and process tissue microarray blocks. It should be noted that the prostate tissue microarray is gained from patients with a suspicion of having prostate cancer. For the training dataset, the Gleason-level determination of the core image of each tissue microarray is associated with the most prevalent and second prevalent Gleason grading of expert annotations.

The 2019 MICCAI Gleason Competition provides annotations from six experts, and the annotations provided by each expert are not necessarily complete. Fortunately, the annotation by all annotators completely covers all the data. Therefore, using these annotations becomes a primary challenge. All TMA core images are annotated as benign and Gleason patterns 3, 4, and 5. Six pathologists drew regions (closed contours) on pathological images and labeled each area with grades. Although all of the TMA core images are annotated in detail by pathologists, none of them is complete. There are even two images in the expert’s annotation without a corresponding image in the training set. However, one real tag is needed for network training; we need to merge the tags of six experts to enhance the robustness. Only four of the six pathologists label all the images, while the other two pathologists label only 191 and 92 images. To make better use of all expert annotations, we use the STAPLE algorithm to build the finally “ground truth label” via merging the annotations provided by multiple annotators.

To demonstrate the effectiveness of our method, we participate the MICCAI Gleason 2019 Challenge ¹ . A total of 331 images (including 224 train images and 87 test images) are used from tissue microarray blocks. These images are manually annotated by professional dermatologists.

We use PyTorch for the distributed parallel training of algorithmic models on an Ubuntu high-performance graphic workstation. The learning rate decay strategy is η = η 0 ( 1 − n N ) β , where η ₀ = 0.002 is the initial learning rate, n is the current training rounds, N = 200 is the total training rounds, and β = 0.9.

Due to the large size of the prostate pathological slice image, it cannot be directly input to the network for learning. In this paper, we directly scale the short side of the input image to a size of 1,024 pixels and then randomly crop an 800 × 800 image patch as the input of the network. All samples are divided into five parts (each subset has an approximately equal number of samples). We repeat the entire process five times to avoid possible deviation of dataset partition during cross-validation. The final results are calculated by averaging five group results. To avoid network overfitting caused by insufficient data, we propose a data augmentation technique by randomly resizing and mirror training datasets.

Performance metrics such as accuracy, mean, entropy, and standard deviation were used to evaluate performance. In this article, we evaluate the network performance through the evaluation metrics such as the distance similarity coefficient (DSC), Jaccard Index (JA), Hausdorff distance (HD), Cohen’s kappa coefficient, and F1 score.

The distance similarity coefficient (DSC) is defined as follows:

where │·│ represents a set of pixels, and A and B represent the real label and segmentation result, respectively. The Hausdorff distance (HD) is defined as follows:

where X_S and Y_S represent the point set of the real label and the segmentation result, respectively. h(X_S , Y_S ) and h(Y_S , X_S ) can be calculated as follows:

where ║·║ represents Euclidean distance. The average surface distance is another distance evaluation index. The lower the HD value, the better the network performance. Cohen’s kappa efficient k is defined as follows:

Where p ₀ is the relative scoring consistency of raters, and p_e refers to the observed data used to calculate the hypothetical probability. If the scores of the two scores are the same, then κ = 1.

where TN, TP, FN, and FP represent true negative, true positive, false negative, and false positive, respectively.

The 2019 MICCAI prostate grading challenge provides a unique dataset and strict evaluation conditions of Gleason grading for the challenging task. During the experiment, each region in the pathological section is mapped to the Gleason’s mode, and the low-level mode corresponds to the nearly normal prostate tumor. The differentiation level with the largest area is registered as the most important differentiation value, while the differentiation level with the second largest area is registered as the secondary differentiation value. The third largest or less than 5% minority is ignored. The prostate grading challenge includes two different tasks. One is the prediction of pixel-level Gleason grade, the other one is the prediction of core-level Gleason grade. Task 1 is regarded as a segmentation task, and we utilize PSPNet to accomplish this task. For task 2, we do not train a different network but provide a prediction of task 1 according to the Gleason grading system.

For the training of the deep neural network, the amount of image information is associated with the size and number of input images. However, in the process of network training, the increase in network depth may bring additional optimization problems. According to the previous literature (38), FCN based on the residual network solves this problem via skipping connection in each block. We compare the four proposed different classical semantic segmentation networks, i.e., FCN (22), SegNet (39), U-Net (40), and DeepLabv3 (41), to verify the segmentation performance. The results are reported in Table 2 .

To maintain consistency, we compare the three networks based on ResNet or with a network structure like ResNet in our prediction of pixel-level Gleason grade and core-level Gleason grades. During the experiment, we quantify Cohen’s quadratic kappa statistics to compare the consistency among pathologists, and the annotator consistency between models and pathologists. The comparison of pixel-level grade prediction of different methods is shown in Table 3 .

As the doctors are concerned about the benignity and the malignancy of tumors at actual clinical practice, they are interested to make the distinction between high-risk cancer (Gleason patterns 4, 5) and low-risk cancer (Gleason pattern 3). Also, the classification results are based on high and low scores. Comparing the prediction results with the real labels, the core-level Gleason classification performance of each network is shown in Table 4 . The comparative results of the proposed method with several recently proposed methods are reported in Table 5 .

From Table 3 , we can know that the PSPNet used in this study is superior to other comparative methods, which is consistent with the scores of six experts (calculated by Cohen’s kappa coefficient k) or the macro-average and micro-average F1 scores. On the test cohort, the minimum and maximum k of each of the six experts respectively scored by PSPNet are 0.23 and 0.62, while the minimum and maximum k between the six experts are 0.38 and 0.70. Therefore, the consistency of the PSPNet score between each expert and Gleason grade at the pixel level is within the consistency range of these six experts’ scores.

As is shown in Tables 4 , 5 , the proposed PSPNet achieves the best performance with respect to various evaluation metrics for distinguishing benign from malignant. For distinguishing high and low score tumors, most performance metrics of the methods listed in the table are reduced, indicating that more difficulty existed to identify the high and low score tumors. However, PSPNet still acquires the best result, which further illustrates the stability and reliability of the method.