The segmentation network S takes image X as the input and produces the segmentation prediction Ŷ of the same size as the input. Here, X can be either the source domain image X_s, or the target domain image X_t. Hence, the source domain prediction Ys^=S(Xs), and the target domain prediction Yt^=S(Xt). From the perspective of GAN (Goodfellow et al., 2014) framework, the segmentation network S can be thought as the generator module.

We train S to generate the source domain segmentation predictions Ys^ to be similar to the source domain ground-truth labels Y_s. Since in Unsupervised Domain Adaptation (UDA) the ground-truth labels are not available for target images, we can not compute any supervised pixel-level loss for target predictions. In practice, we found that combining dice-coefficient loss and entropy minimization loss is more effective than simply using binary cross-entropy loss for nuclei segmentation tasks. Therefore, we define segmentation loss L_seg as:

where Ys′ and Ys′^ are the flattened Y_s and Ys^, respectively.

Here, question may arise that why we are using single segmentation network S in NuSegUDA although we have two different domains. Since we are particularly looking for nuclei from both domain images, it is very unusual to use multiple segmentation networks. Additionally, using two segmentation networks would increase the number of learnable parameters which would slow down the training process in turn. Therefore, single segmentation network helps to prevent the memory issues and training latency in NuSegUDA.

Training the segmentation network S with only the annotated source data teaches S to make accurate predictions for source images. However, this segmentation network may generate incorrect outputs for target images as there are visual discrepancies between source images and target images (see Figure 2). This visual gap between domains causes the domain shift problem. According to our aforementioned observation that nuclei segmentation outputs are domain-invariant, we require S to produce target domain predictions as much as close to the source domain predictions. In other words, we want to make the distribution of target predictions Yt^ closer to the distribution of source predictions Ys^. For this reason, we utilize Prediction Discriminator D_P in NuSegUDA, and we define the prediction adversarial loss as:

where Ŷ_t = S(X_t), and H_p and W_p are height and width of the prediction discriminator output D_P(Ŷ_t). The details of the Prediction Discriminator D_P is discussed in Section 3.2.3.

The prediction adversarial loss in Equation (4) helps S to fool the prediction discriminator so that it considers Yt^ as source domain segmentation outputs. Segmentation loss and the prediction adversarial loss jointly guide S to generate target domain predictions Ŷ_t which look similar to source domain ground-truths.

As we mentioned earlier, the segmentation network S produces domain-invariant predictions for both domains. In other words, we want to generate the target domain predictions in a way so that they become similar to the source domain predictions. However, it is highly probable that the target predictions are not well-correlated with corresponding target input images. In this scenario, the ability of reconstructing the images from the predictions with similar visual appearance as input images will ensure that there is a correlation between the input image and segmentation output.

To ensure that our target domain predictions spatially correspond to the target domain images, reconstruction network R is used in NuSegUDA. In a similar way to Xia and Kulis (2017), we consider the segmentation network S and the reconstruction network R as an encoder and a decoder, respectively. R reconstructs target images from the corresponding predictions. Thus, S and R altogether works as an autoencoder.

Using our reconstruction network R, we first reconstruct target input images X_t from Ŷ_t. Then, we calculate the reconstruction loss as:

where, R(Ŷ_t) is the output of reconstruction network for Ŷ_t, and C is the number of channels of input image X_t.

Although we use above reconstruction loss to reconstruct the target domain images from its predictions, the reconstructed images may have very different textures and styles (for both of nuclei and background) than the original images (see Figure 4). The reason is that the pixel-wise reconstruction loss L_recons (in Equation 5) can not capture the overall pixel distribution of target domain images. To solve this issue, in addition to L_recons, we also utilize an Image Discriminator D_I to distinguish the original images and the reconstructed images. To train R and S to generate original-alike reconstructed images, we define image reconstruction adversarial loss as:

where Xt~=R(Ŷt), and H_i and W_i are height and width of the image discriminator output DI(Xt~). This adversarial loss L_advI trains R and S to reconstruct target domain images of similar distributions (in terms of texture, style, color distribution, etc.) to the original images from target domain.

In NuSegUDA, L_advP helps the segmentation network S to generate target predictions Yt^ to be similar to the source predictions Ys^. And, due to L_advI, reconstruction network R learns to reconstruct target images (i.e., Xt~) which are very similar to the original target images in terms of texture, style, color distribution, etc. In other words, S maps both domain images (i.e., X_s and X_t) to a common prediction subspace ℝpn, and from ℝpn R reconstructs the images in target domain. Therefore, using S and R we can translate source domain images X_s to the target domain. Thus, target translated source domain images X_s→t = R(S(X_s)). Figure 4 shows the visualizations of the impacts of image reconstruction adversarial loss L_advI on X_s→t. Finally, we train the segmentation network S with {(X_s→t, Y_s)} using following L_trans loss which is a combination of dice-coefficient loss and entropy minimization loss:

where Ys~=S(Xs→t). And, Ys′ and Ys′~ are the flattened Y_s and Ys~, respectively.

We utilize two discriminators in NuSegUDA: Prediction Discriminator (D_P) and Image Discriminator (D_I). Prediction Discriminator distinguishes between source domain outputs and target domain outputs, whereas Image Discriminator distinguishes between original images and reconstructed images. We discuss the details of both discriminators in the following.

Prediction discriminator As our goal is to generate similar predictions for both of source images and target images, we incorporate prediction discriminator D_P in NuSegUDA. This discriminator takes source domain prediction or target domain prediction as input, and then distinguishes whether the input (i.e., prediction) comes from the source domain or the target domain. To train D_P, we use following cross-entropy loss:

where z_p=0 when D_P takes target domain prediction as its input, and z_p=1 when the input comes from source domain prediction.

Image discriminator We use image discriminator D_I in NuSegUDA so that the reconstructed image distribution becomes similar to original image distribution. The input of D_I is either the original target image or the reconstructed target image. Then, D_I distinguishes whether the input is original or the reconstructed one. Similar to D_P, we use following cross-entropy loss to train D_I:

where z_i=0 when D_I takes reconstructed target image Xt~ as its input, and z_i=1 when the input comes from original target images X_t.

In addition to image-level domain adaptation at the outputs, we also apply feature-level domain adaptation in NuSegUDA to reduce the domain gap in the feature space. We assume that, our segmentation network S is composed of an encoder S_E and a decoder S_D (i.e., S = S_EoS_D). Here, the encoder S_E works as a feature extractor. Due to the discrepancy of input statistics across domains, there is also a shift of feature distribution in the feature space spanned by S_E. Similar to Toldo et al. (2021), we utilize a clustering loss at the feature-level to serve as a constraint toward a class-conditional feature alignment between domains.

Given source image X_s and target image X_t, we first extract the features F_s = S_E(X_s) and F_t = S_E(X_t). Then, the clustering loss is computed as:

where f_i is the feature vector corresponding to a spatial location of F_s or F_t, ŷ_i is the corresponding predicted class, and C is the set of semantic classes which is {0, 1} for our nuclei segmentation problem. To compute ŷ_i, the segmentation prediction Ŷ is downsampled to match the spatial dimension of F. We set the function d(.) to L1 norm. In Equation (12), c_j denotes the centroid of semantic class j, which is computed using following formula:

where δ_j,ŷi is equal to 1 if ŷ_i = j, and to 0 otherwise.

In Equation (12), the clustering loss is composed of two terms: the first term measures how close the features are from their respective centroids, and the second term measures how far the semantic class centroids are from each other. Therefore, according to the first term, the feature vectors of the same class from same or different domain are tightened around the class feature centroids. And, because of the second term, features from different classes gets a repulsive force applied to feature centroids which moves them apart.

Thus, we minimize the following total loss when training our segmentation network S and reconstruction network R:

where, λ_advP, λ_recons, λ_advI, λ_trans, and λ_cl are the weights to balance corresponding losses.

Experiment-1 (KIRC → TNBC) In our first experiment, we choose KIRC as source domain and TNBC as target domain, denoted by KIRC → TNBC. In our experiment, we choose U-Net (Ronneberger et al., 2015) as the representative of Convolutional Neural Network (CNN) based approaches. Fully-supervised segmentation model U-Net gives an insight of how it performs when directly applying transfer learning (i.e., training with only KIRC and then test it on TNBC without any modifications). AdaptSegNet (Tsai et al., 2018) and OrClEmb (Toldo et al., 2021) represent generic Unsupervised Domain Adaptation (UDA) models. DA-ADV (Dong et al., 2018), CellSegUDA (Haq and Huang, 2020), EndoUDA (Celik et al., 2021), SelfEnsemb (Li et al., 2021), and MaNi (Sharma et al., 2022) are chosen as the representatives of UDA model for biomedical image segmentation.

From Table 1, we see that source-trained U-Net gives the lower-bound of experimental performance (see first row of Table 1) which happens because of the visual domain gap between source training images and target test images, also known as domain shift problem. We see that, our proposed UDA model NuSegUDA outperforms all UDA baseline models in terms of IoU%, Dice score, and Hausdorff distance. Specifically, NuSegUDA has 1.28 and 0.42 higher IoU% than best generic UDA baseline OrClEmb, and best biomedical UDA baseline MaNi, respectively. Figure 5 shows the visualization results of CellSegUDA, SelfEnsemb, MaNi, and NuSegUDA. In Table 1, the second to last row [i.e., U-Net (target-trained)] shows the upper-bound of experimental performance (i.e., training U-Net with TNBC-train and testing it on TNBC-test).

Experiment-2 (TNBC → KIRC) We conduct another experiment in the similar way to experiment-1 by selecting TNBC as source and KIRC as target domain. This experiment also reflects the excellence of NuSegUDA compared to other approaches in terms of segmentation accuracies (see last three columns of Table 1).

Experiment-1 (KIRC → TNBC) In experiment-1, we assess our Semi-Supervised Domain Adaptation (SSDA) method NuSegSSDA for KIRC → TNBC. Table 2 shows the experimental performances of NuSegSSDA. For this experiment, the source dataset KIRC is the same as UDA experiments. However, now we treat TNBC as partially labeled. We train NuSegSSDA considering 10%, 25%, 50%, and 75% images from TNBC-train dataset have annotations available. Then, testing on TNBC-test gives us increasing IoUs and Dice scores, and decreasing Hausdorff Distances. This happens because more false negative nuclei can be identified and some false positive nuclei can be removed by NuSegSSDA as we train it with more target annotations (see Figure 6). We compare NuSegSSDA with fully-supervised model U-Net (Ronneberger et al., 2015), and baseline biomedical SSDA model CellSegSSDA (Haq and Huang, 2020) to demonstrate the superiority of our proposed SSDA model. To train U-Net, we combine full KIRC dataset with the same 10%, 25%, 50%, and 75% of TNBC-train we chose to train NuSegSSDA. We observe that, the accuracy of NuSegSSDA approaches to the upper-bound (only lower by 1.35 IoU%) as we train with more annotations from target domain.

Experiment-2 (TNBC → KIRC) In our second experiment, we select TNBC as source and KIRC as target domain. The second experiment also demonstrates the excellence of NuSegSSDA compared to U-Net (Ronneberger et al., 2015) and CellSegSSDA (Haq and Huang, 2020) (see last three columns of Table 2). Similar to experiment-1, for the second experiment we again see that the segmentation accuracies of NuSegSSDA increase when more target images are annotated.

To verify the robustness of proposed UDA framework, we perform extensive ablation studies on the adaptation of NuSegUDA from KIRC to TNBC, and from TNBC to KIRC. First, we examine the contribution of each loss to the final IoU%, Dice score, and Hausdorff Distance; then, we investigate the effects of different segmentation network backbones on NuSegUDA.

Effectiveness of losses The contribution of image adversarial loss L_advI, target-translated source supervised loss L_trans, and clustering loss L_cl to our proposed NuSegUDA model is shown in Table 3. We see that, simply applying only L_advI or L_cl to CellSegUDA (Haq and Huang, 2020) gives little better performance than CellSegUDA alone. However, when we apply only target-translated source supervised loss L_trans to CellSegUDA, the performance is inferior due to the absence of L_advI loss. Without applying image-adversarial loss L_advI, target-translated source images X_s→t looks very different from the target-domain images in terms of texture, style, color distribution, etc. (see Figure 4). As a result, the performance of the model (i.e., CellSegUDA w/ L_trans) decreases when trained with these X_s→t images.

Similarly, NuSegUDA w/o L_advI gives much worse performance than NuSegUDA which happens because of training NuSegUDA with less-realistic target-translated source domain images. This again validates the effectiveness of L_advI on NuSegUDA. Finally, with all the proposed losses enabled, we achieve the best performing model NuSegUDA for both of the experiments which demonstrates the combined impact of newly proposed image adversarial loss, target-translated source supervised loss, and clustering loss on NuSegUDA. Figure 7 shows the visualization results of NuSegUDA w/o L_advI, NuSegUDA w/o L_trans, NuSegUDA w/o L_cl, and NuSegUDA.

Impacts of different segmentation networks In NuSegUDA, we use U-Net (Ronneberger et al., 2015) as the backbone segmentation network. We also assess the model performance by replacing the backbone segmentation network with two more frequently-used Convolutional Neural Network (CNN) based approaches: FCN (Long et al., 2015) and UNet++ (Zhou et al., 2018). As mentioned earlier, CNN based approaches are still the dominant ones for semantic segmentation of nuclei. However, due to the intrinsic locality nature and limited receptive fields of convolution operations, CNN based models may be incapable of capturing the global context of the input (Chen et al., 2021; Jia et al., 2021; Zheng et al., 2021). To this end, we explore the feasibility of Transformers, an alternative to CNNs, as the backbone segmentation network in NuSegUDA. Transformer mainly utilizes self-attention mechanism to extract inherent features (Tay et al., 2020), and due to this self-attention mechanism, transformers are powerful at modeling the global context of an input (Zheng et al., 2021). To examine the effectiveness of Vision Transformer based model, we replace U-Net in NuSegUDA with TransUNet (Chen et al., 2021) which basically combines a hybrid CNN-transformer encoder architecture with a decoder.

Table 4 shows the quantitative results of using different segmentation networks in NuSegUDA. We see that, among CNN-based models, UNet++ and U-Net outperform other CNN approaches in Experiment-1, and Experiment-2, respectively. We also see that, Transformer-based model TransUNet does not give any better accuracy than U-Net and UNet++ for both of the experiments. This happens due to our small-sized training datasets, because Vision Transformers (VT) need lot of data for training, usually more than what is necessary to standard CNNs (Liu et al., 2021).