Compared to the UNet and improved versions of U-shaped networks, this paper selects the Token-MLP module to replace the conventional combination of convolution, batch normalization, and ReLU, as shown in Figure 2 . By incorporating the advantages of the Swin transformer (34) and axial attention (35), this module integrates two shifted MLP modules to independently shift features along the height and width dimensions, partitioning them into distinct partitions and performing positional shifts along the specified axis. The objective of this design is to introduce local contextual information and enhance the module’s perceptual capability in feature processing.

The Token-MLP module operates by translating and projecting features, transforming them into a series of tokens. Initially, a convolutional kernel with a size of 3 is used to perform translation operations on the features, while adjusting the channel count to the embedding dimension E, where E represents the number of tokens. Subsequently, these tokens are processed through a shifted MLP module. By shifting features along both the height and width dimensions and dividing them into different partitions, the module effectively introduces local context and enhances its ability to perceive local features through the creation of random windows.

Furthermore, the Token-MLP module introduces improvements through the integration of residual connections. Within the shifted MLP module, residual connections are incorporated by adding the original token features as residuals to the final output. Such residual connections facilitate gradient propagation, addressing issues such as gradient vanishing and explosion. Moreover, they enable the network to better learn low-level features and enhance the expressive capacity of the network.

The PanNuke dataset consists of H&E-stained images of 19 cell types, as shown in Figure 3 . Due to the variations in cell types, the cell nuclei exhibit significant differences in shape, size, and color, especially in shape. Some cell nuclei are circular ( Figures 3A, B, E ), some are elliptical ( Figure 3D ), and there are even filamentous cell nuclei ( Figure 3C ). The ISIC2018 dataset also showcases the diversity of skin lesions. Some skin lesions have lighter colors ( Figure 3I ), while others have darker shades ( Figures 3H, F ). Additionally, certain lesions may be affected by hair and artificial markings ( Figures 3G, J ). Therefore, achieving high computational accuracy and robustness for the network is a major challenge.

To further enhance the accuracy of semantic segmentation, Drawing inspiration from spatial attention mechanisms (36) and cross-channel attention mechanisms (37), this study proposes a MFE module that extracts high-level and low-level semantic information in the network’s decoder. In this module, we integrate spatial attention mechanisms, cross-channel attention mechanisms, and asymmetric convolutions to enhance the network’s ability to extract semantic features, identify common characteristics among different types of cells or skin lesions, and improve the robustness of network segmentation.

As illustrated in Figure 4 , the MFE module framework comprises a total of five branches, with two branches utilizing skip connections and an additional two employing asymmetric convolutions. A combination of 1 × 1 convolution, 1 × 3 convolution, 3 × 1 convolution, 1 × 5 convolution, and 5 × 1 convolution is applied to the feature x to capture semantic information of objects with diverse shapes. The extracted feature information is fused and input into the Squeeze-and-Excitation (SE) module (37), establishing relationships and dependency models among channels. Subsequently, the feature map undergoes non-linear transformations through a 1x1 convolutional layer, generating four separate branches C i 1 , C i 2 , C i 3 , C i 4 , as depicted in Equation 1.

Where x represents the semantic features output by the Token-MLP module on the decoding side. W 1 , 1 T , W 1 , 3 T , W 3 , 1 T , W 1 , 5 T , and W 5 , 1 T respectively denote 1 × 1 convolution, 1 × 3 convolution,   3 × 1 convolution,   1 × 5 convolution, and 5 × 1 convolution. F SE ( x ) represents the Squeeze and Excitation operation. * denotes the convolution operation. The semantic information extracted from the four branches is subsequently connected and re-encoded through two spatial attention mechanisms, resulting in the generation of O i 1 and O i 2 , as illustrated in Equation 2.

Where σ 1 ( x ) = max ( 0 , x ) corresponds to the ReLU activation function. σ 2 ( x i , c ) = 1 1 + exp ( − x i , c ) corresponds to the sigmoid function.  W 3 , 3 T respectively denote 3 × 3 depthwise convolution. Finally, non-linearity is enhanced through a 1 × 1 convolution, followed by element-wise multiplication with the input feature map, and then processed through another 1 × 1 convolution to obtain the ultimate output feature map, denoted as O ^ i , as illustrated in Equation 3.

In order to accurately segment and predict target objects, the standard CNN architecture gradually downsamples the feature map grid to capture semantic contextual relationships. However, for small objects with significant shape variations, reducing false positive predictions solely through skip connections becomes challenging. Inspired by the Attention Gate (30), this paper proposes a DA block to replace the skip connection part of the U-shaped network. As shown in Figure 5 , in contrast to the Attention Gate ( Figure 5A ), the DA block adopts a symmetrical structure ( Figure 5B ) that not only focuses on the coarse-grained semantic information from the encoder end but also pays attention to the fine-grained semantic information from the decoder end. Additionally, the introduction of DWConv enhances the accuracy and generalization ability of the model while encoding positional information of the encoded features. According to the study by (38), the convolutional layers within the MLP block sufficiently encode positional information and outperform standard positional encoding methods. Moreover, DWConv has fewer parameters and computational costs, making it relatively computationally efficient.

Specifically, firstly, the features x l and g undergo non-linear transformations through consecutive 1 × 1 convolutions and 3 × 3 DWConv. The transformed vectors of x l and g are then added together, and the result is passed through the ReLU activation function to retain significant activations. Subsequently, the feature map is propagated through a 3 × 3 depthwise convolutional layer to generate C i , as shown in Equation 4.

Where x l represents the coarse-grained semantic information gradually extracted through layer-wise processing of local information by the Token-MLP module, while g represents the fine-grained semantic information extracted at the decoder end by the MFE module. W x , 1 T and W g , 1 T denote 1 × 1 convolutions, and W x , 3 T , W g , 3 T and W 3 T represent 3 × 3 depthwise convolutions. The term b i represents the bias term. Additive attention (39) is employed to obtain gating coefficients, which has been experimentally shown to achieve higher accuracy compared to multiplicative attention (40), despite being computationally more expensive.

Subsequently, we apply the sigmoid function to transform C i into the non-linear space, as shown in Equation 5.

The DA block is characterized by a set of parameters Θ a t t , including W x ∈ ℝ F l × F i n t , W g ∈ ℝ F g × F i n t , ψ ∈ ℝ F i n t × 1 , bias terms b ψ ∈ ℝ , b g ∈ ℝ F i n t . The features x l and g are nonlinearly mapped to the R-dimensional intermediate space, which is called attention based on vector cascading (41).

Finally, we compute the element-wise multiplication between the input feature map and the attention coefficients to obtain the final output of the DA block, as shown in Equation 6:

Where i and c represent the spatial and channel dimensions.

This paper used the International Skin Imaging Collaboration (ISIC2018) (32) and PanNuke (33). The ISIC2018 dataset is a large medical image dataset for skin lesion detection and classification tasks. It contains 2693 skin lesion images from various locations around the world, including malignant melanoma, benign melanocytic nevi, and other benign skin lesions. Each image is manually annotated by professional dermatologists and provides corresponding pathological diagnosis and classification information. PanNuke is an open-source Pan-Cancer histology dataset used for classifying and segmenting nuclei instances. The dataset presents a semi-automatically generated collection of exhaustive nuclei labels over 19 diverse tissue types. The dataset comprises 7825 images. Since the initial input images do not have the same size, before feeding the data directly into the network, we reformatted them into the same size. For the PanNuke dataset, we employed a cropping technique, which aims to fully retain essential information of the images, to crop all the initial large-size images into images of the same size 256×256. For the ISIC2018 dataset, we utilized an image resizing method to adjust all the images into the same 256×256 size.

We divided the dataset into training and testing sets in a 9:1 ratio. Moreover, within each training epoch, we utilized a random allocation technique at an 8:1 ratio to distribute data between the training and validation sets. More precisely, in the case of the PanNuke dataset, the training set encompassed 7043 samples, while the testing set comprised 728 samples. In the case of the ISIC2018 dataset, the training set consisted of 2424 samples, with 269 samples designated for the testing set.

This article used the Pytorch framework to develop MD-UNet. The training and testing platform is the Ubantu18.04 system, the graphics card is GTX1070Ti, and the video memory is 10G. This article uses the Adam optimizer with a learning rate of 0.001 and a momentum of 0.9. Since the neural network is very unstable at the beginning of training, a corresponding training strategy, namely cosine annealing learning, is added to reduce the risk of over fitting, so that the model has strong robustness and good convergence to occlusion. In the cosine annealing strategy, the learning rate is reduced in the form of a cosine function, which ensures a smoother learning rate reduction and prevents the model from failing to converge due to the learning rate dropping too fast. The minimum learning rate is 0.00001. The batch size is set to 32. This paper trains MD-UNet up to 500 times in total.

In medical image segmentation, the variations in the shape and size of the lesion of interest can cause the loss function to sharply drop to a local minimum during the training process. This occurrence may lead to suboptimal performance and an inability of the neural network to achieve the best segmentation. To address this concern, researchers mainly use cross-entropy (42) as the criterion to assess the proximity between the actual and predicted outputs. A lower value of cross-entropy delineates a more accurate prediction by the model. Additionally, the Dice coefficient is a standard metric to evaluate the segmentation effect and quantify the disparity between segmentation results and labels (43). Given the imbalanced nature of medical image datasets, the usage of Dice loss (DL) as a loss function is prevalent in segmenting ROI lesions and handling background imbalances. DL effectively reduces segmentation bias. The Equations of Binary Cross Entropy (BCE) and DL are as shown in Equation 7 and Equation 8.

Where y represents the actual value and y ^ represents the predicted result. In this paper, MD-UNet is trained using a combination of binary cross-entropy (BCE) and dice loss. The Equation for the loss L between the predicted value y ^ and the target value y is shown in Equation 9.

This paper compares the performance of MD-UNet with recent widely used medical image segmentation frameworks. The parameters compared in the experiment are IoU, Dice, number of parameters and computational complexity. Among them, the computational complexity is calculated according to the number of floating-point operators (FLOPs), and the Equation is shown in Equation Equation 10 and Equation 11.

Where T P , T N , F P and F N respectively represent True Positives, True Negatives, False Positives and False Negatives.

The results of this study are presented in Table 2 . It can be seen that our proposed network exhibits significant improvement in segmentation accuracy compared to other networks, measured through Dice, IoU, parameter count, and GFLOPs (highlighted in bold in Table 2 ). Specifically, when compared to the state-of-the-art network Cenet, MD-UNet achieves a higher IoU and Dice of 4.86% and 4.84%, respectively, on the PanNuke dataset. On the ISIC2018 dataset, MD-UNet achieves a higher IoU of 4.17% and Dice of 2.31% compared to Cenet. Furthermore, in terms of computational efficiency, our proposed network demonstrates the lowest GFLOPs value of 0.241, whereas Cenet has 8.9, MedT has 21.245, and UNet has 55.840. Regarding parameter count, our proposed network has only 0.73M parameters, whereas Cenet has 29M parameters, MedT has 1.60M parameters, and UNet has 31.13M parameters. These findings indicate that MD-UNet not only outperforms other networks in terms of segmentation performance but also stands out as the most lightweight network, which refers to a network that requires fewer parameters and lower computational complexity.

The runtimes of different methods are presented in Table 3 . From the table, it is evident that MD-UNet has the shortest runtime at 15.92ms (highlighted in bold in Table 3 ), while Cenet, UNeXt, and MedT exhibit runtimes of 18.71ms, 17.07ms, and 89.35ms. These findings highlight MD-UNet’s superior runtime performance, rendering it a more real-time and efficient option.

Figure 6 shows the IoU score plotted against the number of parameters or GLOPs. The graph reveals that MD-UNet outperforms other methods in terms of segmentation performance, computational complexity, and the number of parameters. To illustrate the improvement of our model compared to the baseline, we provide qualitative comparisons of the ISIC2018 and the PanNuke datasets in Figure 7 . The results indicate that MD-UNet generates segmentation predictions with a competitive edge detail that is closer to the ground truth than that of other methods.

To provide a visual comparison between MD-UNet and various state-of-the-art networks, this study includes semantic segmentation result images and heatmaps for MD-UNet and other networks, as depicted in Figure 7 and Figure 8 . The first eight rows correspond to the experimental comparisons on the PanNuke dataset, while the last four rows represent the comparisons on the ISIC2018 dataset. From the figures, it is evident that MD-UNet demonstrates a strong consistency between the predicted boundaries and the ground truth for different cell types and skin lesions of varying shapes. Compared to other advanced networks, MD-UNet exhibits higher accuracy and smoother segmentation results.

Specifically, for cell nuclei segmentation, the first row reveals that MD-UNet successfully eliminates interference from external objects (e.g., circular objects). Furthermore, the second, third, and fourth rows demonstrate MD-UNet’s precise segmentation of elongated, circular, and elliptical cell shapes, respectively. Even for irregularly shaped cells, as shown in the seventh row, MD-UNet still displays a certain degree of segmentation capability. Regarding skin lesion segmentation, the ninth and tenth rows illustrate MD-UNet’s ability to achieve high-quality segmentation even in the presence of hair follicle interference. Additionally, the eleventh and twelfth rows indicate MD-UNet’s ability to achieve high-quality segmentation for objects of different colors.