The medical image I_s first can be divided into foreground image I_f and background image I_b by the segmentation mask I_m. I_m labels the foreground information as 1 and the background information as 0. Therefore, I_f and I_b can be formulated as follows:

Then, I_f and I_b are respectively fed into the foreground encoder Efi and the background encoder Ebi to extract the foreground feature Ffi and the background feature Fbi, where i = 1, 2, 3 denotes the layer index. Finally, Ffi and Fbi are input to the foreground decoder D_f and the background decoder D_b to reconstruct the foreground image If′ and the background image Ib′. The self-supervised foreground reconstruction loss L_f and self-supervised background reconstruction loss L_b focus on extracting foreground and background information of the medical segmentation image, which can be formulated as:

where ||·||₁ represents the l₁ norm.

As shown in Figure 2, Ffi and Fbi from each layer are fed into the adaptive feature modulation module Φai, thereby helping SIR to obtain more significant foreground and background features. Specifically, Ffi, Fbi and the initial pixel-level feature Fri first perform channel feature concatenation to generate the fusion feature F_u, and then the global semantic features are extracted by using global average pooling. Further, we utilize dual-stream convolutional blocks to generate calibration parameters α and β to guide Fri. This calibration process can be represented as:

Next, the calibrated pixel-level fine-grained feature F^ri is fed into the reconstruction decoder to reconstruct the medical image. Finally, the medical image reconstruction loss Ls ensures that the pixel-level fine-grained features can reconstruct a complete segmentation image, which can be expressed as follows:

where I_s denotes a medical image, and Ir′ represents a reconstructed medical image.

In Section 3.1, we obtain pixel-level fine-grained feature F^ri from FIIR. Specifically, as shown in Figure 3, I_s is first fed into the segmentation encoder Esi to extract the segmentation semantic feature Fsi. Then, Fsi and F^ri are input to the bi-directional fusion module Φbi to obtain a complete feature representation with strong semantics and rich details. Specifically, we compute respectively the cross-attention weights between Fsi and F^ri, thereby jointly modeling the complementary relationship between the reconstruction branch and the semantic branch. In this process, F^ri uses semantic clues to guide Fsi to enhance structural perception, while Fsi employs textural details to enhance the spatial resolution of F^ri. In particular, First, Fsi generates the query vector Q_s and F^ri generates the key-value pair (K_r, V_r). Similarly, Q_r is obtained via F^ri, and (K_s, V_s) is generated through Fsi. The two-stream cross-modal attention is computed as follows:

where d is the channel dimension of each attention head and Softmax is an activation function. Attn_r denotes the segmentation-guided attention map, and Attn_s represents the reconstruction-guided attention map. Subsequently, we adopt feature aggregation and residual concatenation to generate two enhanced features F~ri and F~si, which can be formulated as:

where C11 denotes one convolutional layer with 1 × 1 kernel. Finally, we fuse F~ri and F~si to generate the refined segmentation feature F̄si through concatenation and convolution operations, thus enhancing the feature representation ability.

Multi-branch vision mamba is constructed to force the model to mine high-level semantic information, thus improving the feature representation. Specifically, as shown in Figure 4, F̄s3 firstly is fed into E_M to perform flattening and normalization, thereby generating segmentation sequence feature N_s. Then, we divide N_s into four groups to learn the important representations of different sub-regions, which can be represented as:

Next, each subsequence is respectively fed into the weight-sharing Mamba module to perform state modeling, and then refine the representation by residual operations:

where γ is a scaling factor. M(·) denotes the Mamba function. Then, the updated subsequence N_j is performed to feature concatenation from the channel dimension, thus generating the enhanced sequence representation:

where Concat(·) denotes the feature concatenation operation. Subsequently, N~s is normalized and linearly transformed to project to the original feature dimension, which can be expressed as:

where LN(·) represents layer normalization, and Proj(·) indicates linear projection.

Therefore, we utilized the state-space mechanism of Mamba to capture long-distance contextual information. Then, multi-branch decomposition is used to enhance the feature representation between different sub-regions. In this way, multi-branch vision mamba establishes the dependency between global semantics and local details, thus helping the model to improve the segmentation accuracy of key objects.

To constrain the difference at the pixel level between the reconstructed image and the segmentation image, the image reconstruction head D_f, D_r and D_b adopt the reconstruction loss Lrec, which can be defined as follows:

where Ls, Lf and Lb denote foreground reconstruction loss, background reconstruction loss and medical image reconstruction loss, respectively.

The BCE loss Lbce aims to predict the per-pixel classification accuracy. The Dice loss Ldice can measure the overall overlap region between the prediction mask Is′ and the ground truth I_gt. Thus, we jointly Lbce and Ldice constrain the segmentation head S_h, which can be expressed as:

Finally, the total training loss can be expressed as:

GLAS (57) dataset consists of 165 microscopy images of colorectal adenocarcinoma tissue sections at stage T3 or T4, stained with H&E. Each image has a resolution of 128 × 128 pixels and is collected from a different patient. Due to variations in cancer progression, the lesions exhibit significant differences in shape and distribution. Meanwhile, since all samples originate from the same type of tissue, the surrounding environments are relatively consistent. Additionally, some cells are damaged or ruptured during the sampling process, resulting in large inter-cell variability. These factors make the dataset highly challenging. According to the official split, the training set contains 85 images and the test set contains 80 images. This dataset is mainly used to assess the model's capability in segmenting dense lesion regions and small targets.

ISIC2016 (58) and ISIC2017 (59) datasets were released by the International Skin Imaging Collaboration (ISIC) in 2016 and 2017, respectively. They were used as official datasets for the skin lesion analysis challenges held in those years. The goal of these datasets is to raise global awareness of skin disease diagnosis and to improve the detection of melanoma and other benign or malignant lesions. Both datasets contain a large number of samples and include various types of skin lesions. Due to the diversity of lesion types and the wide range of patient backgrounds, the samples show high variability in texture, color, and structure. In addition, some mild lesions look very similar to normal skin, which makes it hard to identify lesion boundaries. This increases the difficulty of the segmentation task. In this study, we evaluate the segmentation performance of our model using the ISIC2016 and ISIC2017 datasets. Both datasets follow the official training and testing splits: ISIC2016 includes 900 training images and 379 testing images, while ISIC2017 consists of 2,000 training images and 600 testing images. All images are resized to 256 × 256 pixels to ensure consistency during the experiments.

PH2 (60) is a public dataset designed for dermoscopic image segmentation and classification. It aims to support research on computer-aided diagnosis of melanocytic lesions. The images were collected at the dermatology department of Pedro Hispano Hospital in Portugal. All images were captured under the same conditions using the Tuebinger mole analyzer system with 20 × magnification. The dataset contains 200 dermoscopic images of melanocytic lesions, including 80 common nevi, 80 atypical nevi, and 40 melanomas. PH2 serves as a reliable benchmark for evaluating lesion detection, segmentation, and classification algorithms. In our experiments, all images were resized to 256 × 256 pixels. It is worth noting that we used PH2 as an external validation dataset. We tested it directly using the model trained on ISIC2016 to assess the effectiveness of our method and its potential for future clinical applications.

In the quantitative analysis, we adopt widely used evaluation metrics in the field of medical image segmentation. Specifically, we use Precision, Recall, F1, and Intersection over Union (IoU) to assess the performance of the proposed model. Here, TP denotes true positives, FP denotes false positives, TN denotes true negatives, and FN denotes false negatives. These metrics jointly provide a comprehensive evaluation of the model's accuracy and completeness from multiple perspectives.

Precision measures the proportion of true positives among all regions predicted as positive (e.g., lesion areas). It reflects the model's ability to control FP. In medical image segmentation, a high Precision means the model can avoid wrongly identifying normal areas as lesions, which helps reduce the risk of misdiagnosis. When Precision is high, most of the predicted lesion regions are actually correct, and the FP rate is low. This is especially important in cases with small lesions or strong background noise. In such situations, Precision is a key metric to evaluate how well the model limits over-segmentation. The formula is given as:

Recall measures the model's ability to detect all positive targets. It shows how many of the actual positive pixels are correctly identified. In medical image segmentation, a high Recall means the model can successfully detect most lesion areas, which helps reduce missed detections and is important for clinical diagnosis support. The formula is given as:

F1 is the harmonic mean of Precision and Recall. It is used to evaluate both the accuracy and completeness of the model. When there is a large gap between Precision and Recall, F1 provides a more balanced result. In segmentation tasks, F1 is especially useful for assessing model performance under class imbalance, such as small lesions against large background regions. A higher F1 indicates that the model achieves a good balance between accuracy and completeness. The formula is given as:

IoU is one of the most widely used metrics in image segmentation. It measures the overlap between the predicted region and the GT. It is defined as the ratio of the intersection area to the union area of the prediction and the GT. IoU directly reflects how well the predicted boundary matches the actual boundary. A higher IoU means the predicted region aligns more closely with the GT, indicating better segmentation accuracy. Compared to F1, IoU is more sensitive to small differences and is suitable for evaluating boundary localization. The formula is given as:

We use NVIDIA GeForce RTX 4090 GPU to train and inference the model. The network framework is Pytorch. EPOCH is set to 150, and Batch is 4. The optimizer is Adam that uses momentum strategy to steadily update the model parameters. We employ warm-up and cosine annealing schedulers to achieve slow startup in the early stages and fine convergence in the later. The initial learning rate is 1e-3 and gradually decays to 1e-5.

To evaluate the contribution of each module in the model, we conducted a systematic ablation study on the GLAS dataset. The quantitative results obtained after removing different components are presented in Table 5, and the corresponding visual segmentation outputs are shown in Figure 13, offering a clear view of how each module affects the final performance.

As can be seen from the visual results, w/o FIIR (1) leads to evident deficiencies along the object boundaries and causes incomplete structural predictions. This demonstrates that FIIR plays an important role in enhancing pixel-level detail and supporting the extraction of informative segmentation features. When the adaptive feature modulation module Φai is removed, and then replaced with feature summary (2) or concat (3), the model tends to produce over-segmentation. This is reflected in the increased number of false positives in the predicted maps. Although the recall remains relatively high, reaching 96.10 and 96.31 respectively, the precision drops significantly to 78.18 and 66.37, suggesting that the model becomes less capable of regulating foreground and background responses effectively.

Moreover, we adopt summary (4) and concat (5) operations to replace the bi-directional fusion module Φbi. The predicted structures remain mostly intact, but the boundaries are less precise, indicating that this module still contributes to enhancing local detail and structural consistency. Further, the removal of the multi-branch vision mamba module E_M (6) results in a decrease in both IoU and F1, and the predicted boundaries become less distinct. This shows that E_M plays a critical role in aggregating hierarchical features and is particularly helpful in capturing complex object shapes.

Among all the configurations, the complete model (7) achieves the best overall performance. It obtains an F1 of 89.73, an IoU of 82.07, a precision of 91.01, and a recall of 89.18. Its visual results are also the most aligned with the ground truth annotations. These observations confirm that the synergy between the proposed modules leads to significant improvements in both segmentation accuracy and visual quality.

We further validate the effect of each model component by conducting ablation experiments on the ISIC2016 dataset. Table 6 reports the numerical performance under different ablation settings, while Figure 14 illustrates the corresponding segmentation outputs for visual comparison. w/o FIIR (1) leads to a noticeable decline in IoU and F1, which drops to 84.33 and 90.81, respectively. Despite the recall and precision being reasonably balanced, the visual outputs exhibit weaker boundary fidelity, particularly in areas with low contrast, where the predicted masks tend to deviate from the lesion margins. Interestingly, we adopt w/o Φai concat (3) to produce the highest recall at 98.48, suggesting that the model becomes more permissive in capturing lesion pixels. However, this also comes at the cost of increased false positives, as reflected in the relatively lower precision and the presence of redundant red areas in the predicted masks. w/o Φai summary (2) causes the prediction accuracy to decrease, reinforcing that the absence of the modulation structure compromises the foreground-background balancing mechanism.

To verify the effectiveness of the bi-directional fusion module Φbi, we use Φbi summary (4) and Φbi concat (4) instead of Φbi. Specifically, w/o Φbi concat (5) has a clearer negative effect, with IoU decreasing to 81.19, accompanied by more pronounced boundary irregularities in the visualization. w/o E_M on IoU and F1 metrics scores lower than the full model. This suggests that although the primary structure still functions, the lack of high-low feature interaction leads to reduced segmentation confidence near ambiguous regions. With all components intact, the full model (7) achieves the strongest performance across all metrics F1 reaches 94.14, IoU improves to 89.40, and both precision and recall are maximized. The output masks are tightly aligned with the lesion contours, even under challenging conditions such as blurry or low-contrast boundaries, confirming the complementary nature of all proposed modules.