Early research into automated polyp segmentation primarily relied on traditional, hand-crafted feature-based approaches Pogorelov et al. (26). These methods typically targeted low-level visual cues such as color, texture, and shape to distinguish polyps from the surrounding colonic mucosa. Common techniques included color-space analysis, local binary patterns (LBP) for texture description, and edge detection algorithms to identify polyp boundaries Mamonov et al. (27) Maghsoudi (28) Rahim et al. (29). While these methods laid important groundwork, they were often sensitive to variations in illumination, viewpoint, and polyp morphology. Their reliance on manually engineered features limited their generalization capabilities, making it difficult to achieve robust performance across the wide spectrum of polyp appearances seen in clinical practice.

The advent of deep learning, particularly Convolutional Neural Networks (CNNs) Akbari et al. (13) Brandao et al. (30) He et al. (31)Simonyan and Zisserman (32) Cai et al. (33) Tomar et al. (34) Zhang et al. (35) Sengar et al. (36) Singh and Sengar (37) Khan et al. (38), marked a paradigm shift in medical image segmentation. The U-Net architecture, with its seminal encoder-decoder structure and skip connections, became a foundational model, demonstrating remarkable success in preserving both high-level semantic context and fine-grained spatial details Ronneberger et al. (39) Zhou et al. (40). This spurred the development of numerous variants, such as UNet++ Zhou et al. (40) and ResUNet Zhang et al. (41), which introduced innovations like nested skip pathways and residual connections to further improve feature representation.

More recently, models have been designed specifically to address the unique challenges of polyp segmentation. PraNet Fan et al. (42), for instance, introduced a parallel reverse attention network to explicitly model boundaries and regions, achieving a significant performance leap. Concurrently, the success of Vision Transformers Dosovitskiy et al. (43) in capturing global dependencies led to the development of hybrid models like TransUNet Chen et al. (44) Khan et al. (45), which combines the strengths of both CNNs and Transformers Vaswani et al. (46). The latest research trend involves leveraging large-scale, pre-trained foundation models, such as the Segment Anything Model (SAM) Kirillov et al. (47), as powerful backbones. These models provide robust, generalized feature extraction capabilities that can be fine-tuned for specialized medicaresl tasks like polyp segmentation, representing the current state-of-the-art and the context in which our work is positioned Li et al. (48, 49).

The overall architecture of our proposed SpectraNet is depicited in Figure 1. The network takes a colonoscopy image I ∈ ℝ^3×H×W as input and produces a pixel-level polyp segmentation probability map G ∈ ℝ^H×W. SpectraNet is composed of three primary components: (1) a frozen SAM2 Ravi et al. (50) backbone with a lightweight trainable adapter that extracts robust, multi-scale visual features; (2) a hybrid-domain enhancement unit within the skip connections that first sharpens indistinct boundaries using the Spectral-Guided Boundary Enhancement (SGBE) Module and then adaptively refines features with the Function-Specialized Mixture-of-Experts (FS-MoE) Module; and (3) a progressive refinement decoder that hierarchically fuses the enhanced features from different scales to reconstruct a high-resolution, boundary-precise segmentation mask, supported by multi-level deep supervision.

Automated polyp segmentation presents a significant challenge, as the visual characteristics of polyps—particularly flat or early-stage lesions—often lack strong, defining features, blending subtly with the surrounding healthy mucosa. To capture these nuanced patterns, a powerful and robust feature extractor is required. To this end, we employ the Hiera Ryali et al. (51) encoder from SAM2 as our foundational feature extractor. The encoder processes an input image I∈ℝ3×H×W and generates a four-level feature pyramid FL∈ℝB×HL×WL×CL for L={1, 2, 3, 4}. These feature maps have progressively decreasing spatial resolutions (H_L = H/2^{L+ 1}, W_L = W/2^{L+ 1}) and increasing channel dimensions (C_L = {144, 288, 576, 1152}), capturing a rich hierarchy of representations from fine-grained textures to abstract semantics.

To specialize the powerful, general-purpose features of SAM2 for the medical domain, we introduce a lightweight, trainable adapter into each Hiera block of the encoder, inspired by Houlsby et al. (52) Qiu et al. (53). The adapter, which consists of a linear layer for down-sampling, a GeLU activation function, another linear layer for up-sampling, and a final GeLU activation, processes the feature map F_L to generate a task-specific adaptation vector ΔF_L. This vector is then integrated back into the feature map via a residual connection before the main attention block, producing an adapted feature FL" as defined in Equation 1:

This in-place adaptive mechanism allows our model to inject polyp-specific priors directly into the feature extraction hierarchy, effectively steering the general encoder to become a specialized extractor finely tuned for discerning subtle pathological tissues. By keeping the original SAM2 encoder weights frozen and only training the lightweight adapters, we preserve the model’s strong generalized representations while minimizing additional training overhead and reducing the risk of overfitting on smaller medical datasets.

A primary difficulty in polyp segmentation is the ambiguous and low-contrast nature of polyp boundaries, which challenges convolutional networks that rely on local spatial gradients. However, the structural information that defines these boundaries, while subtle in the spatial domain, is more robustly encoded in the phase component of a signal’s frequency spectrum. Therefore, we designed the SGBE module to operate directly in the frequency domain to amplify this crucial structural information and enhance boundary representation, shown in Figure 1.

The module takes the adapted feature map FL′ from the encoder as input. First, a 1 × 1 convolutional layer projects the features into a uniform channel dimension (d = 64), resulting in the feature map X_L. We then transition to the frequency domain by applying a 2D Fast Fourier Transform (FFT). The FFT decomposes the feature map into its amplitude spectrum AL∈ℝB×d×HL×WL and phase spectrum PL∈ℝB×d×HL×WL. The amplitude spectrum primarily encodes the energy of spatial frequencies, which corresponds to low-level image statistics like contrast and brightness. The phase spectrum, conversely, is critically important as it preserves the high-frequency structural information that defines the precise spatial location of object boundaries and edges. Both spectra are then passed through parallel enhancement branches, each composed of a sequence of convolutional, BatchNorm, and ReLU layers, to learn enhancement residuals, ΔAL and ΔPL. These are added element-wise to the original spectra to yield the enhanced versions, A^L and P^L, as formulated in Equations 2, 3, respectively:

The enhanced amplitude and phase spectra are recombined to form an enhanced complex frequency tensor. Subsequently, an Inverse FFT (IFFT) is applied to transform this representation back into the spatial domain, yielding the boundary-enhanced feature map XLenh. To maintain training stability and preserve the original feature context, the final output of the module, FLSGBE, is formed by integrating the enhancement with the input features via a learnable, weighted residual connection as defined in Equation 4:

where α is a learnable scalar parameter that adaptively controls the contribution of the frequency-domain enhancement. This allows the network to dynamically balance spatial feature fidelity with spectral boundary refinement.

The significant morphological diversity of colorectal polyps, which vary widely in size, shape, and surface texture, demands a highly adaptive feature refinement strategy. To address this challenge, we introduce the Function-Specialized Mixture-of-Experts (FS-MoE) Module. Inspired by the success of MoE designs Zhou et al. (54), our approach utilizes a dynamic routing mechanism to create a content-aware feature processing pipeline. Unlike conventional MoE systems that use homogeneous experts, our FS-MoE employs a compact set of heterogeneous experts, where each is meticulously designed for a distinct and complementary function. The module processes the input feature map F_L^SGBE through three parallel, function-specialized expert branches – Sobel Edge Expert Branch, Multi-Scale Texture Expert Branch and Dilated Context Expert Branch, which is shown in Figure 2.

The Sobel Edge Expert is designed to explicitly capture and enhance high-frequency boundary information. It consists of a lightweight, depthwise 3×3 convolution where the kernels, Ksobel∈ℝd×1×3×3, are initialized with classic Sobel operators Kittler (55). This provides a strong inductive bias for edge detection, which is subsequently fine-tuned during training. The operation is formally defined in Equation 5:

where ∗_dw denotes the depthwise convolution operation, BN is BatchNorm, and σ is the GeLU activation function.

The Multi-Scale Texture Expert aims to capture varied surface patterns at multiple scales. It employs a three-branch parallel design where the input feature F is first projected into a low-dimensional space, Fmid=Conv1×1d→d/8(F). The three branches then operate on Fmid to produce outputs B₁, B₂, B₃ with varying receptive fields, as formulated in Equation 6:

As shown in Equation 7, these outputs are concatenated and fused via a final 1 × 1 convolution to restore the original channel dimension d, producing a rich, multi-scale texture representation:

To efficiently aggregate broad contextual information, the Dilated Context Expert utilizes a depthwise separable convolution. This two-stage process involves a 3 × 3 depthwise convolution with a large dilation rate (dil = 2), followed by a 1 × 1 pointwise convolution to combine channel features, as expressed in Equation 8:

The core of the module is its adaptive gating mechanism. A lightweight gating network, G(·), processes the input feature F_L^SGBE to dynamically generate a set of scalar weights w∈ℝ3. As formulated in Equation 9, the refined feature map, XLref, is obtained by a weighted sum of the expert outputs, Ej(·):

Finally, a residual connection with a learnable scalar parameter, β, is applied to form the final output, FLFS−MoE according to Equation 10:

This adaptive fusion empowers our network to intelligently tailor its feature refinement strategy, improving its ability to accurately segment polyps across their wide spectrum of visual presentations.

Although our enhanced skip connections provide rich, multi-scale feature representations, effectively fusing them to reconstruct a precise segmentation mask is non-trivial. Directly merging features from different scales can lead to coarse boundaries or the dilution of semantic information. To address this, we employ a Progressive Refinement Decoder (PRD), inspired by top-down fusion approaches proven effective in dense prediction tasks. The decoder is designed to hierarchically aggregate the enhanced multi-scale features, progressively recovering fine spatial details while preserving high-level semantic context.

The decoder reconstructs the final prediction through a sequence of refinement stages, beginning with the deepest feature map from the FS-MoE module, D4=F4FS−MoE. For each subsequent decoding stage i ∈ {3, 2, 1}, a Progressive Refinement Module (PRM) integrates the upsampled features from the deeper stage, D_{i+ 1}, with the corresponding enhanced skip connection feature, FiFS−MoE. This integration process is formally defined in Equation 11:

where Upsample(·) denotes a 2× bilinear upsampling operation and Φ_i(·) is a refinement block composed of a 3 × 3 convolution, BatchNorm, and a GeLU activation. This fusion process yields a series of intermediate decoder features {D₃, D₂, D₁} with progressively increasing spatial resolution.

To guide the learning process at all scales, a prediction head, Hi(·), is applied to the output of each decoder stage to produce an auxiliary segmentation map, out_i. Each head consists of a 1 × 1 convolution followed by a sigmoid activation function. These auxiliary maps (out₄, out₃, out₂) are upsampled to the original input resolution and used for deep supervision, as detailed in the following section. The final, primary segmentation map, out₁, is produced from the feature map of the last and highest-resolution decoder stage, D₁.

This progressive, top-down refinement structure ensures that strong semantic guidance from the deeper layers is effectively propagated and fused with the detailed, boundary-rich features at shallower layers. This process enables the network to achieve both a globally consistent understanding of the polyp and pixel-level precision in the final segmentation mask.

To ensure robust feature learning across all scales, the network is trained using a deep supervision strategy where a loss is applied to the output of each of the four decoder stages. This total loss function, L_total, is formulated in Equation 12 as a weighted sum of the loss from the primary output (out₁) and the three auxiliary outputs (out₂, out₃, out₄):

where G is the ground-truth polyp mask and λ_i are loss weights that balance the gradients from different scales. Based on empirical evaluation, we set the weights for the auxiliary outputs to λ_i = 0.1 for i ∈ {2, 3, 4}.

Following Dong et al. (17) Fan et al. (42), our core segmentation loss, L_seg, is a hybrid loss composed of the sum of Binary Cross-Entropy Loss (L_BCE) and Dice Loss (L_Dice), as shown in Equation 13:

The L_BCE term enforces pixel-level correctness, while the L_Dice term improves performance on imbalanced classes by maximizing the spatial overlap between the predicted mask and the ground truth.

This deep supervision scheme ensures that the intermediate layers of the decoder are explicitly guided toward producing semantically correct feature maps. By enforcing consistency across multiple scales, the model learns to effectively align high-resolution details from shallower layers with the robust semantic context from deeper layers, leading to more accurate and coherent polyp segmentation.

To rigorously evaluate our proposed model, we curated a high-quality segmentation dataset named the PolypSegDataset. This dataset comprises 1,302 images containing a total of 1,342 meticulously annotated polyp instances. A key characteristic of this dataset is its strong focus on large, clinically significant polyps, which constitute the vast majority of the samples. Furthermore, the dataset is highly standardized, with all images sharing a uniform resolution of 560 × 480 pixels, which facilitates direct and fair model comparison. The high fidelity of the ground-truth masks makes this dataset particularly well-suited for evaluating boundary-level segmentation accuracy.

To provide a comprehensive overview of the dataset’s characteristics, we conducted a detailed statistical analysis, with the key statistics summarized in Table 1. Regarding the Polyp Area Distribution, the data is overwhelmingly dominated by large polyps (≥ 5000 pixels), which account for a striking 87.1% of all instances. In contrast, small polyps (< 1000 pixels) are nearly absent at only 0.2%. This distinct composition makes the PolypSegDataset an ideal benchmark for developing and testing models on well-developed, clinically significant lesions rather than incipient polyps.

Annotation Quality. Figure 3 provides insight into the quality and precision of the ground-truth masks. The distribution of the number of vertices per annotation has a high average of 38.79. This metric underscores the meticulous detail with which the polyp boundaries were traced, moving far beyond simple bounding boxes or coarse outlines. Such high-precision annotation is critical for the rigorous evaluation of segmentation algorithms, especially for assessing boundary accuracy.

In addition to our curated PolypSegDataset, we also evaluate our model’s performance on two widely-used public benchmarks to ensure a comprehensive and fair comparison. We use CVC-ClinicDB Bernal et al. (56), a standard dataset containing 489 images with a variety of polyp types, and Kvasir-SEG Jha et al. (57), a larger and more diverse dataset consisting of 800 annotated images. Unlike our dataset, these public benchmarks feature a broader range of polyp sizes, shapes, and imaging conditions, allowing us to assess the generalizability and robustness of our proposed method.

We implement our proposed model using the PyTorch framework and conduct all experiments on a single NVIDIA 4090 GPU. The encoder backbone is initialized with the weights of SAM2-Hiera-Large pre-trained on the SA-1B dataset. For all three datasets—PolypSegDataset, CVC-ClinicDB, and Kvasir-SEG—we perform a stratified split into training (80%), validation (10%), and testing (10%) sets, using a fixed random seed of 42 to ensure the reproducibility of our results. The specific splits for the training, validation, and testing sets are 1041/130/131 for PolypSegDataset, 489/61/62 for CVC-ClinicDB, and 800/100/100 for Kvasir-SEG, respectively.

During the training phase, all input images are resized to a uniform resolution of 352 × 352 pixels. We train the model with a batch size of 12 for a total of 100 epochs. We employ the AdamW optimizer with an initial learning rate of 3 × 10⁻⁴ and a weight decay of 1 × 10⁻⁵. To ensure training stability, a learning rate warmup strategy is utilized for the first 20 epochs. Furthermore, to prevent overfitting and select the best-performing model checkpoint, we implement an early stopping mechanism that monitors the segmentation loss on the validation set.

To provide a comprehensive and multi-faceted evaluation of segmentation performance, we employ eight widely-used metrics: mean Dice Coefficient (mDice), mean Intersection over Union (mIoU), Mean Absolute Error (MAE), S-measure (S_α), weighted F-measure (Fβw), mean E-measure (mEξ), mean Sensitivity (meanSen), and mean Specificity (meanSpe).

Among these, mDice and mIoU are region-based similarity metrics that evaluate the overlap between the predicted mask and the ground truth, primarily assessing the overall accuracy of the segmented object. MAE offers a direct pixel-by-pixel comparison by calculating the average absolute difference between the continuous prediction map and the binary ground-truth mask. To evaluate the model’s classification performance on foreground and background pixels respectively, we report meanSen and meanSpe. Sensitivity measures the model’s ability to correctly identify polyp pixels (true positives), while Specificity measures its ability to correctly identify background pixels (true negatives). We also include three advanced metrics that capture more complex, human perception-aligned aspects of segmentation quality. The weighted Fβw balances precision and recall using non-uniform weights to better match visual assessment. The S_α evaluates the structural similarity between the prediction and the ground truth at both the object and region levels. Finally, the mEξ simultaneously captures both image-level statistics and local pixel-level matching.

To benchmark the performance of our proposed model, we conduct a comprehensive comparison against eight state-of-the-art and representative models from the field of semantic and medical image segmentation. The selected methods include classic architectures such as U-Net Ronneberger et al. (39), UNet++ Zhou et al. (40), ResUNet Zhang et al. (41), and DeepLabV3 Chen et al. (58), as well as more recent Transformer-based models like TransUNet Chen et al. (44). We also compare against several models designed specifically for polyp segmentation, including PraNet Fan et al. (42), Polyp-PVT Dong et al. (17), and CTNet Xiao et al. (59). For a fair and direct comparison, we utilize the publicly available, open-source implementations for all baseline methods.