In this work, we validate the efficacy of our method using two datasets: a proprietary dataset and the publicly available PRIME-FP20 dataset (Ding et al., 2020).

• The private dataset used in this study was collected at Ningbo Yinzhou People’s Hospital. The fundus UWF image dataset was captured using an Ultra-WideField Laser Scanning Ophthalmoscope of Optos 200Tx with a resolution of 3900 × 3072 . The paired FFA images were obtained by a Heidelberg Spectralis HRA Instrument of the same unit, with a resolution of 768 × 868 . Based on this data, we constructed a UWF-FFA dataset (UWF-GAN) and a dataset for segmentation (UWF-SEG). The UWF-GAN dataset consists of 120 aligned UWF image patches and FFA image patches, which are intercepted and aligned from the paired whole UWF fundus images and FFA images. The UWF-SEG dataset comprises 65 UWF fundus images, each annotated with pixel-level vessel labels for evaluating segmentation performance. The vessel labels of the UWF fundus images were annotated by an experienced ophthalmologist using the Pair annotation tool. All images were collected with explicit patient consent, and we rigorously adhered to data security regulations by meticulously anonymizing all personal information to ensure patient privacy adequately. Figure 1 illustrates examples of the original UWF fundus images, the preprocessed UWF images, and the corresponding paired FFA image. We utilize histogram equalization to preprocess UWF fundus images, effectively highlighting the vascular structures. Both our method and the comparison methods use the preprocessed images as input.

The proposed framework is illustrated in Figure 2, comprising two main components: a segmentation network for UWF fundus images and a style transfer network that generates FFA images from UWF fundus images. The segmentation network includes an encoder with a Local Perception Fusion Module (LPFM), a Selective Fusion Module (SFM), a decoder equipped with a Multi-level Attention Upsampling Module (AUM), and an auxiliary tiny encoder-decoder structure. Meanwhile, the style transfer network utilizes a CycleGAN-based architecture (Zhu et al., 2017).

Our method consists of four stages. Stage 1: Use small patches to pre-train a transfer network to obtain FFA features. Stage 2: Use middle patches to pre-train an encoder, aiming to reduce computational overhead during segmentation network training. Stage 3: Similarly, large patches are used to pre-train a large-scale encoder. Stage 4: Freeze the network parameters from the first three stages and begin training the entire segmentation network. In the fourth stage, UWF fundus images X from the dataset are simultaneously fed into both the segmentation network and the style transfer network initially. In the segmentation network, the LPFM processes the UWF images, extracting a feature map Z 1 . This feature map is passed through the decoder with the multi-level AUM and subsequently processed by a compact U-shaped network to produce the final blood vessel segmentation results. Concurrently, in the CycleGAN-based style transfer network, the UWF images undergo encoding and decoding, resulting in another feature map Z 2 . During the training phase, Z 1 and Z 2 are subjected to a search and matching process via the SFM, which aligns the feature maps from both networks. This alignment establishes a correspondence between the UWF and FFA images. Supervision for both networks is provided through a loss function based on paired UWF and FFA image biplets. The SFM filters the most relevant Z 1 , which is subsequently fed into the downstream segmentation framework. The following sections provide detailed descriptions of the vessel segmentation network, the Selective Fusion Module, and the Multi-level Attention Upsampling Module.

The large size of UWF fundus images presents a challenge in balancing segmentation accuracy and computational efficiency for blood vessel segmentation networks. Traditional methods, such as down-sampling, patch cropping, and cascade modeling, struggle to achieve this balance effectively. In this paper, we propose an innovative cropping strategy to address this issue. We utilize an efficient approach by randomly extracting smaller patches from the original UWF image (size 3900 × 3072 ), resulting in patches of size 256 × 256 , referred to as ‘small patches’. From these small patches, we generate medium and large patches with dimensions of 512 × 512 and 768 × 768 , respectively. Centered on the small patch, we expand it by a factor of two to obtain a middle patch of 512 × 512 , and similarly, expand it by a factor of three to obtain a large patch of 768 × 768 . All three patch sizes are then resized to a uniform size of 256 × 256 and simultaneously input into three ResNet (He et al., 2016) feature extractors. The feature extractors for the medium and large patches are pre-trained, and their parameters are fixed during the segmentation process. This results in three distinct sets of feature maps corresponding to different scales.

After that, we apply the Local Perception Fusion Module (LPFM) to enhance the contextual positional features of the small patches across different scales. The process of the LPFM is illustrated in Figure 3. First, we compute the inner products between the small patch and both the medium and large patches, obtaining attention maps through softmax activation. These attention maps are then used to generate new feature maps by performing inner product operations with the small patches. The resulting feature maps are regularized, and their weights are reassigned before being fused to form the final feature vector. Finally, these fused results are passed through the Selective Fusion Module (SFM), followed by the decoder and a small U-Net, to generate the final vessel binary mask.

In the subsequent sections, we introduce the specifics of the selective fusion module and the multi-level attention upsampling module, which constitute components of the proposed segmentation network.

The details of the module are displayed in Figure 2. As a crucial component for linking the two feature maps generated by the segmentation network and style transfer network, the SFM is based on knowledge related to feature engineering. Specifically, we propose to select FFA features to replace the broken blood vessel features ensuring that redundant information does not negatively impact segmentation performance. Initially, the network calculates the similarity between the feature map Z 1 output from the encoder of the segmentation network and the feature map Z 2 output from the encoder of the style transfer network. The similarity is computed using the Euclidean distance, as shown in Equation 1 below: d = ∑ i = 1 n x i − y i 2 (1)

Where x i and y i denote the pixel values of the two feature maps, respectively, and n denotes the number of pixels of the feature maps. It is worth noticing that the images in this study are rich in information, and based on the limitation of computational resources and time cost, this study carries out partial sampling computation on the feature maps for local similarity comparison. We set a threshold and range of 0.5 to confirm the strength of similarity. Eventually, SFM will fuse the feature map Z 2 with high similarity to the feature map Z 1 to generate the fusion feature M . If the similarity between the two feature maps is low, it indicates that Z 2 does not provide more valuable information for vessel segmentation, and only Z 1 is passed to the subsequent network layers.

Following the selection and fusion operations, multi-modal information is introduced in this study, which provides higher recognisability for retinal blood vessel segmentation. Additionally, similarity calculation and screening also avoid the interference of irrelevant features on the network when generating FFA images.

As shown in Figure 4, this module is designed to address the issue of vascular information loss during upsampling. The multi-level AUM primarily comprises two compact double convolutional blocks, denoted as E 1 and E 2 , which utilize an attention mechanism to capture feature representations at different levels. First, the feature map M generated by the SFM is input into E 1 , the detailed architecture of which is depicted in Figure 4. E 1 comprises two convolutional paths of different scales: path a serves as the primary path and encompasses two 7 × 7 convolutional layers, while path b consists of a single 7 × 7 convolutional layer. After the feature map M passes through path a and path b , two feature maps Z a and Z b are obtained. We calculate the attention weight W 1 of Z a with respect to Z b by employing Z a as the query ( Q ) , key ( K ) and Z b as value ( V ) . Subsequently, we perform element-wise multiplication between Z b and the weight W 1 , and fuse the product with Z a to acquire the upsampling result, integrating feature information across diverse scales. This result is then forwarded to E 2 , the structure of which is illustrated in Figure 4. E 2 closely resembles the architecture of E 1 , with the key difference being the scale of convolution operations. Similar to E 1 , E 2 comprises two convolutional paths c and d , with path c serving as the primary path. After passing through both paths, we obtain two distinct feature maps, denoted as Z c and Z d . Subsequently, these feature maps undergo attention computation and fusion steps to yield the decoded result. The process is represented by Equations 2–5: A 1 = Att C 2 C 1 M , C 2 C 1 M , C 3 M (2) O u t p u t * = C 2 C 1 M + A 1 (3) A 2 = Att C 5 O u t p u t * , C 5 O u t p u t * , C 4 C 1 M (4) O u t p u t = C 5 O u t p u t * + A 2 (5)

Where A 1 , O u t p u t * , and A 2 are the intermediate values computed by the module for the feature map. A 1 and A 2 represent attention weight, while O u t p u t * represents the output of E 1 . O u t p u t refers to the final output of the module, and Attn represents the attention mechanism. C 1 , C 2 , C 3 , C 4 , C 5 correspond five different convolutional layers.

By employing multi-scale convolutions and integrating an attention mechanism, this study enhances retinal blood vessel segmentation with improved structural comprehensiveness. The module excels at fusing feature information across different scales, thereby reducing information loss and omissions during the upsampling process. As a result, it more effectively addresses the challenge of incomplete segmentation of fine retinal blood vessels in UWF fundus images.

In the experiment, UWF fundus images are preprocessed using the contrast adaptive histogram equalization method to enhance the contrast of the images. The UWF-SEG dataset contains a total of 65 images, with 20 images allocated to the test set and the remaining 45 images utilized for training purposes. The PRIME-FP20 dataset consists of 15 images, with 10 images used for training and the remaining 5 images for testing. To ensure consistency, all images are resized to dimensions of 1024 × 1024 pixels, with small patches resized to 256 × 256 pixels. Data augmentation techniques, including random flipping and rotation, are applied during the experiments. The training process spans 200 epochs, enabling the model to converge and effectively capture underlying patterns within the data.

In the experimentation of the proposed method, the model was implemented using PyTorch 3.8 and trained on a workstation equipped with four NVIDIA GeForce RTX 3090 GPUs. A batch size of 5 was used for the UWF-SEG dataset, and 2 for the PRIME-FP20 dataset throughout the training process. The initial learning rate for the vascular segmentation network was set to 0.0004, with a weight decay rate of 0.0005. The Adam optimizer was employed for gradient updates.

To evaluate the performance of the proposed retinal vessel segmentation method, four widely adopted metrics are used in this work, namely,: the Dice Similarity Coefficient (Dice), Sensitivity (Sen), and Balanced Accuracy (BACC). In the evaluation of the method, the sigmoid function was used as the activation function for the final output, and the confusion matrix was calculated (comprising True Positives (TP), False Positives (FP), True Negatives (TN), and False Negatives (FN)) to evaluate the accuracy of the method. The relevant formulas and their significance are as follows:

• Dice Similarity Coefficient (Dice) = 2 × TP/(2 × TP + FP + FN).

Where True Positives (TP) represent the number of samples correctly classified as positive, False Positives (FP) refer to samples incorrectly classified as positive, True Negatives (TN) are the samples correctly classified as negative, and False Negatives (FN) are the samples incorrectly predicted as negative when they should have been positive.

A comprehensive series of comparative experiments were conducted in this section to demonstrate the effectiveness of the proposed method. These experiments included both quantitative and qualitative analyses, evaluating the performance on both public and private test sets. Several state-of-the-art approaches in fundus vessel segmentation were selected for comparison in this study, including CE-Net (Gu et al., 2019), CS-Net (Mou et al., 2021), TransUnet (Chen et al., 2021), SwinUnet (Cao et al., 2022), U-Net (Ronneberger et al., 2015), ResUnet (Xiao et al., 2018) and ConvUNext (Han et al., 2022). All these methods have demonstrated superior performance in fundus image analysis. To ensure the fairness of the experiments, identical image preprocessing and cropping techniques were applied across all methods, and the datasets were divided into the same training test sets.

The metrics analysis shows that the proposed method outperforms other methods in terms of the effectiveness of retinal blood vessel segmentation in UWF fundus images. Specifically, the M3B-Net method presented in this chapter surpasses other methods across all evaluated metrics (Dice, Sen, and BACC) as demonstrated in Table 1. For example, on the private UWF-SEG dataset, our method achieves improvements of approximately 5.19%, 3.35%, and 1.60% in Dice, Sensitivity, and BACC, respectively, compared to the ConvUNext model (Han et al., 2022). Additionally, our method provides more detailed vascular information, facilitating a finer observation of the retinal vessels. This improvement is primarily attributed to the proposed framework’s focus on enhancing the vascular signal while minimizing the influence of background noise during the reconstruction process.

This significant performance improvement can be attributed to the proposed method’s enhanced capability in segmenting fine retinal blood vessels in UWF fundus images. Specifically, the proposed method utilizes cross-modal image style transfer to achieve effective feature enhancement. During the learning process, the selective fusion module enables the segmentation network to select two feature maps based on correlation calculations, which significantly improves the recognition of fine retinal vessels. This approach mitigates the risk of information loss that can occur when directly applying enhancement techniques. Furthermore, given the wide imaging range of UWF fundus images, the study incorporates a local perceptual fusion module and a multi-level attention upsampling module. These modules ensure that global information is effectively integrated during the encoding process while minimizing the loss of vascular details during the decoding stage.

As shown in Figure 5, the original image from the private UWF-SEG dataset, along with the segmentation results from our method and other comparative approaches, are presented. From the figure, it is evident that TransUnet (Chen et al., 2021), a network structure based on the attention mechanism, is better at capturing both local and global feature relationships than traditional convolutional neural networks. However, the segmentation results exhibit obvious under-segmentation, which indicates that it is more seriously affected by the artifacts and lesion areas in the UWF images during the segmentation process. Additionally, other comparative methods show obvious vessel breaks and unrecognized small vessel ends in the segmentation results. In contrast, our method can more effectively identify the small vessels in UWF fundus images, reducing under-segmentation while also mitigating the over-segmentation issues, as highlighted in the orange boxes in Figure 5.

Similar to the experimental evaluation metrics for the private dataset, this study also conducts both quantitative and qualitative analyses of all methods on the public PRIME-FP20 dataset. As shown in Table 1, a comparison of the experimental results from our method and other segmentation approaches on the public dataset demonstrates the superior performance of our method. Specifically, on the PRIME-FP20 dataset, our method shows a 3.14% improvement in the Dice score compared to CE-Net (Gu et al., 2019) and outperforms the ConvUNext (Han et al., 2022) by 2.52% in SEN and 1.16% in BACC. These results highlight the effectiveness of our method. Furthermore, our method performs better on the public dataset than on the private dataset, primarily due to the segmentation challenges introduced by lesion regions in the private dataset.

Additionally, we present both the over-segmentation and under-segmentation results of our proposed method and other comparative methods in Figure 6. It is evident that our method is more effective in recognizing fine vessels and vessels in low-contrast regions. As shown in the orange boxes of Figure 6, although all methods tend to under-segment fine blood vessels near vessel terminals, our method demonstrates superior vessel recognition compared to the others. Furthermore, methods such as SwinUnet (Cao et al., 2022), TransUnet Chen et al. (2021), and CE-Net (Gu et al., 2019) exhibit more pronounced over-segmentation, primarily due to interference from artifacts and lesion areas. In contrast, our method mitigates the impact of these artifacts and noise, thereby reducing the over-segmentation problem.

To evaluate the performance of the proposed method for retinal vessel segmentation and to further assess the effectiveness of each individual component, this section establishes various baselines for comparison. Extensive experiments are conducted on the publicly available dataset PRIME-FP20, utilizing identical image preprocessing and cropping methods across all baseline methods. As shown in Table 2, LPFM, SFM, and AUM components represent the local perception fusion module, the selective fusion module, and the multi-level attention upsampling module, respectively. The symbol √ denotes the inclusion of a component, while × indicates its exclusion. A comparative analysis of rows 3 to 6 demonstrates that the inclusion of each component significantly improves the results, highlighting their effectiveness. Specifically, incorporating the LPFM component alone improves our method’s BACC, Sen, and Dice metrics by 1.11%, 2.21%, and 0.63%, respectively. Adding the SFM component alone outperforms the baseline method by 2.45%, 4.98%, and 2.09%, respectively. Including the AUM component alone surpasses the baseline method by 1.04%, 2.09%, and 0.96%, respectively. The SFM significantly enhances the segmentation performance of the method by introducing FFA features from the style transfer network. The LPFM enhances the network’s multi-scale perception capability, while the AUM notably improves the results, particularly the sensitivity, indicating that it reduces feature loss during the upsampling process. Additionally, as depicted in rows 7 to 9 of Table 2, the combination of any two out of the three components also enhances performance. Specifically, adding the SFM yields a greater improvement than combining LPFM and AUM, suggesting that FFA features play a crucial role in enhancing retinal blood vessel segmentation in UWF fundus images. Finally, our method achieves better performance by integrating all these components.

To further demonstrate the effectiveness of our method and its components, LPFM, SFM, and AUM. We also conducted the GradCAM (Selvaraju et al., 2017) visualization of the output from the final layer of the network decoder, as shown in Figure 7. As observed in Figures 7B, C, and Figure 7D, the orange boxes indicate stronger responses to small blood vessels compared to Figure 7A, particularly in Figure 7C. This demonstrates the effectiveness of the designed components and underscores the advantages of incorporating the FFA modality for low-contrast vessel segmentation. Ultimately, by combining all three components, our method achieves superior vessel segmentation performance.