Given the stability and significant progress made by these networks of FCN (43), U-net (UNet) (44), PSPNet (46), DeepLabV3+ (47), UPerNet (48), and SegFormer (49) in medical image segmentation, these networks were selected as the image segmentation networks to construct multi-object region segmentation models for dynamic knee joint X-ray radiography.

First, considering the advantages of UNet in fusing multi-scale features through skip connections to improve high-resolution details in segmentation and FCN in fusing feature maps at different levels, a network with UNet as the backbone, FCN_UNet, is constructed. Second, the R50c variant replaces a conventional 7 × 7 convolutional stem with three successive 3 × 3 convolutions, thereby better preserving fine spatial details, which are beneficial for dense prediction tasks (57). Therefore, three networks with R50c as the backbone—PSPNet_R50c, DeepLabV3+_R50c, and UPerNet_R50c—are constructed. Lastly, given the computational and memory requirements of MiT-B2, the SegFormer_B2 with the MiT-B2 backbone performs best (49). Therefore, the SegFormer_B2 is selected as a segmentation network in this study.

The weighted CE loss function with the internal weight has been recognized as a universal loss function in medical image segmentation (13–16, 52). To train the network for segmenting multi-object regions of dynamic knee joint X-ray radiography, the loss function is usually defined as Equation 1 with the same external weights (α₁ = β₁ = γ₁ = η₁).

To balance losses across different multi-object regions in dynamic knee joint X-ray radiography during network training, a dual-level weighted cross-entropy loss function (LCE2) with external and original internal weights based on multi-object region area is proposed for multi-object region segmentation. Specifically, the same external weights, α₁, β₁, γ₁, η₁, are adjusted to the different external weights, α₂, β₂, γ₂, η₂, based on the multi-object region area of the patella, femur, tibia, and patellar tendon, defined by Equation 2.

The introduction of the proposed dual-level weighted cross-entropy loss function, L_CE2, is as follows. First, the area of each object region (S_Patella, S_Femur, S_Tibia, and S_{Patellar tendon}) in the labeled knee joint X-ray images of the training set is calculated. Second, the total region area (S_total) is calculated by summing the areas of each object region S_Patella, S_Femur, S_Tibia, and the S_{Patellar tendon}. Third, the ratios of each region area (S_{Patella_ratio}, S_{Femur_ratio}, S_{Tibia_ratio}, and S_{Patellar tendon_ratio}) relative to the S_total are calculated. Fourth, let ω1*S_{Patella_ratio} = ω2*S_{Femur_ratiol} = ω3*STibia_ratio=ω4*S_{Patellar tendon_ratio} = ε, obtaining ω₁ = ε/S_{Patella_ratio}, ω₂ = ε/S_{Femur_ratiol}, ω₃ = ε/S_{Tibia_ratio}, and ω₄ = ε/S_{Patellar tendon_ratio}. Lastly, these weights, ω₁, ω₂, ω₃ and ω₄, are separately normalized, obtaining the object region weights α₂= 0.2704, β₂= 0.0324, γ₂= 0.0360, η₂= 0.6612. The above specific implementation details are represented by mathematical (Equations 3–6):

where, the pixel represents the number of pixels in each object region of the patella, femur, tibia, and patellar tendon, and the S_pixel represents the area of each pixel. In addition, x₁, x₂, x₃, and x₄ represent the multi-object region of the patella, femur, tibia, and patellar tendon.

Besides the CE loss function, this study also considers DICE and BD loss functions (L_DICE and L_BD) to construct a mixed loss function based on the proposed weighted cross-entropy loss function L_CE2. Therefore, three types of mixed loss functions, L_{Total_loss1}, L_{Total_loss2}, and L_{Total_loss3}, are defined based on L_CE2 by (Equations 7–9):

To further consider the role of each loss function of the three types of mixed loss functions in network training, individual loss weights, τ₁, τ₂, and τ₃, are set separately for each loss function. Therefore, Equations 7–9 have been rewritten as (Equations 10–12):

In network training, the learning rate is a key hyperparameter that determines the model's convergence. Dynamic learning rate, by adaptively adjusting the learning rate during training, can quickly converge in the early stages and be finely tuned in the later stages, thereby achieving better model performance. In this study, an improved AdamW optimizer (58) is used, with an initial learning rate of 1e-4. The learning rate decay of the AdamW optimizer includes an initial learning rate decay stage, a stable learning rate decay stage, and a fine-tuning learning rate decay stage.

Among them, the initial learning rate decay stage based on the initial learning rate uses warm-up [warmup_steps] (59, 60), which provides fast gradient descent during the initial stage of network training while maintaining stability. The stable learning rate decay stage is polynomial decay (61), maintaining a relatively stable learning rate during long-term network training. The fine-tuning learning rate decay stage uses cosine learning rate decay (62), applied at the end of training.

In the first aspect, the performance of the multi-object region segmentation models based on the five selected networks and the traditional and proposed dual-level weighted CE loss functions is evaluated, respectively. The traditional weighted CE loss function is the equal CE weights of the patella, femur, tibia, and patellar tendon. The traditional and proposed dual-level weighted CE loss functions, L_CE1 and L_CE2, are defined by (Equations 13, 14):

Where, L_{Patella_CE}, L_{Femur_CE}, L_{Tibia_CE}, and L_{Patellar tendon_CE} are the binary CE loss functions of the patella, femur, tibia, and patellar tendon, respectively. N denotes the number of samples used to calculate loss, and C denotes the number of segmentation categories (C = 0, 1). When C = 1, C presents the patella/femur/tibia/patellar tendon.

In the second aspect, the performance of the multi-object region segmentation models based on the five selected networks and the proposed mixed loss functions based on L_CE2 in Equations 10–12 is evaluated.

To assess the performance differences in the comparative experiment, five standard evaluation metrics, the Intersection over Union (IoU), Dice, Precision, Recall, median 95th Hausdorff distance (HD95), and Average Symmetric Surface Distance (ASSD), are adopted to calculate the performance of the multi-object region segmentation models on each knee joint X-ray image in the test set. Then, the mean IoU, mean Dice, mean Precision, mean Recall, mean HD95, and mean ASSD are calculated by the IoU, Dice, Precision, Recall, HD95, and ASSD of the test set.

Meanwhile, a single evaluation metric is insufficient for comprehensively evaluating the performance of a multi-object region segmentation model. Specifically, the higher the mean IoU, Dice, Precision, and Recall, the better. On the contrary, the smaller the values of HD95 and ASSD for these evaluation scales, the better. Therefore, based on the evaluation rules of the above six evaluation metrics, two comprehensive evaluation metrics, Mean_metric 1 and Mean_metric 2, are defined separately to evaluate the performance of the multi-object region segmentation models, defined by (Equations 15, 16):

Where, the mean_IoU, mean_Dice, mean_Precision, mean_Recall, mean_HD95, and mean_ASSD represent the mean value of the IoU, Dice, Precision, Recall, HD95, and ASSD of the test set, respectively. In addition, IoU_i, Dice_i, Precision_i, Recall_i, HD95_i, and ASSD_i represent the IoU, Dice, Precision, Recall, HD95, and ASSD of the i^th knee joint X-ray image in the test set. N represents the number of knee joint X-ray images in the test set.

To determine the optimal multi-object region segmentation model, a scoring criterion based on the two comprehensive evaluation metrics is proposed. First, rate the first comprehensive evaluation metric, Mean_metric 1, in descending order, and rate the second comprehensive evaluation metric, Mean_metric 2, in ascending order. Among them, the maximum score is set to the number of models that need to be scored, and the minimum score is set to 1. Second, sum the scores of the two comprehensive evaluation metrics for each segmentation model to obtain the final score. Finally, based on the final scores of all multi-object region segmentation models, sorted from high to low, the level of each model is determined, and the model with the highest level is selected as the optimal multi-object region segmentation model. The process of determining the optimal multi-object region segmentation model as described above is mathematically defined by Equations 17–20:

where, the Rank_{Ascending order} and Rank_{Descending order} separately represent that the first comprehensive evaluation metric, Mean_metric 1, is rated in descending order and the second comprehensive evaluation metric, Mean_metric 2, is rated in ascending order. The Mean_metric 1_i and Mean_metric 2_i separately represent the i^th model of a total of N models. These two score vectors, ScoreMean_metric i1⃗ and ScoreMean_metric 2⃗, separately represent the score of two comprehensive evaluation metrics, Mean_metric 1 and Mean_metric 2, of all models. The Score_i represents the score of the i^th model. The ranking score vector, ScoreRank⃗, scores in descending order. model_optimal represents the optimal multi-object region segmentation model.

Table 2 reports that Experiments 1–25 use different loss functions across five networks during training to generate 55 multi-object region segmentation models for dynamic knee joint X-ray radiography.

Specifically, to demonstrate the effectiveness of the proposed L_CE2, experiments 1, 2, 6, 7, 11, 12, 16, 17, 21, and 22 are conducted on L_CE1 and L_CE2 using FCN_UNet, PSPNet_R50c, DeepLabV3+_R50c, UPerNet_R50c, and SegFormer_B2, respectively. To demonstrate the effectiveness of the mixed loss function and determine the optimal multi-object region segmentation model with an appropriate ratio of each loss function in the mixed loss function, Experiments 3–5, 8–10, 13–15, 18–20, 23–25 are conducted on the mixed loss function constructed by the combination of the L_CE2, L_DICE, and L_BD based on the FCN_UNet, PSPNet_R50c, DeepLabV3+_R50c, UPerNet_R50c, and SegFormer_B2, respectively.

To ensure that the multi-object region segmentation networks are adequately trained, the maximum number of iterations is set to 50,000. In addition, to further prevent overfitting in multi-object region segmentation networks, three criteria, connected in series, are set to stop training.

First, one of the basic criteria for stopping network training is that the total loss value between the knee joint multi-object segmentation images and their GTs in the training set is less than 0.2. Second, the validation set and the knee joint multi-object segmentation model are used to evaluate the model's overall segmentation performance in the four object regions every 50 iterations. Based on the total loss value between the knee joint multi-object segmentation images and their GTs in the training set being less than 0.2, the total mean Intersection over Union (IoU) between the knee joint multi-object segmentation images and their GTs in the validation set being greater than 0.8 is set as the second basic criterion for stopping the network training. Third, due to the small area of the patellar tendon, it is difficult to achieve good performance. To provide more sufficient training for the minor patellar tendon, a mean IoU greater than 0.85 is set as the third criterion for stopping training based on the validation set of patellar tendon object segmentation images and their GTs. Lastly, after meeting the above three criteria, if the total mean IoU between the four object segmentation regions and their GTs in the validation remains for 10 consecutive times (i.e., the relative fluctuation of the total mean IoU between the four object segmentation regions and their GTs in the 10 validation sets is less than 0.001), the network training can be stopped. If the above three criteria are not achieved, stop training the network when the maximum number of iterations is reached. After the network's training is complete, the segmentation model that performs best on the validation set will be selected for testing on the test set.

Table 3 shows the specific set of development environment and requirements.

Table 4 reports the comparative evaluation metrics, including the mean, standard deviation, and maximum-to-minimum values of L_CE1 and L_CE2 on the patellar tendon of the test set. In addition, Figure 2 shows the visualized comparative evaluation metrics of the L_CE1 and L_CE2 on the patellar tendon of the test set.

Specifically, compared to FCN_UNet, PSPNet_R50c, DeepLabV3+_R50c, UPerNet_R50c, and SegFormer_B2 with the L_CE1, the Mean IoU of these networks with the L_CE2 has been comprehensively improved by 0.13, 1.27, 1.86, 2.13, and 5.64%, respectively. Compared to PSPNet_R50c, DeepLabV3+_R50c, UPerNet_R50c, and SegFormer_B2 with the L_CE1, the Mean Dice of these networks with the L_CE2 has been improved by 0.90, 1.42, 1.78, and 4.21%, respectively. Compared to PSPNet_R50c with LCE1, the Mean Precision of this network with LCE2 has improved by 0.41%. Compared to FCN_UNet, PSPNet_R50c, DeepLabV3+_R50c, UPerNet_R50c, and SegFormer_B2 with the L_CE1, the Mean Recall of these networks with the L_CE2 has been comprehensively improved by 5.16, 1.54, 5.24, 7.84, and 8.71%, respectively. Compared to FCN_UNet, PSPNet_R50c, and SegFormer_B2 with the L_CE1, the Mean HD95 of these networks with the L_CE2 has been improved by 2,211.15, 7.02, and 177.58%, respectively. Compared to FCN_UNet, PSPNet_R50c, UPerNet_R50c, and SegFormer_B2 with the L_CE1, the Mean ASSD of these networks with the L_CE2 has been improved by 234.17, 5.86, 3.29, and 20.29%, respectively. Compared with the first comprehensive evaluation metrics, mean_metric 1, for the FCN_UNet, PSPNet_R50c, DeepLabV3+_R50c, UPerNet_R50c, and SegFormer_B2 with the LCE1, the LCE2 has comprehensively improved these networks by 1.84%. Compared to the second comprehensive evaluation metrics, Mean_metric 2, of the FCN_UNet, PSPNet_R50c, DeepLabV3+_R50c, UPerNet_R50c, and SegFormer_B2 with the L_CE1, the performance of these networks with the L_CE2 has been comprehensively improved by 241.35%.

Tables 5–8 report the comparative evaluation metrics and scores for different networks with single or mixed loss functions on the test set for the patella, femur, tibia, and patellar tendon, respectively. Figure 3 shows a visual example of typical multi-object region segmentation using the five networks with different loss functions. Furthermore, Figure 4 shows the top 10 multi-object region segmentation models and the evaluation metrics for the best-performing model. Lastly, Figure 5 shows the visualized multi-object region segmentation images of dynamic knee joint X-ray radiography using the best multi-object region segmentation model. Specifically, the detailed evaluation metrics for different networks using the mixed loss function on the test set for the patella, femur, tibia, and patellar tendon are presented in Table A1 of the Appendix.