This study used a single-center retrospective analysis. Approval was obtained from the Research Ethics Committee (2023072602), and the requirement for informed consent was waived.

A total of 576 patients from the China-Japan Union Hospital of Jilin University from January 2020 to December 2022 were retrieved from the electronic medical record system. Inclusion criteria: (1) postoperative pathology confirmed as PTC (including papillary carcinoma, papillary adenocarcinoma, papillary carcinoma follicular subtype, micropapillary carcinoma); (2) First thyroid surgery with total thyroidectomy, central lymph node dissection and lateral cervical lymph node dissection; (3) Postoperative pathology confirmed the presence of LLNM; (4) No history of head and neck radiation exposure.

Exclusion criteria: (1) other types of thyroid cancer (including follicular thyroid cancer, medullary thyroid cancer, undifferentiated thyroid cancer, and high-risk subtypes of PTC such as diffuse sclerosis); (2) previous history of thyroid surgery; (3) PTC patients with tumors located in the isthmus or occupying most or even all of the glands (to better determine the effect of tumor location on SLLNM); (4) Postoperative pathology confirmed that there was no LLNM; (5) There were other malignant tumors; (6) History of head and neck radiotherapy.

In this study, all patients underwent total thyroidectomy, central lymph node dissection, and lateral cervical lymph node dissection. Central lymph node dissection ranged from superiorly by the hyoid bone, inferiorly to the sternal notch, and lateral to the carotid sheath, posteriorly to prevertebral fascia. Lateral lymph node dissection covers zones II, III, IV, and V. Zone I or VII dissection was performed only if evidence of metastasis was found on imaging, cytopathology, or intraoperatively. In our study, no patient underwent zone I and VII clearance. All enrolled patients had preoperative findings of suspected LLNM or were at high risk of LLNM based on clinician experience.

Three nomograms used to predict SLLNM were considered in our study. For all included patients, parameters related to Hu 2020 nomogram were collected, such as age (> 55, ≤ 55), bilaterality under pathology, and location (upper pole, non-upper pole); Wang 2020 nomogram parameters such as age, pathological tumor size (≤ 10mm, > 10mm), location (upper, middle and lower poles); Zhao 2023 nomogram parameters such as age (> 40, ≤ 40), tumor size under ultrasound (< 9.1 mm, ≥ 9.1 mm), location (upper pole, non-upper pole), margin and extrathyroidal extension (ETE), as shown in Table 1 . In addition to this, patient’s gender, BMI; ultrasound characteristics such as bilaterality, multifocality, calcification status, aspect ratio, and nodule blood supply; and pathologic characteristics such as bilaterality, multifocality, capsular invasion, ETE, Hashimoto’s thyroiditis (HT), nodular goiter (NG). In the above three models, the grouping criteria for some variables were inconsistent; for example, for multifocal, Hu 2020 specified ≥2 lesions in a unilateral lobe, and Wang 2020 and Zhao 2023 specified ≥2 PTC lesions in bilateral lobes; the grouping criteria for age, location, and size were also inconsistent. Here, capsular invasion refers to the tumor invading the thyroid capsule but not penetrating it. ETE is the tumor penetrating the capsule or invading the surrounding soft tissues and structures (26).

In addition, three variables, Capsular distance ( Figure 1 ), US-SLNM, and intraoperative CLND number, were collected into the model to see whether there was a gain. US-SLLNM, referring to the 2013 European Thyroid Association guidelines (27), which divided lymph nodes into normal, uncertain, and suspected malignant; uncertain and suspicious malignancy on ultrasound was defined as suspicious lymph nodes; no abnormality was found in the central lymph nodes under ultrasound, and the suspicious cases of lateral cervical lymph nodes were positive.

Hu 2020, Wang 2020 and Zhao 2023 nomograms are shown in the Supplementary Figures .

Statistical analysis was performed using R software 4.3.0 and SPSS 26.0 software. Categorical variables were described as frequencies and percentages; continuous variables were expressed as means and standard deviations. The predictive performance of original nomograms was assessed by area under the ROC curve (AUC), calibration curves, and decision curve analysis (DCA). AUC values quantified the differentiation of the external validation cohort across the three nomograms; calibration curves were used to assess the agreement between predicted and actual SLLNM risk; and DCA was used to visualize the net benefit of the three nomograms. Taking SLLNM or not as the dependent variable, the predicted probability of the original nomogram and Capsular distance, US-SLLNM, and CLND number as the independent variables, logistic regression was used to incorporate each of the three new variables into the original nomograms and to plot the ROC curves and the DCA in order to compare the predictive performance of the original model to the model with the addition of the new variables. Next, univariate and multivariate logistic regression analyses were performed based on larger sample sizes to screen for stable independent predictors and construct a dynamic nomogram, with values of P<0.05 considered statistically significantly different. Internal validation was performed using 1000 Bootstrap, ROC curves, and calibration curves were used to evaluate the predictive accuracy and consistency of the model, and DCA reflected the net benefit of the model to patients. The original models were all static nomograms, and the regression parameters of the models were not provided, so we used an approximation, which is also the only approach used clinically. We placed the information of each patient into three original models to obtain the corresponding total score and predicted probability. Flow chart of the entire experiment is shown in Figure 2 .

In the external cohort, the incidence of SLLNM was 12.0%, while the incidence of SLLNM was 9.7%, 11.6%, and 15.4% in Hu 2020 (Model 1), Wang 2020 (Model 2), and Zhao 2023 (Model 3), respectively. The baseline characteristics of the external validation cohort and the original cohort of the three models are shown in Table 2 .

None of the original models provided the optimal cutoff value; therefore, in the external validation queue, the cutoff value when maximizing the Yoden index was selected based on the ROC curves of each model ( Figure 3 ). From the ROC curves of the three models, it is easy to see that, although they are all reduced to a certain extent compared with the original models, they also all show a better degree of differentiation, see Tables 3 , 4 .

Model 1: The optimal cutoff value for this nomogram was 8.2% (AUC = 0.695, 95CI%: 0.630-0.761), and sensitivity and specificity associated with the optimal threshold were 71.0% and 67.9%, respectively. Applying this cutoff value, 29.0% (20/69) of SLLNM-positive patients would be below the threshold, would be incorrectly judged to be at low risk of developing SLLNM, and would be advised against the need for lateral cervical lymph node dissection.

Model 2: The optimal cutoff value of this nomogram was 26.4% (AUC = 0.792, 95CI%: 0.739-0.845), and sensitivity and specificity associated with the optimal threshold were 63.8% and 83.6%, respectively. Applying this cutoff value, 36.2% (25/69) of patients would have escaped positive screening.

Model 3: The optimal cutoff value for this nomogram was 31.7% (AUC = 0.769, 95CI%: 0.713-0.824), and sensitivity and specificity associated with the optimal threshold were 66.7%, respectively, 79.1%. Applying this cutoff value, 33.3% (23/69) of patients would miss positive screening. We believe that this non-negligible percentage of patients is too high and that this finding represents a major limitation of the original nomograms.

The calibration curves for Model 1, Model 2, and Model 3 are shown in Figure 4 . Overall, the three original models passed the Hosmer-Lemeshow goodness-of-fit test, which showed that the models all fit poorly (P < 0.05.) Model 1 was more consistent at predicted probabilities of 15.5% to 24.5% and underestimated the risk of SLLNM in the current cohort at predicted probabilities of 6.5% to 15.5% and >24.5% ( Figure 4A ). With Model 2, there is more agreement with actual risk at predicted probabilities <35%; however, when larger predicted probabilities (>35%) occur, there tends to be an overestimation of observed risk ( Figure 4B ). Model 3, on the other hand, has a degree of overestimation over the entire range of predicted probabilities ( Figure 4C ). The current incidence of SLLNM ranges from 3.4% to 22.5%, as described in the relevant literature (2–17). Thus, in the lower-risk patient cohort, Model 1 underestimates the actual risk to some extent, Model 2 is more consistent with the actual risk, and Model 3 overestimates the actual risk.

The DCA of Model 1 showed that the threshold probability was between 6.5% -20% and 22% -56%, with a certain clinical net benefit. The optimal Cutoff value of this nomogram is 8.2%, and the use of which will bring a net clinical benefit of 5.5% ( Figure 5A ). The DCA of Model 2 and Mode 3 showed a net clinical benefit when the threshold probabilities were 17.9% -36.3% and 4.1% -37%, respectively ( Figures 5B, C ). When the respective optimal Cutoff values are taken, there is also a certain clinical net benefit, indicating that the use of this nomogram has clinical significance.

Three new variables, Capsular distance, US-SLLNM, and CLND number, were included in this study. The ROC curve analysis of the CLND number showed that the AUC was 0.713 (0.643-0.783) ( Figure 6A ). In order to divide its optimal cut-point, this study selected the cutoff when the Yoden index is at its maximum, which is 5 ( Figure 6B ). As can be seen from Figure 3 , except for the Capsular distance in Model 2, which did not increase the AUC (Figure 3B), the remaining two variables increased the AUC to varying degrees in all three original models. Comparison of AUCs for the original cohort, the external cohort, and the cohort calibrated with the added variables is shown in Table 4 . The DCA in Figure 5 shows that the model with the addition of the three variables resulted in a significant increase in both the risk threshold range and the net benefit compared to the original model.

Based on the gain brought by the new variables to the model, a larger sample size (576 cases) will be used this time to explore the influences of previous studies again, in order to construct a model with higher performance. The sample sizes for the three nomograms constructed above were 521, 378, and 267, respectively. In the univariate logistic regression analysis, it was found that female (P=0.014); age (P<0.001); BMI ≥25Kg/m² (P=0.030); upper pole (P<0.001); maximal diameter of the tumor ≤10mm (P=0.009); Capsular distance < 0mm (P=0.006); US-SLLNM (P<0.001); and CLND number ≤ 5 (P<0.001) were significantly associated with SLLNM in PTC patients (P<0.05). On the contrary, the differences in margins, aspect ratio, nodule blood supply, calcification, bilaterality, multifocality, capsular invasion, ETE, HT, and NG were not statistically significant between the SLLNM (+) group and the SLLNM (-) group (all P>0.05). The results of univariate logistic regression analysis are shown in Table 5 .

The eight variables that were statistically different from those mentioned above were included in the multivariate logistic regression, and the results showed that female (OR 2.740, 95%CI 1.279-5.870, P=0.009), age (OR 1.061, 95%CI 1.031-1.091, P<0.001), tumor located in the upper pole (OR 3.427, 95%CI 1.749- 6.715, P < 0.001), maximum tumor diameter ≤ 10 mm (OR 2.544, 95%CI 1.360-4.758, P = 0.003), Capsular distance < 0 mm (OR 3.287, 95%CI 1.217-8.875, P = 0.019), US-SLLNM (OR 3.009, 95%CI 1.635-5.536, P<0.001), and CLND number ≤5 (OR 3.244, 95%CI 1.766-5.959, P<0.001) were independent risk factors for cervical SLLNM in PTC patients. The results of the multivariate logistic regression analysis are shown in Table 6 .

Based on the seven independent risk factors described above, the logistic regression equation of the resulting model is as follows: Y= -7.893 + 1.119(χ₁)+ 0.060(χ₂) - 0.298(χ_4-2) + 1.239(χ_4-3) + 0.978(χ₅) + 0.613(χ_6-2) + 1.203(χ_6-3) + 1.053(χ₇) + 1.173(χ₈) (χ₁:Female; χ₂:Age value; χ_4-2: Lower; χ_4-3:Upper; χ₅:Maximum diameter ≤ 10mm; χ_6-2:Capsular distance=0mm; χ_6-3:Capsular distance<0mm; χ₇:US-SLLNM(Yes); χ₈:CLND number ≤ 5).

Apart from this, static ( Figure 7 ) and dynamic nomograms were constructed respectively: https://thyroidnomo.shinyapps.io/dynnomapp.

The new model had an AUC value of 0.869 (95% CI:0.833-0.906), with a sensitivity of 89.9% and a specificity of 69.4%. In order to evaluate the ability of our nomogram to predict SLLNM in PTC patients, we used 1,000 bootstrap resamplings for internal validation and obtained a mean AUC value of 0.870 (95% CI:0.839-0.901), showing good discrimination ( Figure 8A ); the calibration curve was also close to the ideal curve, with good consistency ( Figure 8B ). On the DCA curve, the net clinical benefit was seen from the 0 - 62% prediction probability interval ( Figure 8C ). According to the ROC curve, 7.9% is the optimal critical value of the model. When the prediction probability is 7.9%, a clinically significant net benefit can be obtained. Moreover, 89.9% (62/69) of PTC patients were correctly screened for SLLNM, which is of great significance in guiding the treatment of patients.