Cytologically indeterminate thyroid nodules (ITNs) account for 20–30% of the fine-needle aspiration (FNA) samples from suspicious thyroid nodules (TNs) (1). These nodules correspond to Bethesda categories III–V, classified according to the Bethesda System for Reporting Thyroid Cytopathology (2). Bethesda III, IV, and V nodules carry a malignant risk of 13–30%, 23–34%, and 67–83%, respectively (2). Consequently, more than half of patients with ITNs opt for diagnostic surgery (3), although 60–80% of these excised nodules are benign on final pathological analysis (4, 5). Senior radiologists achieve excellent diagnostic efficacy for Bethesda V TNs using ultrasound (US) features (3). However, diagnosing Bethesda III and IV nodules remains challenging, despite reports that microcalcifications (6) and hypoechoic features (7) can predict malignancy. Grayscale US has significant limitations, exhibiting low diagnostic specificity (44–67.3%) (8–10) and high inter-observer variability (9, 11, 12), particularly for highly suspicious nodules (e.g., ACR TR4 and TR5). Differential diagnosis of ITNs requires a new solution to overcome the impact of radiologists, techniques, and equipment.

Radiomics has emerged as a promising approach for predicting the pathology, prognosis, and lymph node metastasis of TNs (13–18). Radiomics models based on US images demonstrate superior diagnostic efficacy compared with conventional US risk stratification systems (19). These models offer advantages, such as high accuracy (0.761–0.874) (14, 15, 20), lower intra-observer variability (14, 21), and reduced rates of unnecessary FNA procedures (3.1–37.7%) (19, 22) for TNs. However, previous models on common TN perform poorly for ITNs. The well-established artificial intelligence (AI) adjunct diagnostic tools have also demonstrated poor accuracy (e.g., 0.64 in accuracy for 88 Bethesda III nodules), despite achieving an AUC of 0.92 for common TNs (23). Few radiomics studies have focused on ITNs diagnosis or the diagnostic performance of ITN-specific radiomics models remains suboptimal, with area under curves (AUCs) ranging from 0.64 to 0.74 (23–25). A proportion of ITN patients undergo guideline-recommended follow-up observation or ablative minimally invasive treatment, making it difficult to collect ITNs with definitive cytopathology and postoperative histopathology. Due to the absence of such ITNs in training data, pilot studies propose that the efficiency of radiomics models could improve if trained specifically on ITN US images (24, 25). High indices, such as a negative predictive value (NPV) of 93.9% and a positive predictive value (PPV) of 93.8%, have been reported for Bethesda III nodules, indicating the potential utility of these models in supporting follow-up management of benign ITNs (26). However, such studies are limited, involving only dozens of ITNs. The critical questions remain unanswered regarding the diagnostic performance of ITN-specific radiomics models, their potential to enhance radiologists’ diagnostic accuracy, their role in reducing unnecessary aspiration biopsies, and their comparability to published AI adjunct diagnostic tools.

In this study, we aimed to address these gaps by developing an ITN-specific US radiomics model and comparing its performance with that of radiologists and published AI diagnostic tools. We assumed that radiomics could provide invisible and valuable features beyond radiologists’ observation. By combining conventional US and radiomics features of ITNs, the new method could improve the preoperative differentiation between benign and malignant ITNs. Using pathological diagnosis as the gold standard, we developed and evaluated the radiomics models in comparison with the Thynet online tool, the ITS 100 system, and two junior radiologists. Our aim was to improve the accuracy of preoperative ITN diagnosis and minimize unnecessary invasive interventions.

In this study, we developed three radiomics models using US images of ITNs. The models were constructed as follows: the Rad model, based solely on radiomics features; the Radsize model, which incorporated nodular size and radiomics features; and the Radunion model, which included the Bethesda classification along with the Radsize model features. The models achieved diagnostic accuracies ranging from 74.1% to 85.3% across all ITN cohorts, outperforming both junior radiologists and two AI-assisted diagnostic tools. This demonstrates the potential of radiomics models to differentiate malignant from benign ITNs. Notably, the Radsize model reduced unnecessary biopsy rates by at least 13.8%, while the Radunion model could potentially spare 13%–20% of ITNs from diagnostic surgery prior to intervention.

Beyond radiomics features, our findings identified nodular size, Bethesda classification, and microscopic capsular invasion as significant predictive factors for ITN malignancy. Interestingly, none of the five ACR TI-RADS-recommended features (composition, echogenicity, shape, border, and echogenic foci) significantly predicted malignancy in ITNs within our cohort (28). This suggests a potential need to refine conventional diagnostic criteria for ITNs.

In our cohort, benign and malignant ITNs exhibited significant differences in nodule size, which corroborates findings by Xavier et al., who identified nodular size as a key factor in model development (25). The ACR guidelines associate larger nodules with higher malignancy risks, recommending FNA for nodules >2.5 cm in TR3 categories or follow-up for nodules <1.5 cm. However, our results showed that most malignant nodules were smaller, likely reflecting the increased prevalence of papillary thyroid microcarcinomas (<10 mm). Bethesda classification showed that Bethesda IV nodules were at higher risk of malignancy than Bethesda III nodules, which aligns with existing guidelines (2).

The Radsize model demonstrated significantly improved performance in both the training and entire cohorts compared to the Rad model. Furthermore, including the Bethesda classification in the Radunion model enhanced diagnostic precision, reducing the need for diagnostic surgery. Similarly, Grégoire et al. incorporated Bethesda classifications into logistic regression models for Bethesda III–V nodules, demonstrating comparable improvements (20). Although microscopic capsular invasion showed no preoperative diagnostic value, gross extrathyroidal extension diagnosed via preoperative US remains a key determinant for surgical planning in thyroid cancers (29).

The diagnostic accuracy of the Rad model was comparable to that of an SVM-based model by Chen et al. (74.1% vs. 71.8%) (26), which utilized clinical and sonographic features such as composition, echogenicity, margins, shape, echogenic foci, and nodule size in 194 ITNs (Bethesda III/IV/V). The AUC of the Radsize model outperformed that of the ResNet-50 model, which integrated radiomics features from 88 ITNs (0.840 vs. 0.740) (25), and was comparable to the multiple-modality models by Grégoire et al. (20), which combined clinical data with the Bethesda and French TI-RADS categories. These findings suggest that US radiomics may play an important role in enhancing the differential diagnosis of ITNs.

The Radunion model achieved an AUC of 0.860 and the highest accuracy of 76.6% among junior radiologists, the Thynet online tools, and the ITS 100 system. In contrast, the previous Thynet tool, based on a deep learning algorithm and trained on 22,354 US images, achieved an AUC of 0.922 (23) but yielded an accuracy of only 65% in our ITN cohort. Part of Thynet’s training set included Bethesda II or VI nodules, which lack the characteristic features commonly observed in ITN US images. Most training images were from surgical nodules with a high malignant potential, which may explain why the Thynet model was less capable of generalizing to ITN images and tended to assign cases to the malignant category. This could also explain the discrepancies observed in the ITS 100 system. The commercial ITS 100 system, which examined 1,007 TN US images, exhibited a sensitivity of 92.21%, specificity of 83.20%, and accuracy of 89.97% (30). However, in our ITN cohort, the sensitivity was 70.7%, specificity was 61.4%, and accuracy was 65%. Similarly, the S-Detect unit, an AI model for TNs (31), achieved an accuracy of 81.7% for 454 TNs but only an AUC of 0.795 for 159 ITNs (32). In the current cohort, the Radunion model misclassified 47 ITNs, including 11 benign ITNs and 36 malignant ITNs for pathology. The nodule sizes were evenly distributed from 0.3 to 3.7 cm, 26 Bethesda III nodules with a size distribution of 0.3-3.7 cm, and 21 Bethesda IV nodules with a size distribution of 0.4-2.1 cm.

These AI models were designed to reduce clinical workload and improve the efficiency of junior radiologists (33). One of the primary objectives of US radiomic studies is to avoid unnecessary biopsies in patients with benign nodules. Park et al. (22) combined radiomics with the ACR or American Thyroid Association guidelines (3, 28) and found that all readers showed improved performance and reduced unnecessary fine-needle aspiration (FNA) rates. Huang et al. (27) developed a radiomics nomogram that achieved an unnecessary FNA rate of 18.66% while maintaining an accuracy of 82.48% for TNs. The Thynet-assisted strategy, a well-established method, reduced the number of FNAs from 61.9% to 35.2% in a simulated scenario (23). In this study, we provide evidence that the Radsize model can reduce the unnecessary biopsy rate by up to 48.9% compared to junior radiologists, achieving an unnecessary biopsy rate of 21.1%. These results indicate that US radiomics models hold significant promise in the preoperative diagnosis of ITNs, especially for less experienced radiologists.

This study had some limitations. First, the proportion of malignant nodules in the entire cohort of ITNs from a single medical center (71.1%, 140/197) was higher than that reported in other studies (19, 22), potentially introducing selection bias, such as lower specificity and lower NPV. The proportion of malignant cases would influence the generalizability and robustness of models among other dataset. Diagnostic thresholds for models may be low, leading to reduced predictive power for low-risk populations. Models may be overfitted to high-risk characteristics, making it difficult to accurately identify people at average risk. No more populations for external validation is the second shortage. The majority of patients at our center present with higher-risk nodules and tend to prefer ablative therapy when the nodular size <10 mm (34), some patients with nodular size >10 mm also require to try ablation therapy, leaving the patients with bigger or more risky nodule have to undergo the surgery, which results in a higher percentage of malignant nodules among those undergoing surgery. To address this limitation, collaboration across multiple medical centers is needed to further optimize and validate the performance of these radiomics models by different populations. Meanwhile, the retrospective design and potential variability in US image acquisition also effects the results. Thus, collecting row data in prospective research and expanding the range of imaging data, including contrast-enhanced US, microvascular imaging, and super-resolution US, is necessary. Combining multiple-modality models will be promising in improving diagnostic performance and minimizing unnecessary biopsies for ITNs (35, 36). Finally, we should consider further optimization and applicability studies for the model performance. We should establish clear conditions for the applicability of such a model in the clinical process and the management of its use, including a systematic training program for its users.

The US radiomics models developed in this study, particularly the Radsize and Radunion models, demonstrate the potential to serve as convenient and accurate adjunct tools for predicting malignancy in ITN. These models can significantly enhance diagnostic performance, particularly for junior radiologists, by improving accuracy and reducing unnecessary interventions, such as biopsies and surgeries. Our findings highlight the broader implications of adopting radiomics-based approaches in clinical practice, including more standardized diagnoses and improved patient management. Future studies should prioritize validating these models across diverse populations and integrating additional imaging modalities, such as contrast-enhanced and super-resolution US, to further optimize their diagnostic capabilities.