Front. Oncol. Frontiers in Oncology Front. Oncol. 2234-943X Frontiers Media S.A. 10.3389/fonc.2023.1060702 Oncology Review Artificial intelligence in thyroid ultrasound Cao Chun-Li 1 2 † Li Qiao-Li 1 2 † Tong Jin 1 Shi Li-Nan 1 2 Li Wen-Xiao 1 2 Xu Ya 1 Cheng Jing 1 Du Ting-Ting 1 Li Jun 1 2 * Cui Xin-Wu 3 * 1 Department of Ultrasound, The First Affiliated Hospital of Shihezi University, Shihezi, China 2 NHC Key Laboratory of Prevention and Treatment of Central Asia High Incidence Diseases, First Affiliated Hospital, School of Medicine, Shihezi University, Shihezi, China 3 Department of Medical Ultrasound, Tongji Hospital, Tongji Medical College, Huazhong University of Science and Technology, Wuhan, Hubei, China

Edited by: Jan Baptist Vermorken, University of Antwerp, Belgium

Reviewed by: Xiaowen Liang, University of South China, China; Zbigniew Adamczewski, Medical University of Lodz, Poland

*Correspondence: Jun Li, 1287424798@qq.com; Xin-Wu Cui, cuixinwu@live.cn

†These authors have contributed equally to this work

 12 05 2023 2023 13 1060702 03 10 2022 07 04 2023 Copyright © 2023 Cao, Li, Tong, Shi, Li, Xu, Cheng, Du, Li and Cui 2023 Cao, Li, Tong, Shi, Li, Xu, Cheng, Du, Li and Cui

This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner(s) are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.

Artificial intelligence (AI), particularly deep learning (DL) algorithms, has demonstrated remarkable progress in image-recognition tasks, enabling the automatic quantitative assessment of complex medical images with increased accuracy and efficiency. AI is widely used and is becoming increasingly popular in the field of ultrasound. The rising incidence of thyroid cancer and the workload of physicians have driven the need to utilize AI to efficiently process thyroid ultrasound images. Therefore, leveraging AI in thyroid cancer ultrasound screening and diagnosis cannot only help radiologists achieve more accurate and efficient imaging diagnosis but also reduce their workload. In this paper, we aim to present a comprehensive overview of the technical knowledge of AI with a focus on traditional machine learning (ML) algorithms and DL algorithms. We will also discuss their clinical applications in the ultrasound imaging of thyroid diseases, particularly in differentiating between benign and malignant nodules and predicting cervical lymph node metastasis in thyroid cancer. Finally, we will conclude that AI technology holds great promise for improving the accuracy of thyroid disease ultrasound diagnosis and discuss the potential prospects of AI in this field.

 artificial intelligence thyroid ultrasound machine learning deep learning section-at-acceptance Head and Neck Cancer Introduction

With the rapid progress of modern medicine, especially the continuous development of imaging technology, the detection rate of thyroid diseases and thyroid cancer has shown a rapid growth trend both domestically and internationally (1, 2). Various auxiliary examination methods, such as ultrasound, computed tomography, and radioisotope scanning, are used to evaluate thyroid diseases (3). Among them methods, ultrasound has become the primary means of thyroid examination and diagnosis because of its advantages, such as convenience, real-time imaging display, non-radiation, and good tolerance (4, 5). However, the accurate identification of ultrasound images is highly related to physician’s experience, and the differences between different observers can be significant. Therefore, inexperienced physicians are at greater risk of misdiagnosis, underestimating the condition, or unnecessarily performing fine-needle aspiration (FNA) biopsies (6, 7). Therefore, taking advantage of artificial intelligence (AI) in thyroid disease ultrasound screening and diagnosis not only assist the radiologists in achieving more accurate imaging diagnosis with higher efficiency, but also lessen the radiologists’ workload (8).

AI, a branch of computer science that encompasses both machine learning (ML) algorithms and deep learning (DL) algorithms, is gaining increasingly popularity in the field of medicine. Due to its ability to process pixel values and derive insights from images, AI techniques are particularly well-suited for fields that rely on imaging data, such as gastroenterology (9), ophthalmology (10), dermatology (11), pathology (12), radiology (13), and ultrasonography (14). The exponential growth in the volume of medical data over the past decade has spurred the development of AI, which can automatically analyze complex medical images and provide more accurate and efficient diagnoses. By leveraging AI for thyroid disease ultrasound screening and diagnosis, radiologists can reduce their workload and improve the accuracy of their diagnoses.

Several scholars have examined the application of ultrasound AI in thyroid diseases thus far. However, most review studies focus on specific topics, such as distinguishing benign or malignant thyroid nodules or predicting cervical lymph node metastasis using ultrasound features. As a result, there is a need for a comprehensive review of the current state and future possibilities of AI in thyroid ultrasound. This article aims to provide a comprehensive review by discussing the fundamental theoretical knowledge of AI, including traditional ML and DL algorithms, and their clinical application in ultrasonic imaging of thyroid diseases, such as thyroid disease detection, thyroid segmentation, and differential diagnosis of thyroid nodules. Finally, this article addresses the challenges and prospects of AI in the clinical application of thyroid ultrasound.

 Basic theoretical knowledge of AI Conventional ML algorithms

Traditional ML algorithms typically rely on the pre-defined engineered features that accurately describe the regular patterns inherent in data extracted from regions of interest (ROI) with explicit parameters on the basis of expert knowledge. In the medical field, common ML algorithms such as support vector machines (SVM), Bayesian classifiers, et al. rely heavily on these predefined features (15, 16). While these features are considered to be discriminative, conventional ML algorithms are limited by their dependence on expert-defined features and cannot adapt to changes in different imaging methods or variations in signal-to-noise ratios.

 DL algorithms

Unlike traditional ML algorithms, DL algorithms do not require predetermined features and regions of interest set by humans. Instead, they can automatically learn representations of information and gain experience from raw data (17). DL algorithms are composed of simple and nonlinear modules that are particularly effective at extracting features from ultrasound images (18). Various DL architectures have been explored to solve problems, with the convolutional neural network (CNN) being the most commonly utilized architecture (19).

In the 1990s, the use of CNNs expanded to include image processing (20). Compared to other approaches, CNNs utilize spatial and structural information more effectively. The network can directly input the original image, eliminating the need for preprocessing and complex feature extraction procedures that can lead to errors and classification biases. The structure of the convolutional neural network CNN generally includes the following layers: input layer, convolutional layer, pooling layer, fully linked layer and output layer ( Figure 1 ). These layers map the input image information to the critical endpoint in turn through mapping, and learn more advanced image functions at the same time. The convolutional layer is the essential component of a CNN, responsible for extracting features from input images. Through data sharing between the input and output feature maps, the convolutional layer reduces the number of trainable parameters and overall model complexity, thereby facilitating network training. The initial convolutional layer typically extracts basic features, while subsequent layers iteratively extract increasingly complex content from lower-level functions (21). Pooling layers are periodically inserted between consecutive convolutional layers for feature extraction and information filtering. Its function is to decrease the dimension of each feature graph, reduce computing resources and control overfitting effectively while improving the fault tolerance of the model. The operations performed by the pooling layer are usually of the following types: maximum pooling, mean pooling, random pooling, gaussian pooling and training pooling, of which maximum pooling is the most common method (22). When a linear classifier is employed in the classification layer, maximum pooling has better classification performance than average pooling due to its better classification performance (23). The fully connected layer in a CNN serves as the classifier for the entire network, responsible for categorizing the extracted features. It combines the local information with feature identification from the convolution and pooling layers, refitting the extracted features and reducing loss of information. Due to its numerous connection weights, overfitting is a potential risk with the fully connected layer. To mitigate this risk, sparse connections and dropout methods have been suggested (24).

A typical convolutional neural network model.

 Application of AI in thyroid ultrasound Detection of thyroid disease

Thyroid nodule images obtained through ultrasound are often distorted by echo disturbance and speckle noise. Thus, accurate recognition of these images typically requires the expertise of experienced physicians. However, DL algorithms that incorporate multiple image patterns have become increasingly prevalent in detecting thyroid lesions (25–28). As an automated method, a computer-aided detection system (CAD) can recognize and process predefined features, which is used in clinical practice. The combination of CAD and ultrasound in thyroid ultrasound image detection, compared with visual assessment, this method is helpful to find the pathological features that cannot be recognized by the naked eye, so as to improve the detection rate of thyroid lesions.

Ma et al. (25) performed a cascaded CNN to detect thyroid nodules automatically, utilizing 21532 images from 5842 patients. The model was designed to bypass potential errors that might arise during preprocessing, leading to inaccurate results and classification bias due to the feature set’s lack of robustness. And the result demonstrated that the model performs good detection efficiency with an area under the summary receiver operating characteristic curve (AUROC) of 98.51%. Another study by Li et al. (26) developed a papillary thyroid cancer detection model based on R-CNN, which demonstrated a sensitivity of 93.5%. Liu et al. (27) also employed the multi-scale detection network to automatically detect thyroid nodules with an accuracy rate of 97.5%. Acharya et al. (28) proposed a CAD system called ThyroScan, which utilizes seven significant wavelet features extracted from thyroid images of 232 normal thyroid and 294 Hashimoto’s thyroiditis patients. The fuzzy classifier showed an accuracy of 85% in detecting Hashimoto’s thyroiditis.

 Segmentation: to achieve the segmentation of the precise boundary of the lesion

Thyroid ultrasound image segmentation, as one of the most commonly used image preprocessing methods, is usually used to detect and diagnose nodules and to estimate the volume. It is an essential part of CAD systems and the diagnosis of thyroid diseases (29). However, Raw thyroid ultrasound images contain inaccurate and incomplete information, leading to erroneous segmentation results. Therefore, precise segmentation of thyroid nodules is essential for accurate diagnosis of thyroid nodules. In addition, thyroid segmentation can be applied to estimate thyroid volume, which helps evaluate thyroid hormone secretion. Therefore, thyroid segmentation and thyroid nodule segmentation are essential links to improve the development of thyroid AI research, which is conducive to providing a reliable theoretical basis for radiologists’ diagnostic decisions.

Thyroid segmentation and nodule segmentation are classified into contour- and shape-based methods, region-based methods, machine and DL methods, and hybrid methods (30). Although these methods are used with different objectives, they share the same classification.

a) Contour- and shape-based methods: The method processes ultrasound images of the thyroid by obtaining information about the border or shape of the thyroid or thyroid nodule. Edge segmentation is a primary image segmentation method using a different edge detection operator (31). And the characteristic of contour segmentation is that the energy function is used as a measure of the coincidence between the prior model and the image data to make the contour curve approach the target contour (32). However, interfering with original image contrast and image artefacts, borders between thyroid nodules were sometimes discontinuous or false borders were detected. In addition, thyroid nodules are often irregular. Therefore, initial contour and prior shape information are usually required to improve the segmentation accuracy. b) Region-based method: This approach assumes that different regions in thyroid ultrasound images are inhomogeneous, thereby obtaining a minimized boundary energy function. However, when the differences between different regions are less significant, the application of this method is limited. c) Machine and DL methods: The method is based on a machine and DL algorithm to construct a classifier that can automatically extract features, and finally segment the target tissue area accurately, which increases the accuracy of classification. However, machine and DL classifiers require an amount of training data and take a long time to train (21). d) Hybrid methods: The above two or more methods are combined to achieve the purpose of improving the segmentation accuracy.

Thyroid segmentation

In several studies, different automatic segmentation methods were used to segment thyroid ultrasound images with varying degrees of success. Poudel et al. (33) employed the graph cut (GC) method (34) to segment 1416 images, achieving a dice coefficient of 76.5%. Meanwhile, Narayan et al. (29) developed a method based on the principle of echo consistency to segment 52 thyroid images, resulting in a dice coefficient of 84.47%, higher than the average of 83.23% obtained by two experts. Chang et al. (35) developed a radial basis function neural network (RBFNN) approach to automatically segment thyroid gland in 3D. The segmentation method includes four steps thyroid region localization and image enhancement, feature extraction, training RBFNN and thyroid recovery. This study used 60 training patterns to train the RBFNN and evaluated the performance of the method by testing thyroid ultrasound images. The study found that the accuracy of the method is 96.52%. Selvathi et al. (36) developed support vector machines and extreme learning machines to develop an automatic segmentation method, resulting in segmentation accuracies of 84.78% and 93.56%, respectively. Poudel et al. (33) also proposed a 3D U-Net CNN model (37) for thyroid segmentation and compared its performance with four other methods(ACWE, GC, RF, and DT), achieving an average dice coefficient of 87.6%, which was higher than other methods.

 Thyroid nodule segmentation

Nugroho et al. (38) used bilateral filtering (39) to preprocess thyroid ultrasound images and then applied an active contour without edges (ACWE) model to segment thyroid nodules. The approach produced clearer nodule localizations and more accurate segmentation outcomes. However, the ACWE model assumes homogeneity of both foreground and background in the thyroid ultrasound image, leading to potential inaccuracies. To overcome the limitations of the ACWE model, Maroulis et al. (40) improved the ACWE model and proposed a variable background active contour (VBAC) model, which can mitigate the effects of inhomogeneous tissue in ultrasound images. The VBAC model was used to segment 71 thyroid nodule ultrasound images, and compared with the ACWE model. The VBAC model achieved a higher average overlap value of 91.1% compared to the ACWE model’s 84.8%, indicating its superior performance in nodule segmentation. Mylona et al. (41, 42) added Orientation Entropy (OE) based on the ACWE model to make the contour of the nodule closer to the target edge and leave the continuity of the contour unaffected. This study showed that the OE-ACWE model evolves faster than the ACWE model, and the average overlap rate of segmentation results was 83.70%. In addition, some scholars have studied the segmentation of thyroid nodules using DL methods. Ma et al. (43) proposed a CNN model to segment nodules in 22,123 thyroid ultrasound images with an average overlap of 86.83% using ten-fold cross-validation. Some scholars (44) have also proposed a cascaded convolutional neural network (CCNN) model for thyroid nodule segmentation, using 1000 images in the dataset and achieving an average overlap rate of 87.00% in the test set.

In addition, hybrid segmentation techniques have been investigated to improve segmentation models and automate the segmentation of solids. Zhou et al. (45) combined the GC model and the ACWE model to segment thyroid nodules. After research, it is concluded that the hybrid model performed better in segmentation and addressed boundary leakage issues more effectively. However, this study lacked quantitative results and could not yet be compared with other models. Legakis et al. (46) combined a maximum likelihood algorithm and an active contour model to segment nodules in thyroid ultrasound images. The study found that the average overlap rate of segmentation results is 92.30%.

Thyroid segmentation and thyroid nodule segmentation have become indispensable for modern medical ultrasound imaging diagnosis, aiding clinicians in making optimal diagnostic decisions. However, they do have certain limitations. Firstly, the current research on thyroid segmentation mainly focuses on the segmentation of normal thyroid, leaving the segmentation of an abnormal thyroid underexplored. The size and shape of abnormal thyroid tissue may present new challenges for thyroid segmentation. Secondly, most studies related to thyroid nodule segmentation aim to distinguish the nature of thyroid nodules but do not identify specific disease types.

 Differentiation of malignant and benign thyroid nodules ML

Currently, some studies have combined the maximum likelihood algorithm with the analysis of ultrasonic image texture features for the differential diagnosis of thyroid nodules ( Table 1 ). The main objective of most researchers is to evaluate the efficacy of ML algorithms in distinguishing between benign and malignant thyroid nodules. In this regard, CAD systems that utilize ML algorithms have become increasingly significant in assisting ultrasound imaging to enhance the precision of nodule assessment.

Main results of ML algorithm in thyroid nodules ultrasound image studies.

Reference Published year Technique No. ofsubjects Features Classifier Main Performance
Hirning et al (47) 1989 US 55 Hist, GLCM SVM Accuracy:85.0%
Tsantis et al (16) 2005 US 120 Hist, GLCM,GLRLM SVM Accuracy:96.7%
Tsantis et al (48) 2009 US 85 Shape and size, Fractal, Wavelet SVM, PNN AUROC:0.96
Savelonas et al (46) 2009 US 171 His, shape and size, Fractal SVM, k-NN AUROC:0.95
Chang et al (49) 2010 US 61 GLCM, GLRLM;Wavelet SVM Accuracy:100%
Iakovidiset al (50) 2010 US 75 Hist, GLCM, FLBP SVM Accuracy:97.5%
Legakis et al (51) 2011 US 142 Textural, shape feature vectors SVM AUROC:0.93
Luo et al (52) 2011 Elastography 98 Waveform LDA Sensitivity:100%; Specificity:75.6%
Ding et al (53) 2011 Elastography 125 Hist,GLCM SVM Accuracy:93.6%; AUROC:0.97
Acharya et al (31) 2011 3D CEUS 20 GLCM, Wavelet K-NN, PNN, DT Sensitivity:98.0%; Specificity:99.8%;Accuracy:98.9%;AUROC:0.99
Acharya et al (54) 2012 3D HRUS 20 GLCM, Wavelet AdaBoost Sensitivity:100.0%; Specificity:100.0%;Accuracy:100.0%;AUROC:1.00
Acharya et al (55) 2012 3D CEUS 20 GLCM, Wavelet K-NN, PNN, DT Sensitivity:98.0%; Specificity:99.8%;Accuracy:98.9%
Acharya et al (56) 2012 3D HRUS3D CEUS 20 FD, LBP, FS, LTE SVM, DT, Sugeno Fuzzy, GMM, KNN, PNN, NB HRUS Accuracy:100.0%CEUS Accuracy:98.1%
Acharya et al (57) 2013 3D CEUS 20 Wavelet Fuzzy Accuracy:99.1%
Zhu et al (58) 2013 US 689 na ANN Accuracy:83.1%; AUROC:0.83
Kim et al (59) 2015 USRTE 613 Hist, GLCM na AUROC:0.68
Song et al (60) 2015 US 155 GLCM SVM, RT, RF, boost, logistic, ANN AUROC:0.84
Ardakaniet al (61) 2015 US 70 Hist, GLCM, GLRLM 1-NN, ANN Sensitivity:94.5%; Specificity:100.0%;Accuracy:97.1%;AUROC:0.97
Ardakaniet al (62) 2015 US 60 Wavelet 1-NN Sensitivity:100.0%; Specificity:100.0%;Accuracy:100.0%;AUROC:1.00
Acharya et al (63) 2016 3D HRUS 242 Safe-LevelSMOTE SVM, KNN, MLP,C4.5decision tree Accuracy:94.3%
Bhatia et al (64) 2016 SWE 105 GLCM na Sensitivity:97.5%; Specificity:90.0%;AUROC:0.97
Chang et al (65) 2016 US 59 Hist, intensity differences, elliptical fit, GLCM, GLRLM SVM AUROC:0.99
Wu et al (66) 2016 US 970 na NB, SVM, RBF-NN AUROC:0.91
Raghavendra et al (67) 2017 US 242 SGLD, Fractal DT, LDA, QDA, NB, PNN, k-NN, SVM with differentkernel functions Sensitivity:90.3%; Specificity:98.6%;Accuracy:97.5%; AUROC:0.94
Yu et al (68) 2017 US 543 Hist, GLCM,GLRLM, NGLDM,Fractal SVM, ANN Sensitivity:100.0%; Specificity:87.9%;Accuracy:92.0%
Zhang et al (69) 2019 USRTE 2064 na k-NN, RF, k-SVM,Logistic, LDA, CNN, adaptive boosting, NB, neural network AUROC:0.94
Ouyang et al (70) 2019 US 1036 na Ridge, Lasso-penalty, Elastic Net, RF,k-SVM, Neural Network AUROC:0.95
Shin et al (71) 2020 US 348 GLCM, GLRLM, Gabor, and Haar wavelet SVM, ANN SVM: Accuracy:79.4%ANN: Accuracy:69.0%
Zhao et al (72) 2021 USSWE 743 contour, shape, textural phenotype, Hist, GLCM, GLRLM, GLSZM,NGTDM,GLDM,LTP,LDP,LBP, ect. DT, NB, KNN, logistics regression, SVM, KNN-based bagging, RF, extremely randomized trees (xgboost), multi-layer perception, and gradient boosting tree ML‐assisted SWE+US visual approach: AUC:0.951SWE+US radiomics approach: AUC:0.834
Vijay et al (73) 2021 US 99 GLCM ANN, SVM Accuracy:96.0%
Matti et al (74) 2022 US 8339 Hist, GLCM Random forest AUC:0.75
M. Keutgen et al (75) 2022 US 1052 Gray-level co-occurrence matrix texture Two-class BANN AUC:0.75

US, ultrasonography; CEUS, contrast-enhanced ultrasonography; HRUS, high-resolution ultrasonography; RTE, real-time elastography; SWE, shear wave elastography; AUROC, area under ROC curve; Hist, histogram; GLCM, Gray-Level Co-occurrence Matrix; GLRLM, Gray-Level Run-Length Matrixes; NGLDM, Neighboring Gray-Level Dependence Matrix; FLBP, fuzzy local binary pattern; SVM, Support Vector Machines; PNN, Probabilistic Neural Network; KNN, k-Nearest Neighbor; ANN, artificial neural network; 1-NN, first–Nearest Neighbor; DT, Decision Tree; RT, Random Tree; RF, Random Forest; RBF-NN, radial basis function-neural network LDA, Linear Discriminant Analysis; GMM, Gaussian Mixture Model; QDA, Quadratic Discriminant Analysis; NB, Naive Bayes; FD, Fractal Dimension; LBP, Local Binary Pattern; FS, Fourier Spectrum; LTE, Laws Texture Energy; MLP, multi-layered perceptron; Two-class BANN, two-class Bayesian artificial neural networks; na, not available.

In 1989, Hirning et al. (47) published the first study to differentiate thyroid nodules based on ultrasound texture analysis. The overall accuracy of their classification system in classification has reached 85%. Chang et al. (49) used 78 texture features to describe thyroid ultrasound images and applied SVM to classify the images, achieving a remarkable accuracy rate of 100%. Acharya et al. (31, 54–57, 63) proposed a CAD system for automatically classifies malignant and benign thyroid nodules using 3D high-resolution and contrast-enhanced ultrasound (CEUS) images. The classification accuracy of different classifiers tested with these features ranged from 98.1% to 100%, indicating that the CAD system could support radiologists in identifying the nature of thyroid nodules. Raghavenra et al. (67) proposed a fusion method to identify the maturity of thyroid lesions, that is, the spatial gray scale correlation feature (SGLDF) and fractal texture system fusion. The classification efficiency of the SVM classifier was high, with an accuracy rate of 97.5% and a maximum AUROC of 0.95.To determine which ML classifiers have higher classification performance for thyroid nodules, Zhang et al. (69) nine ML classifiers (K-NN, CNN, Random Forest, Logistic, adaptive enhancement, Naive Bayes, neural networks, etc.) were evaluated for their classification performance of thyroid nodules using conventional ultrasound and real-time elastography features in 2064 thyroid gland samples, and compared to experienced radiologists. The Random Forest algorithm demonstrated the highest diagnostic performance among all the classifiers tested. Based on the handcrafted image features, Ouyang et al. (70)analyzed the classification efficacy of linear and nonlinear ML methods when processing thyroid data. They found that both methods had similar accuracy and a simpler prediction process compared to a CAD system, as there was no need to preprocess the image or extract texture features from it.

 DL

The integration of DL techniques with ultrasonography has garnered considerable interest in the identification of benign and malignant thyroid nodules ( Table 2 ). Based on the DL model, it can learn useful texture features to extract features or classify thyroid nodules automatically in ultrasound images, which overcomes the limitations of manual methods.

Main results of DL algorithm in thyroid nodules ultrasound image studies.

Reference Published year Number of subjects Type of DL Main Performance
Ma et al (76) 2017 15000 images fusion of two pre-trained CNNs on the ImageNet Accuracy:83.0%;Sensitivity: 82.4%;Specificity: 85.0%.
Ma et al (43) 2017 22123 images CNN Sensitivity: 91.5%
Chi et al (77) 2017 693 images Fine-Tuning DCNN Accuracy:99.1%
Zhu et al (78) 2017 298 nodules Fine-Tuning DCNN(ResNet18-based network) Accuracy:93.8%
Gao et al (79) 2018 342 nodules multiple-scale CNN Accuracy:82.2%
Peng et al (80) 2021 18049 images ThyNet AUROC:0.921.
Zuo et al (81) 2018 19260 images Alexnet CNN Accuracy:86.0%
Zhu et al (82) 2018 467 nodules DNN Accuracy:87.2%
Buda et al (83) 2019 1377 images(training set:1278;test set: 99) Multitask DCNN Sensitivity:87.0%; Specificity:52.0%.
Guan et al (84) 2019 2836 images(training set:2437;test set:399) DL(Inception-v3) Sensitivity: 93.3%;Specificity: 87.4%.AUROC:0.87
Li et al (85) 2019 42952 training samplesand 2692 test samples DCNN Sensitivity: 93.4%; Accuracy:89.8%;AUROC:0.95
Nguyen et al (86) 2019 237 nodules DCNN Accuracy:90.88%
Song et al (87) 2019 1358 nodulesTest set (internal:55; external:100). DL(Inception-v3) internal test set: Sensitivity:95.2%;external test set: Se:94.0%
Song et al (88) 2019 4309 images multitask cascade CNN Accuracy: 98.2%
Sundar et al (89) 2019 2525 training samplesand 613 test samples Inception-v3, VGG-16 Inception-v3+CNN: Accuracy:93.0%;VGG-16+CNN: Accuracy:79.0%;
Wang et al (90) 2019 5007 nodules YOLOv2 neural network Accuracy:90.3%
Nguyen et al (91) 2020 450 images DCNN Accuracy:92.1%
Wang et al (64) 2020 7803 images CNN Accuracy:84.6%
Wu et al (92) 2021 2082 images(90% training images and 10% testing images) CNN(ResNet-50、Inception-Resnet v2、Desnet-121) ResNet-50: Accuracy:87.4%Inception-Resnet v2: Accuracy:84.6%Desnet-121: Accuracy:84.6%
Zhu et al (93) 2021 16401 training samplesand 1000 test samples CNN (BETNET) Accuracy:98.3%
Liu et al (94) 2021 163 pairs images IF-JCNN Accuracy:89.6%
Kim et al (95) 2022 12327 training samplesand 3082 test samples DL (VGG16, VGG19, ResNet) VGG16: Accuracy:78.0%VGG19: Accuracy:74.0%ResNet: Accuracy:75.0%

Ma et al. (76) first proposed the classification of thyroid nodules based on the CNN fusion method. Their findings indicated an 83.02% diagnostic accuracy for this approach. Chi et al. (77) first attempted to combine the DL method with the TI-RADS scoring system to propose a classification system for thyroid images, utilizing the deep CNN GoogLeNet. By fine-tuning the existing DL network, they achieved a classification accuracy of 99.13%, leading to improved effectiveness of CAD systems in thyroid nodule evaluation. Peng et al. (80) developed a ThyNet-based DL model to differentiate benign and malignant thyroid nodules in a multicenter study, which showed that the AUROC was significantly higher in subjects with ThyNet diagnosed benign and malignant thyroid nodules than in radiologists (0.922 vs 0.839, P< 0.0001). And With the assistance of ThyNet, the number of fine needle aspirations decreased from 61.9% to 35.2%, while the number of missed malignant thyroid nodules decreased from 18.9% to 17.0%. It was concluded that ThyNet could significantly improve radiologists’ diagnosis and help to reduce unnecessary fine needle punctures of thyroid nodules.

Nguyen et al. (86) have developed a method for feature extraction from thyroid images using a cascade classifier architecture to enhance the performance of CAD systems for thyroid nodule classification. This approach combines both handcraft and DL, achieving an overall accuracy of 90.88%. Considering the differences in the DL network structure and the imbalance of image samples, the same group (91) artificially reduced the influence of the imbalance of training samplesby employing a weighted binary cross-entropy loss function in training multiple CNN models. This method achieved a 92.05% accuracy rate for thyroid ultrasound images. Wu et al. (92) combined ACR TI-RADS with CNN to train three commonly used DL algorithms to distinguish malignant from benign thyroid nodules in TI-RADS 4 and TI-RADS 5. The method showed a significant ability to distinguish malignant and benign nodules, demonstrating high clinical value. Liu et al. developed a joint convolutional neural network (IF-JCNN) based on information fusion to improve the diagnostic performance of thyroid nodules. The IF-JCNN was able to achieve an accuracy and AUROC of 0.896 and 0.956, respectively, which outperformed those obtained using only US images (94). Since multiple images from different angles are necessary for a thorough thyroid ultrasound examination, most methods only utilize a single US image for diagnosis. Wang et al. (96) proposed a new CNN structure for attention-based feature aggregation networks that can aggregate features extracted from multiple images in a single inspection. This method improves the ability to identify malignant thyroid nodules using different views.

 S-Detect

S-Detect (Samsung RS80A ultrasound system, Seoul, Korea) is the first commercially available ultrasound CAD based on DL technology for thyroid imaging. The system employs a CNN model that is trained using various TI-RADS hierarchical knowledge to automatically identify and analyze multiple grayscale ultrasound image features, including the internal structure, echo height, boundary, direction, and shape of thyroid nodules. After selecting the region of interest, the system can quickly determine whether the nodule is benign or malignant, either automatically or through manual intervention ( Figure 2 ). Many studies have been carried out on the effectiveness of S-Detect system in differentiating between malignant and benign thyroid masses (97–109) ( Table 3 ). Despite being a novel technique, its clinical applicability remains controversial, and different experiments have yielded varying results. Several studies have evaluated the diagnostic efficacy of the S-Detect system for identifying malignant thyroid nodules. Choi et al. (97) indicated that the difference in sensitivity of the S-Detect system for the diagnosis of malignant thyroid nodules compared to radiologists with 20 years of experience was not statistically significant (88. 4% vs 90.7%, P > 0.05), while the specificity(94.9% vs 74.9%, P<0.05) and discriminative power (AUROC 0.92 vs 0.83, P<0.05) of the S-Detect system were inferior to those of experienced radiologists. In contrast, the results of the S-Detect test by Gitto et al. (98) displayed no statistically significant difference in specificity between the software and radiologists with 5 years’ experience (66.7% vs 81.3%, P > 0.05)., but the diagnostic sensitivity of the software was inferior to the CAD system (21.4% vs 78.6%, P<0.05). Yoo et al. (109) found that the S-Detect system had comparable diagnostic performance to a radiologist with 10 years of experience, and could improve the diagnostic sensitivity and negative predictive value of less experienced radiologists. Jeong et al. (108) found that experienced radiologists had higher sensitivity and accuracy than less experienced radiologists when using the S-Detect system. A subgroup meta-analysis by Zhao et al. (107) showed that the S-Detect system had similar sensitivity to experienced radiologists, but lower specificity.

Thyroid nodules S-detect technique in the Samsung RS80A ultrasound system. (A, B) In a 35-year-old woman with right Hashimoto’s thyroiditis with focal fibrosis on conventional ultrasound (A), S-Detect comes to the correct conclusion through analysis as “Possibly Benign” (B); (C, D) In a 52-year-old woman with left thyroid cancer on conventional ultrasound (C), S-Detect comes to the correct conclusion through analysis as “Possibly Malignant” (D).

Summary of related studies reported on thyroid nodules of compared diagnostic efficacy between the S-Detect and experienced radiologists.

Author Nodules Sensitivity Specificity Accuracy PPV NPV AUROC
Benign/maligant S-D Ra S-D Ra S-D Ra S-D Ra S-D Ra S-D Ra
Choi et al (97) 102(59/43) 90.7% 88.4% 74.6% 94.9% 81.4% 92.2% 72.2% 92.7% 91.7% 91.8% 0.83 0.92
Gitto et al (98) 62(48/14) 21.4% 78.6% 81.3% 66.7% 67.7% 69.4% 25.0% 40.7% 78.0% 91.4% -* -*
Yoo et al (109) 117(67/50) 80.0% 84.0% 88.1% 95.5% 84.6% 90.6% 83.3% 93.3% 85.5% 88.9% 0.84 0.90
Jeong et al (108) 100(56/44) 88.6% 84.1% 83.9% 96.4% 86.0% 91.0% 81.3% 94.9% 90.4% 88.5% -* -*
Xia et al (106) 180(85/95) 90.5% 81.1% 41.2% 83.5% 67.2% 82.2% 63.2% 84.6% 79.5% 79.8% -* -*
Kim et al (99) 218(132/86) 80.2% 84.9% 82.6% 96.2% 81.7% 91.7% 75.0% 93.6% 86.3% 90.7% -* -*
Fresilli et al (100) 107(80/27) 70.4% 81.5% 87.5% 88.8% 65.5% 71.0% 89.7% 93.4% -* -* 0.79 0.85
Wei et al (101) 204(112/92) 91.3% 83.7% 65.2% 63.4% 77.0% 72.5% 68.3% 65.3% 90.1% 82.6% 0.78 0.74
Han et al (102) 454(287/167) 97.6% 97.6% 21.6% 36.2% 49.6% 58.8% 42.0% -* 93.9% -* -* -*
Chung et al (103) 165(140/25) 92.0% 72.0% 87.9% 85.0% 88.5% 83.0% 57.5% 46.2% 98.4% 94.4% -* -*
Barczyński et al (104) 50(40/10) 90.0% 90.0% 95.0% 80.0% 94.0% 82.0% 81.8% 52.9% 97.4% 97.0% -* -*
Molnár et al (105) 200(185/15) 88.6% 80.0% 88.1% 40.5% 88.0% 43.5% -* -* -* -* 0.94 0.66

PPV, Positive predictive value; NPV, negative predictive value; AUROC, area under the summary receiver operating characteristic curve; S-D, S-Detect; Ra, Radiologist. ∗The exact data were not reported in the original articles.

Xia et al. (106) were the first to utilize the S-Detect software to evaluate thyroid cancer subtypes. They found that the S-Detect system exhibited higher diagnostic sensitivity than experienced radiologists in detecting papillary thyroid cancer and follicular thyroid cancer. Nonetheless, the radiologists demonstrated superior diagnostic specificity compared to CAD systems. Kim et al. (99) evaluated the diagnostic efficacy of the S-Detect™ software (Rs85A) in its new version, which was assigned to classify calcification into four distinct categories. However, the accuracy of calcification identification limits the diagnostic performance of S-Detect. In terms of characterizing thyroid nodules, the S-Detect system and radiologists generally agreed on most sonographic features, but there were discrepancies when it came to margin definition. The study of Choi et al. (97) considered that radiologists and the S-Detect system described composition, orientation, echogenicity, and sponginess in substantial agreement (Kappa= 0.66, 0.74, 0.73, 0.66, respectively), while marginal definition showed a fair greement (Kappa = 0.239). Similar findings were reported by Xia et al (106). In addition, Gitto et al. (98) performed an inter-observer agreement between the S-Detect system and the radiologist with a Kappa value of only 0.03 for the margin assessment.

S-Detect is the first commercially available ultrasound CAD based on DL technology for Thyroid. It is specifically designed to support inexperienced radiologists in identifying thyroid nodules’ ultrasound characteristics and thereby enhance their diagnostic accuracy. However, despite its promising potential for clinical use, the S-Detect system’s performance is still largely dependent on the operator. Moreover, the system necessitates the manual input of certain features, and several attempts may be required to segment lesions correctly. Therefore, the system’s performance requires further development, including automatic detection of nodule calcifications and margins. This development will not only save analysis time but also enhance physicians’ overall performance in diagnosing nodules.

 Prediction of lymph node metastasis in thyroid cancer

Assessing the recurrence and prognosis of thyroid cancer heavily relies on the status of lymph nodes, making it a significant indicator in the diagnostic process (110). Among various imaging techniques, ultrasound has become a preferred method due to its non-invasiveness, real-time monitoring, and convenience, providing essential information for diagnosis and treatment (111). To predict lymph node metastasis, the traditional risk prediction model based on risk factors such as tumor size, microcalcification, Hashimoto’s disease (112–115), and blood markers (116, 117), has been commonly used. Predictive models have been constructed to assess the lymph node status of thyroid cancer patients using ultrasound examination, with the area under the AUROC ranging from 0.67 to 0.80. At the same time, other analytical methods, especially radiomics and CNN models, have attracted significant attention because of their feasibility in exploring the correlation between ultrasonic features and the lymph node status of thyroid cancer (118, 119).

The CAD system utilizing DL algorithm was applied to predict lymph node status of thyroid cancer. Lee et al. (120) developed a CAD system using the VGG-Class activation map model to determine lymph node status. The study found that the model’s accuracy was 83.0% and exhibited good diagnostic performance. The system also provides reliability scores and identified regions associated with lymphatic metastasis from ultrasound images. Notably, this study established the first DL-based CAD system intended to assess lymph node status. Some scholars (120) developed a CAD system based on the CNN model to locate and identify the lymph node status of thyroid cancer. The results showed that the accuracy of the CAD system is 83.0% on the test set and effectively detected and diagnosed the location and nature of lymph nodes.

However, the application of the DL algorithm in auxiliary ultrasonic image diagnosis is not mature enough. One reason is the limited number of available ultrasonic images, which also have low resolution that prevents the algorithm from detecting typical features. Additionally, most current studies do not consider various clinical data, such as medical history, clinical inspection results, and other relevant information (121). Some scholars (122) proposed a deep multichannel learning network called MMC-Net to predict lymphatic metastasis of thyroid cancer. The study used clinical data, two-dimensional ultrasound and color Doppler flow imaging (CDFI) images as inputs and proposed a new index to compare the contribution of different channels to prediction. The proposed multi-channel DL network achieved an average F1 score of 0.888 and an average AUC of 0.973, outperforming three single-channel networks. These results indicate that the MMC-Net model is a more effective approach for predicting lymphatic metastasis of thyroid cancer. In recent years, researchers have shown interest in the application of radiology to predict the lymph node status of thyroid cancer. This has been achieved through the use of quantitative medical imaging features. By mining quantitative image feature data in a high-throughput manner and integrating it into clinical decision-making systems, it has been possible to improve the diagnostic accuracy of clinicians (119, 123, 124). Liu et al. (119) analyzed the lymph node status of patients with preoperative thyroid papillary carcinoma based on radiological methods. The radiomics method showed a prediction accuracy of 0.712, indicating the feasibility of radiological analysis of ultrasonic images of patients with thyroid papillary carcinoma. Jiang et al. (125) developed a multimodal ultrasound-based nomogram to predict lymph node metastatic status in papillary thyroid carcinoma. Multimodal ultrasound techniques include shear wave elastography and conventional ultrasound. The results showed that for univariate analysis, the radiological features of B-mode ultrasound and shear-wave elastography radiomics score were significantly correlated with lymph node status. However, the B-mode ultrasound radiomics score did not appear in the final nomogram.

Accurate identification and complete removal of metastatic lymph nodes during preoperative thyroid cancer treatment is crucial for preventing postoperative recurrence. The future CAD system holds promise in predicting metastatic lymph nodes with greater efficiency and accuracy, providing valuable insights for clinical diagnosis and treatment decision-making.

 Challenges and future perspectives

In clinical ultrasound medicine, it is controversial when AI technology can be automatically applied in the clinic, with speculations for the time ranging from a few years to decades. Despite many studies that have confirmed the effectiveness of AI and achieved satisfactory results, most articles have used retrospective analysis or single-center controlled studies, which may lead to inevitable selection bias. For example, the samples in the training set are small, or the selected samples are not from the screening of thyroid nodules, but from the thyroid nodule population with pathological results, which leads to the unrepresentative model or the wrong description of the real population, thus affecting the universality of the model. Therefore, detailed system verification is needed before AI is applied to practical clinical practice. In order to carry out reliable and independent clinical verification, multi-center prospective research is needed in the future, and appropriate inclusion/exclusion criteria are set to make the selection of target population representative, and unused data sets are used for external verification. When the learning model adjusts itself too much on the training data set or the data set used for model development cannot fully represent the patient range (target population) to be applied to clinical practice, over-fitting or spectrum bias will occur (126, 127). Overfitting and spectrum bias may lead to overestimation of accuracy and generalization ability. Therefore, in order to correctly verify the accuracy of AI, doctors should evaluate the performance of AI by avoiding the influence of over-fitting and spectrum bias. DL can handle the complex relationship between dependent variables and independent variables and can make abstract inference at multiple levels. However, this complexity also makes the model a “black box” where the decision making mechanism is not clearly demonstrated, which is not conducive to building social acceptability (128), so further research is needed to address model interpretability or explainability.

The downside of the AI tool is that it can’t solve multiple tasks, and being good at one task doesn’t necessarily mean being good at other tasks. In addition, the difference between the actual efficiency of AI results and the expected results and the cost-effectiveness must be proved by complex and extensive investigations. In the current medical environment, the acquisition of reasonable regulations and reimbursement policies from relevant departments is crucial for the progress of AI technology. At present, a common shortage of AI tools is that they cannot resolve multiple tasks. There is currently no comprehensive AI system capable of detecting multiple abnormalities throughout the human body.

We believe that the future AI system will increase the efficacy of detecting thyroid nodules and predicting the lymph node status of thyroid cancer. At the same time, it can further distinguish specific benign and malignant diseases, such as thyroiditis, thyroid adenoma and nodular goiter. Besides, In the aspect of thyroid ultrasound-guided puncture biopsy and microwave ablation, the AI navigation intervention system can be further developed and perfected. The puncture position can be monitored in real-time by computer software to improve the accuracy of puncture. In addition, complementary information is provided by creating a DL model trained on multimodal images to further improve the diagnostic performance of the DL model.

 Conclusion

At present, the application of AI in medicine has achieved satisfactory achievement, especially in the recognition and diagnosis of imaging pictures. It is rapidly emerging as a promising adjunct to thyroid ultrasound imaging tasks, satisfying the desire of clinical care to improve the efficiency of medical imaging. As an advanced technology, AI has changed the dependence and subjectivity of traditional ultrasound diagnosis on operator’s experience. In addition, AI can also improve diagnostic efficiency and reduce the burden on radiologists. With the continuous increase in the amount of data, AI will be the domain development direction of thyroid ultrasound diagnosis in the future. In order to utilize AI wisely, radiologists must keep up to date with its feasibility, consider the strengths and limitations of different algorithms, understand the impact of overfitting and spectral bias on AI performance, understand that DL technology has its own “black box” nature (lack of interpretability or explainability), and that radiologists need to attempt to compensate for its shortcomings by building rich heterogeneous image datasets, using unused datasets for external validation, etc. We believe that AI will not replace the dominant role of human doctors. Still, AI can provide a credible rationale for doctors to make clinical decisions in some regions of imaging functions.

 Author contributions

C-LC, Q-LL, and JL contributed to the conception and design of the study. JC and JT searched and reviewed studies, extracted and analyzed the data, and wrote the first draft of the manuscript. L-NS, W-XL, and YX reviewed and edited the manuscript. JC, T-TD, X-WC, and JL directed the project and contributed to discussion as well as reviewed and edited the manuscript. All authors contributed to the article and approved the submitted version.

 Funding

This study is funded and supported by Open Research Fund of NHC Key Laboratory of Prevention and Treatment of Central Asia High Incidence Diseases, supported by the Non-profit Central Research Institute Fund of Chinese Academy of Medical Sciences (NO.2020-PT330-003) and grant from the Corps Science and Technology Key Project (No. 2019DB012).

 Conflict of interest

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

 Publisher’s note

All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.

 References Bray F Ferlay J Soerjomataram I Siegel RL Torre LA Jemal A . Global cancer statistics 2018: globocan estimates of incidence and mortality worldwide for 36 cancers in 185 countries. CA: Cancer J Clin (2018) 68(6):394–424. doi: 10.3322/caac.21492 Siegel RL Miller KD Jemal A . Cancer statistics, 2020. CA: Cancer J Clin (2020) 70(1):7–30. doi: 10.3322/caac.21590 Hoang JK Sosa JA Nguyen XV Galvin PL Oldan JD . Imaging thyroid disease: updates, imaging approach, and management pearls. Radiol Clin North Am (2015) 53(1):145–61. doi: 10.1016/j.rcl.2014.09.002 Lee YH Baek JH Jung SL Kwak JY Kim JH Shin JH . Ultrasound-guided fine needle aspiration of thyroid nodules: a consensus statement by the Korean society of thyroid radiology. Korean J Radiol (2015) 16(2):391–401. doi: 10.3348/kjr.2015.16.2.391 Haugen BR Alexander EK Bible KC Doherty GM Mandel SJ Nikiforov YE . American Thyroid association management guidelines for adult patients with thyroid nodules and differentiated thyroid cancer: the American thyroid association guidelines task force on thyroid nodules and differentiated thyroid cancer. Thyroid (2016) 26(1):1–133. doi: 10.1089/thy.2015.0020 Hoang JK Middleton WD Farjat AE Teefey SA Abinanti N Boschini FJ . Interobserver variability of sonographic features used in the American college of radiology thyroid imaging reporting and data system. AJR Am J roentgenol (2018) 211(1):162–7. doi: 10.2214/ajr.17.19192 Lee HJ Yoon DY Seo YL Kim JH Baek S Lim KJ . Intraobserver and interobserver variability in ultrasound measurements of thyroid nodules. J Ultrasound Med (2018) 37(1):173–8. doi: 10.1002/jum.14316 Zhou LQ Wang JY Yu SY Wu GG Wei Q Deng YB . Artificial intelligence in medical imaging of the liver. World J Gastroenterol (2019) 25(6):672–82. doi: 10.3748/wjg.v25.i6.672 Yang YJ Bang CS . Application of artificial intelligence in gastroenterology. World J Gastroenterol (2019) 25(14):1666–83. doi: 10.3748/wjg.v25.i14.1666 Reid JE Eaton E . Artificial intelligence for pediatric ophthalmology. Curr Opin Ophthalmol (2019) 30(5):337–46. doi: 10.1097/icu.0000000000000593 Schlessinger DI Chhor G Gevaert O Swetter SM Ko J Novoa RA . Artificial intelligence and dermatology: opportunities, challenges, and future directions. Semin cutaneous Med Surg (2019) 38(1):E31–7. doi: 10.12788/j.sder.2019 Bera K Schalper KA Rimm DL Velcheti V Madabhushi A . Artificial intelligence in digital pathology - new tools for diagnosis and precision oncology. Nat Rev Clin Oncol (2019) 16(11):703–15. doi: 10.1038/s41571-019-0252-y Hosny A Parmar C Quackenbush J Schwartz LH Aerts H . Artificial intelligence in radiology. Nat Rev Cancer (2018) 18(8):500–10. doi: 10.1038/s41568-018-0016-5 Wang Y Ge X Ma H Qi S Zhang G Yao Y . Deep learning in medical ultrasound image analysis: a review. IEEE Access (2021) 9:54310–24. doi: 10.1109/access.2021.3071301 Kesharaju M Nagarajah R . Feature selection for neural network based defect classification of ceramic components using high frequency ultrasound. Ultrasonics (2015) 62:271–7. doi: 10.1016/j.ultras.2015.05.027 Tsantis S Cavouras D Kalatzis I Piliouras N Dimitropoulos N Nikiforidis G . Development of a support vector machine-based image analysis system for assessing the thyroid nodule malignancy risk on ultrasound. Ultrasound Med Biol (2005) 31(11):1451–9. doi: 10.1016/j.ultrasmedbio.2005.07.009 Hao X Zhang GG Ma S . Deep learning. Int J Semant Comput (2016) 10(3):417–39. doi: 10.1142/s1793351x16500045 Pehrson LM Lauridsen C Nielsen MB . Machine learning and deep learning applied in ultrasound. Ultraschall der Med (Stuttgart Germany: 1980) (2018) 39(4):379–81. doi: 10.1055/a-0642-9545 Anwar SM Majid M Qayyum A Awais M Alnowami M Khan MK . Medical image analysis using convolutional neural networks: a review. J Med Syst (2018) 42(11):226. doi: 10.1007/s10916-018-1088-1 Lecun Y Bottou L Bengio Y Haffner P . Gradient-based learning applied to document recognition. Proc IEEE (1998) 86(11):2278–324. doi: 10.1109/5.726791 Suzuki K . Overview of deep learning in medical imaging. Radiol Phys Technol (2017) 10(3):257–73. doi: 10.1007/s12194-017-0406-5 Voulodimos A Doulamis N Doulamis A Protopapadakis E . Deep learning for computer vision: a brief review. Comput Intell Neurosci (2018) 2018:7068349. doi: 10.1155/2018/7068349 Gokmen T Onen M Haensch W . Training deep convolutional neural networks with resistive cross-point devices. Front Neurosci (2017) 11:538. doi: 10.3389/fnins.2017.00538 Srivastava N Hinton G Krizhevsky A Sutskever I Salakhutdinov R . Dropout: a simple way to prevent neural networks from overfitting. J Mach Learn Res (2014) 15:1929–58. Ma J Wu F Jiang T Zhu J Kong D . Cascade convolutional neural networks for automatic detection of thyroid nodules in ultrasound images. Med Phys (2017) 44(5):1678–91. doi: 10.1002/mp.12134 Li H Weng J Shi Y Gu W Mao Y Wang Y . An improved deep learning approach for detection of thyroid papillary cancer in ultrasound images. Sci Rep (2018) 8(1):6600. doi: 10.1038/s41598-018-25005-7 Liu T Guo Q Lian C Ren X Liang S Yu J . Automated detection and classification of thyroid nodules in ultrasound images using clinical-Knowledge-Guided convolutional neural networks. Med Image Anal (2019) 58:101555. doi: 10.1016/j.media.2019.101555 Acharya UR Sree SV Krishnan MM Molinari F Zieleźnik W Bardales RH . Computer-aided diagnostic system for detection of hashimoto thyroiditis on ultrasound images from a polish population. J Ultrasound Med (2014) 33(2):245–53. doi: 10.7863/ultra.33.2.245 Nikhil S Narayan PM . Jeevendra kanagalingam, MD and Christopher G.L. Hobbs, MD. speckle patch similarity for echogenicity-based multiorgan segmentation in ultrasound images of the thyroid gland. IEEE J BioMed Health Inform (2017) 21(1):172–83. doi: 10.1109/JBHI.2015.2492476 Chen J You H Li K . A review of thyroid gland segmentation and thyroid nodule segmentation methods for medical ultrasound images. Comput Methods Programs BioMed (2020) 185:105329. doi: 10.1016/j.cmpb.2020.105329 Acharya UR Faust O Sree SV Molinari F Garberoglio R Suri JS . Cost-effective and non-invasive automated benign and malignant thyroid lesion classification in 3d contrast-enhanced ultrasound using combination of wavelets and textures: a class of thyroscan™ algorithms. Technol Cancer Res Treat (2011) 10(4):371–80. doi: 10.7785/tcrt.2012.500214 Zhao J Zheng W Zhang L Tian H . Segmentation of ultrasound images of thyroid nodule for assisting fine needle aspiration cytology. Health Inf Sci Syst (2013) 1:5. doi: 10.1186/2047-2501-1-5 Poudel P Illanes A Sheet D Friebe M . Evaluation of commonly used algorithms for thyroid ultrasound images segmentation and improvement using machine learning approaches. J Healthc Eng (2018) 2018:8087624. doi: 10.1155/2018/8087624 Rother C Kolmogorov V Blake A . “Grabcut” - interactive foreground extraction using iterated graph cuts. ACM Trans Graphics (2004) 23(3):309–14. doi: 10.1145/1015706.1015720 Chang CY Lei YF Tseng CH Shih SR . Thyroid segmentation and volume estimation in ultrasound images. IEEE Trans BioMed Eng (2010) 57(6):1348–57. doi: 10.1109/TBME.2010.2041003 Selvathi D Sharnitha VS . (2011). Thyroid classification and segmentation in ultrasound images using machine learning algorithms, in: Proceedings 2011 International Conference on Signal Processing, Communication, Computing and Networking Technologies (ICSCCN 2011), . pp. 836–41. doi: 10.1109/icsccn.2011.6024666 Cicek O Abdulkadir A Lienkamp SS Brox T Ronneberger O . (2016). 3d U-net: learning dense volumetric segmentation from sparse annotation, in: Medical Image Computing and Computer-Assisted Intervention - MICCAI 2016 19th International Conference Proceedings: LNCS 9901, . pp. 424–32. doi: 10.1007/978-3-319-46723-8_49 Nugroho HA Nugroho A Choridah L Ieee . (2015). Thyroid nodule segmentation using active contour bilateral filtering on ultrasound images, in: 14th International Conference on QiR (Quality in Research), Fac Engn Univ Indonesia/Fac Engn Univ Mataram/STMIK Lombok, Lombok, INDONESIA, 2015 Aug 10-13. Tomasi C Manduchi R Ieee . (1998). Bilateral filtering for Gray and color images, in: 6th International Conference on Computer Vision, Bombay, India, 1998 Jan 04-07. Maroulis DE Savelonas MA Iakovidis DK Karkanis SA Dimitropoulos N . Variable background active contour model for computer-aided delineation of nodules in thyroid ultrasound images. IEEE Trans Inf Technol BioMed (2007) 11(5):537–43. doi: 10.1109/titb.2006.890018 Mylona EA Savelonas MA Maroulis D . Automated adjustment of region-based active contour parameters using local image geometry. IEEE Trans Cybern (2014) 44(12):2757–70. doi: 10.1109/tcyb.2014.2315293 Mylona EA Savelonas MA Maroulis D . Self-parameterized active contours based on regional edge structure for medical image segmentation. SpringerPlus (2014) 3:424. doi: 10.1186/2193-1801-3-424 Ma J Wu F Jiang T Zhao Q Kong D . Ultrasound image-based thyroid nodule automatic segmentation using convolutional neural networks. Int J Comput Assist Radiol Surg (2017) 12(11):1895–910. doi: 10.1007/s11548-017-1649-7 Ying X Yu ZH Yu RG Li XW Yu M Zhao MK . (2018). Thyroid nodule segmentation in ultrasound images based on cascaded convolutional neural network, in: 25th International Conference on Neural Information Processing (ICONIP), Siem Reap, CAMBODIA, 2018 Dec 13-16. Zhou JA Ieee . (2017). Thyroid tumor ultrasound image segmentation based on improved graph cut, in: International Conference on Intelligent Transportation, Big Data & Smart City (ICITBS), Changsha, PEOPLES R CHINA, 2016 Dec 17-18. Savelonas M Maroulis D Sangriotis M . A computer-aided system for malignancy risk assessment of nodules in thyroid us images based on boundary features. Comput Methods Programs BioMed (2009) 96(1):25–32. doi: 10.1016/j.cmpb.2009.04.001 Hirning T Zuna I Schlaps D Lorenz D Meybier H Tschahargane C . Quantification and classification of echographic findings in the thyroid gland by computerized b-mode texture analysis. Eur J Radiol (1989) 9(4):244–7. Tsantis S Dimitropoulos N Cavouras D Nikiforidis G . Morphological and wavelet features towards sonographic thyroid nodules evaluation. Computerized Med Imaging Graphics (2009) 33(2):91–9. doi: 10.1016/j.compmedimag.2008.10.010 Chang C-Y Chen S-J Tsai M-F . Application of support-Vector-Machine-Based method for feature selection and classification of thyroid nodules in ultrasound images. Pattern Recogn (2010) 43(10):3494–506. doi: 10.1016/j.patcog.2010.04.023 Iakovidis DK Keramidas EG Maroulis D . Fusion of fuzzy statistical distributions for classification of thyroid ultrasound patterns. Artif Intell Med (2010) 50(1):33–41. doi: 10.1016/j.artmed.2010.04.004 Legakis I Savelonas MA Maroulis D Iakovidis DK . Computer-based nodule malignancy risk assessment in thyroid ultrasound images. Int J Comput Appl (2011) 33(1). doi: 10.2316/Journal.202.2011.1.202-2955 Luo S Kim EH Dighe M Kim Y . Thyroid nodule classification using ultrasound elastography Via linear discriminant analysis. Ultrasonics (2011) 51(4):425–31. doi: 10.1016/j.ultras.2010.11.008 Ding J Cheng H Ning C Huang J Zhang Y . Quantitative measurement for thyroid cancer characterization based on elastography. J Ultrasound Med (2011) 30(9):1259–66. doi: 10.7863/jum.2011.30.9.1259 Acharya UR Faust O Sree SV Molinari F Suri JS . Thyroscreen system: high resolution ultrasound thyroid image characterization into benign and malignant classes using novel combination of texture and discrete wavelet transform. Comput Methods Programs BioMed (2012) 107(2):233–41. doi: 10.1016/j.cmpb.2011.10.001 Acharya UR VS S Molinari F Garberoglio R Witkowska A Suri JS . Automated benign & malignant thyroid lesion characterization and classification in 3d contrast-enhanced ultrasound. Annu Int Conf IEEE Eng Med Biol Soc (2012) 2012:452–5. doi: 10.1109/embc.2012.6345965 Acharya UR Vinitha S Sree Krishnan MM Molinari F Garberoglio R Suri JS . Non-invasive automated 3d thyroid lesion classification in ultrasound: a class of thyroscan™ systems. Ultrasonics (2012) 52(4):508–20. doi: 10.1016/j.ultras.2011.11.003 Acharya UR Sree SV Swapna G Gupta S Molinari F Garberoglio R . Effect of complex wavelet transform filter on thyroid tumor classification in three-dimensional ultrasound. Proc Inst Mech Eng H (2013) 227(3):284–92. doi: 10.1177/0954411912472422 Zhu LC Ye YL Luo WH Su M Wei HP Zhang XB . A model to discriminate malignant from benign thyroid nodules using artificial neural network. PloS One (2013) 8(12):e82211. doi: 10.1371/journal.pone.0082211 Kim SY Kim EK Moon HJ Yoon JH Kwak JY . Application of texture analysis in the differential diagnosis of benign and malignant thyroid nodules: comparison with Gray -scale ultrasound and elastography. AJR Am J roentgenol (2015) 205(3):W343–51. doi: 10.2214/ajr.14.13825 Song G Xue F Zhang C . A model using texture features to differentiate the nature of thyroid nodules on sonography. J Ultrasound Med (2015) 34(10):1753–60. doi: 10.7863/ultra.15.14.10045 Abbasian Ardakani A Gharbali A Mohammadi A . Application of texture analysis method for classification of benign and malignant thyroid nodules in ultrasound images. Iranian J Cancer Prev (2015) 8(2):116–24. Ardakani AA Gharbali A Mohammadi A . Classification of benign and malignant thyroid nodules using wavelet texture analysis of sonograms. J Ultrasound Med (2015) 34(11):1983–9. doi: 10.7863/ultra.14.09057 Acharya UR Chowriappa P Fujita H Bhat S Dua S Koh JEW . Thyroid lesion classification in 242 patient population using gabor transform features from high resolution ultrasound images. Knowledge-Based Syst (2016) 107:235–45. doi: 10.1016/j.knosys.2016.06.010 Bhatia KS Lam AC Pang SW Wang D Ahuja AT . Feasibility study of texture analysis using ultrasound shear wave elastography to predict malignancy in thyroid nodules. Ultrasound Med Biol (2016) 42(7):1671–80. doi: 10.1016/j.ultrasmedbio.2016.01.013 Chang Y Paul AK Kim N Baek JH Choi YJ Ha EJ . Computer-aided diagnosis for classifying benign versus malignant thyroid nodules based on ultrasound images: a comparison with radiologist-based assessments. Med Phys (2016) 43(1):554. doi: 10.1118/1.4939060 Wu H Deng Z Zhang B Liu Q Chen J . Classifier model based on machine learning algorithms: application to differential diagnosis of suspicious thyroid nodules Via sonography. AJR Am J roentgenol (2016) 207(4):859–64. doi: 10.2214/ajr.15.15813 Raghavendra U Rajendra Acharya U Gudigar A Hong Tan J Fujita H Hagiwara Y . Fusion of spatial Gray level dependency and fractal texture features for the characterization of thyroid lesions. Ultrasonics (2017) 77:110–20. doi: 10.1016/j.ultras.2017.02.003 Yu Q Jiang T Zhou A Zhang L Zhang C Xu P . Computer-aided diagnosis of malignant or benign thyroid nodes based on ultrasound images. Eur Arch oto-rhino-laryngol (2017) 274(7):2891–7. doi: 10.1007/s00405-017-4562-3 Zhang B Tian J Pei S Chen Y He X Dong Y . Machine learning-assisted system for thyroid nodule diagnosis. Thyroid (2019) 29(6):858–67. doi: 10.1089/thy.2018.0380 Ouyang FS Guo BL Ouyang LZ Liu ZW Lin SJ Meng W . Comparison between linear and nonlinear machine-learning algorithms for the classification of thyroid nodules. Eur J Radiol (2019) 113:251–7. doi: 10.1016/j.ejrad.2019.02.029 Shin I Kim YJ Han K Lee E Kim HJ Shin JH . Application of machine learning to ultrasound images to differentiate follicular neoplasms of the thyroid gland. Ultrasonography (2020) 39(3):257–65. doi: 10.14366/usg.19069 Zhao CK Ren TT Yin YF Shi H Wang HX Zhou BY . A comparative analysis of two machine learning-based diagnostic patterns with thyroid imaging reporting and data system for thyroid nodules: diagnostic performance and unnecessary biopsy rate. Thyroid (2021) 31(3):470–81. doi: 10.1089/thy.2020.0305 Vadhiraj VV Simpkin A O’Connell J Singh Ospina N Maraka S O’Keeffe DT . Ultrasound image classification of thyroid nodules using machine learning techniques. Medicina (Kaunas) (2021) 57(6). doi: 10.3390/medicina57060527 Gild ML Chan M Gajera J Lurie B Gandomkar Z Clifton-Bligh RJ . Risk stratification of indeterminate thyroid nodules using ultrasound and machine learning algorithms. Clin Endocrinol (Oxf) (2022) 96(4):646–52. doi: 10.1111/cen.14612 Keutgen XM Li H Memeh K Conn Busch J Williams J Lan L . A machine-learning algorithm for distinguishing malignant from benign indeterminate thyroid nodules using ultrasound radiomic features. J Med Imaging (Bellingham) (2022) 9(3):34501. doi: 10.1117/1.JMI.9.3.034501 Ma J Wu F Zhu J Xu D Kong D . A pre-trained convolutional neural network based method for thyroid nodule diagnosis. Ultrasonics (2017) 73:221–30. doi: 10.1016/j.ultras.2016.09.011 Chi J Walia E Babyn P Wang J Groot G Eramian M . Thyroid nodule classification in ultrasound images by fine-tuning deep convolutional neural network. J Digit Imaging (2017) 30(4):477–86. doi: 10.1007/s10278-017-9997-y Ye Zhu ZF . An image augmentation method using convolutional network for thyroid nodule classification by transfer learning. IEEE Int Conf Comput Commun (2017). doi: 10.1109/CompComm.2017.8322853 Gao L Liu R Jiang Y Song W Wang Y Liu J . Computer-aided system for diagnosing thyroid nodules on ultrasound: a comparison with radiologist-based clinical assessments. Head Neck (2018) 40(4):778–83. doi: 10.1002/hed.25049 Peng S Liu Y Lv W Liu L Zhou Q Yang H . Deep learning-based artificial intelligence model to assist thyroid nodule diagnosis and management: a multicentre diagnostic study. Lancet Digit Health (2021) 3(4):e250–9. doi: 10.1016/S2589-7500(21)00041-8 Zuo D Han L Chen K Li C Hua Z Lin J . [Extraction of calcification in ultrasonic images based on convolution neural network]. Sheng Wu Yi Xue Gong Cheng Xue Za Zhi (2018) 35(5):679–87. doi: 10.7507/1001-5515.201710017 Zhu Y Sang Q Jia S Wang Y Deyer T . Deep neural networks could differentiate Bethesda class iii versus class Iv/V/Vi. Ann Trans Med (2019) 7(11):231. doi: 10.21037/atm.2018.07.03 Buda M Wildman-Tobriner B Hoang JK Thayer D Tessler FN Middleton WD . Management of thyroid nodules seen on us images: deep learning may match performance of radiologists. Radiology (2019) 292(3):695–701. doi: 10.1148/radiol.2019181343 Guan Q Wang Y Du J Qin Y Lu H Xiang J . Deep learning based classification of ultrasound images for thyroid nodules: a Large scale of pilot study. Ann Trans Med (2019) 7(7):137. doi: 10.21037/atm.2019.04.34 Li X Zhang S Zhang Q Wei X Pan Y Zhao J . Diagnosis of thyroid cancer using deep convolutional neural network models applied to sonographic images: a retrospective, multicohort, diagnostic study. Lancet Oncol (2019) 20(2):193–201. doi: 10.1016/s1470-2045(18)30762-9 Nguyen DT Pham TD Batchuluun G Yoon HS Park KR . Artificial intelligence-based thyroid nodule classification using information from spatial and frequency domains. J Clin Med (2019) 8(11). doi: 10.3390/jcm8111976 Song J Chai YJ Masuoka H Park SW Kim SJ Choi JY . Ultrasound image analysis using deep learning algorithm for the diagnosis of thyroid nodules. Medicine (2019) 98(15):e15133. doi: 10.1097/md.0000000000015133 Song W Li S Liu J Qin H Zhang B Zhang S . (2019). Multitask cascade convolution neural networks for automatic thyroid nodule detection and recognition. IEEE J Biomed Health Inform 23(3):1215–24. doi: 10.1109/jbhi.2018.2852718 Kandarpa VSS Rajamani K Sai S . Exploring image classification of thyroid ultrasound images using deep learning. In Proceedings of the International Conference on Ismac in Computational Vision and Bio-Engineering. (2019) p. 1635–41. Wang L Yang S Yang S Zhao C Tian G Gao Y . Automatic thyroid nodule recognition and diagnosis in ultrasound imaging with the Yolov2 neural network. World J Surg Oncol (2019) 17(1):12. doi: 10.1186/s12957-019-1558-z Nguyen DT Kang JK Pham TD Batchuluun G Park KR . Ultrasound image-based diagnosis of malignant thyroid nodule using artificial intelligence. Sensors (Basel Switzerland) (2020) 20(7). doi: 10.3390/s20071822 Wu GG Lv WZ Yin R Xu JW Yan YJ Chen RX . Deep learning based on acr Ti-rads can improve the differential diagnosis of thyroid nodules. Front Oncol (2021) 11:575166. doi: 10.3389/fonc.2021.575166 Zhu J Zhang S Yu R Liu Z Gao H Yue B . An efficient deep convolutional neural network model for visual localization and automatic diagnosis of thyroid nodules on ultrasound images. Quant Imaging Med Surg (2021) 11(4):1368–80. doi: 10.21037/qims-20-538 Liu Z Zhong S Liu Q Xie C Dai Y Peng C . Thyroid nodule recognition using a joint convolutional neural network with information fusion of ultrasound images and radiofrequency data. Eur Radiol (2021) 31(7):5001–11. doi: 10.1007/s00330-020-07585-z Kim YJ Choi Y Hur SJ Park KS Kim HJ Seo M . Deep convolutional neural network for classification of thyroid nodules on ultrasound: comparison of the diagnostic performance with that of radiologists. Eur J Radiol (2022) 152:110335. doi: 10.1016/j.ejrad.2022.110335 Wang L Zhang L Zhu M Qi X Yi Z . Automatic diagnosis for thyroid nodules in ultrasound images by deep neural networks. Med Image Anal (2020) 61:101665. doi: 10.1016/j.media.2020.101665 Choi YJ Baek JH Park HS Shim WH Kim TY Shong YK . A computer-aided diagnosis system using artificial intelligence for the diagnosis and characterization of thyroid nodules on ultrasound: initial clinical assessment. Thyroid (2017) 27(4):546–52. doi: 10.1089/thy.2016.0372 Gitto S Grassi G De Angelis C Monaco CG Sdao S Sardanelli F . A computer-aided diagnosis system for the assessment and characterization of low-to-High suspicion thyroid nodules on ultrasound. Radiol Med (2019) 124(2):118–25. doi: 10.1007/s11547-018-0942-z Kim HL Ha EJ Han M . Real-world performance of computer-aided diagnosis system for thyroid nodules using ultrasonography. Ultrasound Med Biol (2019) 45(10):2672–8. doi: 10.1016/j.ultrasmedbio.2019.05.032 Fresilli D Grani G De Pascali ML Alagna G Tassone E Ramundo V . Computer-aided diagnostic system for thyroid nodule sonographic evaluation outperforms the specificity of less experienced examiners. J ultrasound (2020) 23(2):169–74. doi: 10.1007/s40477-020-00453-y Wei Q Zeng S-E Wang L-P Yan Y-J Wang T Xu J-W . The value of s-detect in improving the diagnostic performance of radiologists for the differential diagnosis of thyroid nodules. Med Ultrasonogr (2020) 22(4). doi: 10.11152/mu-2501 Han M Ha EJ Park JH . Computer-aided diagnostic system for thyroid nodules on ultrasonography: diagnostic performance based on the thyroid imaging reporting and data system classification and dichotomous outcomes. AJNR Am J neuroradiol (2021) 42(3):559–65. doi: 10.3174/ajnr.A6922 Chung SR Baek JH Lee MK Ahn Y Choi YJ Sung TY . Computer-aided diagnosis system for the evaluation of thyroid nodules on ultrasonography: prospective non-inferiority study according to the experience level of radiologists. Korean J Radiol (2020) 21(3):369–76. doi: 10.3348/kjr.2019.0581 Barczynski M Stopa-Barczynska M Wojtczak B Czarniecka A Konturek A . Clinical validation of s-Detect(Tm) mode in semi-automated ultrasound classification of thyroid lesions in surgical office. Gland Surg (2020) 9(Suppl 2):S77–85. doi: 10.21037/gs.2019.12.23 Molnar K Kalman E Hari Z Giyab O Gaspar T Rucz K . False-positive malignant diagnosis of nodule mimicking lesions by computer-aided thyroid nodule analysis in clinical ultrasonography practice. Diagn (Basel) (2020) 10(6). doi: 10.3390/diagnostics10060378 Xia S Yao J Zhou W Dong Y Xu S Zhou J . A computer-aided diagnosing system in the evaluation of thyroid nodules-experience in a specialized thyroid center. World J Surg Oncol (2019) 17(1):210. doi: 10.1186/s12957-019-1752-z Zhao WJ Fu LR Huang ZM Zhu JQ Ma BY . Effectiveness evaluation of computer-aided diagnosis system for the diagnosis of thyroid nodules on ultrasound: a systematic review and meta-analysis. Medicine (2019) 98(32):e16379. doi: 10.1097/md.0000000000016379 Jeong EY Kim HL Ha EJ Park SY Cho YJ Han M . Computer-aided diagnosis system for thyroid nodules on ultrasonography: diagnostic performance and reproducibility based on the experience level of operators. Eur Radiol (2019) 29(4):1978–85. doi: 10.1007/s00330-018-5772-9 Yoo YJ Ha EJ Cho YJ Kim HL Han M Kang SY . Computer-aided diagnosis of thyroid nodules Via ultrasonography: initial clinical experience. Korean J Radiol (2018) 19(4):665–72. doi: 10.3348/kjr.2018.19.4.665 Yu J Deng Y Liu T Zhou J Jia X Xiao T . Lymph node metastasis prediction of papillary thyroid carcinoma based on transfer learning radiomics. Nat Commun (2020) 11(1):4807. doi: 10.1038/s41467-020-18497-3 Ke Z Liu Y Zhang Y Li J Kuang M Peng S . Diagnostic value and lymph node metastasis prediction of a Custom−Made panel (Thyroline) in thyroid cancer. Oncol Rep (2018) 40(2):659–68. doi: 10.3892/or.2018.6493 Yang Y Chen C Chen Z Jiang J Chen Y Jin L . Prediction of central compartment lymph node metastasis in papillary thyroid microcarcinoma. Clin Endocrinol (Oxf) (2014) 81(2):282–8. doi: 10.1111/cen.12417 Wu Q Li Y Wang Y Hu B . Sonographic features of primary tumor as independent predictive factors for lymph node metastasis in papillary thyroid carcinoma. Clin Transl Oncol (2015) 17(10):830–4. doi: 10.1007/s12094-015-1313-z Roh JL Kim JM Park CI . Central lymph node metastasis of unilateral papillary thyroid carcinoma: patterns and factors predictive of nodal metastasis, morbidity, and recurrence. Ann Surg Oncol (2011) 18(8):2245–50. doi: 10.1245/s10434-011-1600-z Nie X Tan Z Ge M Jiang L Wang J Zheng C . Risk factors analyses for lateral lymph node metastases in papillary thyroid carcinomas: a retrospective study of 356 patients. Arch Endocrinol Metab (2016) 60(5):492–9. doi: 10.1590/2359-3997000000218 Liu Z Lei J Liu Y Fan Y Wang X Lu X . Preoperative predictors of lateral neck lymph node metastasis in papillary thyroid microcarcinoma. Medicine (2017) 96(10):e6240. doi: 10.1097/MD.0000000000006240 Jin S Bao W Yang YT Bai T Bai Y . Establishing a prediction model for lateral neck lymph node metastasis in patients with papillary thyroid carcinoma. Sci Rep (2018) 8(1):17355. doi: 10.1038/s41598-018-35551-9 Abbasian Ardakani A Reiazi R Mohammadi A . A clinical decision support system using ultrasound textures and radiologic features to distinguish metastasis from tumor-free cervical lymph nodes in patients with papillary thyroid carcinoma. J Ultrasound Med (2018) 37(11):2527–35. doi: 10.1002/jum.14610 Liu T Zhou S Yu J Guo Y Wang Y Zhou J . Prediction of lymph node metastasis in patients with papillary thyroid carcinoma: a radiomics method based on preoperative ultrasound images. Technol Cancer Res Treat (2019) 18:1533033819831713. doi: 10.1177/1533033819831713 Lee JH Baek JH Kim JH Shim WH Chung SR Choi YJ . Deep learning -based computer-aided diagnosis system for localization and diagnosis of metastatic lymph nodes on ultrasound: a pilot study. Thyroid (2018) 28(10):1332–8. doi: 10.1089/thy.2018.0082 Ramachandram D Taylor GW . Deep multimodal learning: a survey on recent advances and trends. IEEE Signal Process Magazine (2017) 34(6):96–108. doi: 10.1109/msp.2017.2738401 Wu X Li M Cui XW Xu G . Deep multimodal learning for lymph node metastasis prediction of primary thyroid cancer. Phys Med Biol (2022) 67(3). doi: 10.1088/1361-6560/ac4c47 Huang YQ Liang CH He L Tian J Liang CS Chen X . Development and validation of a radiomics nomogram for preoperative prediction of lymph node metastasis in colorectal cancer. J Clin Oncol (2016) 34(18):2157–64. doi: 10.1200/JCO.2015.65.9128 Liu T Ge X Yu J Guo Y Wang Y Wang W . Comparison of the application of b-mode and strain elastography ultrasound in the estimation of lymph node metastasis of papillary thyroid carcinoma based on a radiomics approach. Int J Comput Assist Radiol Surg (2018) 13(10):1617–27. doi: 10.1007/s11548-018-1796-5 Jiang M Li C Tang S Lv W Yi A Wang B . Nomogram based on shear-wave elastography radiomics can improve preoperative cervical lymph node staging for papillary thyroid carcinoma. Thyroid (2020) 30(6):885–97. doi: 10.1089/thy.2019.0780 Park SH Han K . Methodologic guide for evaluating clinical performance and effect of artificial intelligence technology for medical diagnosis and prediction. Radiology (2018) 286(3):800–9. doi: 10.1148/radiol.2017171920 England JR Cheng PM . Artificial intelligence for medical image analysis: a guide for authors and reviewers. AJR Am J roentgenol (2019) 212(3):513–9. doi: 10.2214/ajr.18.20490 Park SH . Artificial intelligence in medicine: beginner’s guide. J Korean Soc Radiol (2018) 78(5). doi: 10.3348/jksr.2018.78.5.301