The primary tool for diagnosing thyroid cancer is ultrasonography (US) (1–5). While US exhibits a high diagnostic accuracy, it is inherently operator-dependent and this necessitates appropriate training of related personnel to maintain the quality of examinations. Traditionally, US training isconducted through textbooks, lectures, or one-on-one education sessions between an educator and trainee. While the latter method has been effective, it also has notable disadvantages, such as putting a significant burden on educators and resources and an inability to guarantee a consistent quality of education (6).

Considerable experience is required to make accurate diagnoses with US, and the skill of examiners is known to correlate with the number of scans they have performed (7, 8). Thus, trainees need sufficient practice before performing examinations on people; not only is foundational knowledge of scan techniques or anatomy required but also preparation for actual “diagnosis” or “decision-making” is required. The diagnostic performance of inexperienced readers is known to improve through one-on-one training or structured training in the radiology department (9–11). Considering the pattern-based diagnosis of thyroid nodules in US, simple training with a large number of image examples combined with answers can be helpful when learning how to differentiate benign and malignant thyroid nodules. In a past study, deep learning software achieved similar diagnostic performance to expert radiologists based on 13,560 images (12), and in another, meaningful improvements in diagnostic performance were also observed in college students who had no previous experience in thyroid US, who went through learning sessions using a large training input of image-pathology sets (13).

With the development and commercialization of artificial intelligence-based computer-assisted diagnosis (AI-CAD) in thyroid imaging, potential improvements have been reported in diagnostic performance, particularly among readers with relatively limited experience (14–16). Thyroid Imaging Reporting and Data System (TI-RADS) is commonly used in the evaluation of thyroid nodules, and one study showed that an AI algorithm trained on TI-RADS characteristics outperformed another trained solely on distinguishing benign from malignant nodules (17). Furthermore, another study reported that an AI-proposed new TI-RADS criteria demonstrated superior specificity compared to the established American College of Radiology (ACR) TI-RADS (18). This underscores the potential of AI to enhance diagnostic protocols by leveraging structured reporting systems like TI-RADS. These advancements in AI-CAD not only support diagnostic precision but also provide crucial feedback during the learning phase, directly assisting beginner radiologists. We hypothesize that AI assistance can further aid beginner radiologists in diagnosing thyroid nodules after they undergo a self-learning process, ensuring more consistent and reliable diagnostic outcomes.

In this study, we investigated the value of self-learning and AI-CAD assistance in inexperienced readers.

This study was approved by the Institutional Review Board of Severance Hospital and informed consent was obtained from all participants (No. 4–2022-1562).

Among 26 participants, 18 readers were radiology residents (1^st and 2^nd year), and the other 8 were 4 fellows in endocrinology and surgery and 4 residents in family medicine (3^rd year). All 26 readers had none to little experience with thyroid US (range 0–10 months). The learning process for the 3,000 sets took an average of 222 minutes, and the test for the 120 sets utilizing AI assistance was completed in an average of 85 minutes ( Table 1 ). There was no statistical difference in the duration of exposure between radiology residents and readers of other specialties ( Table 1 ).

In this study, we investigated the effectiveness of online-based self-learning for diagnosing thyroid cancer in 26 inexperienced readers from six different hospitals from diverse specialties. Furthermore, we examined the impact of AI assistance on their diagnostic performance for thyroid nodules. After training with a set of 3,000 images, both AUC and accuracy improved for all readers on average, and AI assistance further enhanced these metrics.

Previously, a similar method of self-learning was proposed with 13,560 images being learned by six college freshmen (13). The six freshmen also showed improved sensitivity, specificity, accuracy, and AUC. However, it took an average of 30 hours for these freshmen to learn with 13,560 images (13), and viewing 13,560 images at a specific learning location for this amount of time poses considerable challenges in real life. In this study, we provided 3,000 images and all training was executed via an online platform, enabling participants to learn in their personal space at their convenience and record their results subsequently. In our study, we trained individuals with little to no experience in thyroid US but found that those more likely to benefit from training were radiology residents, family medicine residents, endocrinology fellows, and surgery fellows. On average, our participants took a mean of 222 minutes to learn from the 3,000 images, and this training led to increase in accuracy and AUC.

When we performed a subgroup analysis according to the medical department, the benefit of digital self-learning was only significant in radiology residents. Although there was no statistical difference in the recorded duration of exposure in the learning session between the radiology and other department groups, radiology residents are continuously exposed to images and cases through lectures and conferences during their training. This aspect of learning is likely to differentiate them from readers from other medical specialties. For groups less familiar or exposed to US images or radiological diagnostics, self-learning with 3,000 images may simply not be enough to achieve significant increase in diagnostic accuracy. Given the variation in outcomes across different specialties, incorporating detailed explanations for correct or incorrect answers during the self-learning phase could potentially enhance understanding and retention, particularly for those less familiar with ultrasound imaging. This method could mirror more interactive learning approaches found in question banks, which have been shown to improve diagnostic skills by reinforcing learning points through immediate feedback.

After the self-learning process, the final test performance with AI-CAD assistance showed additional increases in sensitivity, AUC, and accuracy. Previous research has well-documented the increased advantage that AI-CAD offers to beginners in US (12, 20–24). AI-CAD appears to supplement self-learning by offering direct assistance on specific cases, rather than just amplifying the learning effect. Unlike digital self-learning, AI-CAD assistance was effective for all readers, regardless of whether they were from the radiology department or others.

Additionally, as K-TIRADS is predominantly used for image interpretation in Korea, we also sought to ascertain whether the self-learning program had an impact on K-TIRADS assessment. Although the overall ICC for K-TIRADS assessment did not improve with self-learning, the ICC of readers from other specialties increased to the ICC of radiology residents. While such categorical assessments are known to have high interobserver variability (25), if we take into consideration that our standard reference group of experienced readers had an ICC of 0.908, we can assume that K-TIRADS assessments by inexperienced readers need further calibration. The challenges of these assessments appear hard to overcome with image-diagnosis set training.

Our study was conducted entirely on an online platform, enabling participants to learn at their own pace and schedule. This approach facilitated the recruitment of participants from hospitals located in diverse regions. One major advantage of online learning is its ability to reduce the burden on instructors, offer flexibility in terms of time and location, and provide consistent education to a broad audience (26). The proliferation of online learning, especially post-COVID, means that learners today have a strong propensity for web- and social media-based curricula (27, 28). However, US education isn’t just about gaining knowledge; it encompasses the development of psychomotor skills, visual perception for image acquisition, interpretation, and integration into medical decision-making (29). While our online self-learning can address some of these aspects, we anticipate it being particularly effective as a preparatory step to enhance diagnostic performance and boost confidence before trainees handle real clinical situations.

Similarly, AI-based diagnostic augmentation has shown comparable trends in improving diagnostic performance across other medical fields such as dermatology, cardiology, and oncology, where it enhances accuracy and aids less experienced practitioners. The success of these applications suggests that the learning methods employed in our study could potentially be adapted to these fields. In line with expanding our understanding of AI’s utility in medical training, further research could involve testing readers of different experience levels, including senior radiology residents, fellows, and junior faculty. Such studies would help ascertain if even more senior readers can benefit from AI, potentially broadening the scope of AI tools in supporting ongoing professional development and decision-making processes across various stages of a medical career.

There are some limitations to our study. First, since our approach was entirely based on online learning and testing, we had limited control over the learning process. Although we restricted the learning period to two weeks, outcomes might differ between participants who studied intensively and those who learned sporadically. Second, we assessed the overall effects on 26 learners from various medical departments, but the standard deviation of performance due to their different specialty backgrounds was substantial, especially for readers from other specialties than radiology. This variability makes it challenging to achieve statistical significance. Third, we evaluated performance based on binary diagnoses, which may seem overly simplistic. Finally, although we provided a set of 3000 cases for the one-time self-learning session, repetitive training might change the results.

In conclusion, In conclusion, our study demonstrated that while AI-CAD assists all inexperienced readers in improving diagnostic performance for thyroid nodules, the effectiveness of self-learning appears more pronounced in radiology residents, likely due to their prior ultrasonography knowledge. Further studies could explore its impact on other non-radiologist groups.

The raw data supporting the conclusions of this article will be made available by the authors, without undue reservation.

The studies involving humans were approved by Ethical Review Board of Severance Hospital. The studies were conducted in accordance with the local legislation and institutional requirements. The participants provided their written informed consent to participate in this study.

SL: Data curation, Methodology, Writing – original draft, Writing – review & editing. HK: Data curation, Methodology, Writing – review & editing. HJ: Data curation, Writing – review & editing. JJ: Data curation, Writing – review & editing. JJ: Data curation, Writing – review & editing. JL: Data curation, Writing – review & editing. HH: Methodology, Visualization, Writing – review & editing. EL: Formal analysis, Software, Writing – review & editing. DK: Formal analysis, Software, Writing – review & editing. JK: Conceptualization, Funding acquisition, Project administration, Supervision, Writing – original draft, Writing – review & editing.