Ovarian carcinoma is the leading cause of mortality from gynecological malignancy in the USA, where approximately more than 13,000 deaths are from ovarian carcinoma in 2023 and the 5-year survival rate is no more than 50% (1). The early diagnosis of ovarian carcinoma is associated with a significantly higher 5­year survival rate, which is increased to >90% for stage 1 (2). Therefore, it is important to accurately differentiate malignant tumors from benign tumors, thereby optimizing patient triaging and reducing unnecessary surgeries without missing cancer. Although several imaging modalities such as MRI and CT play a role in the assessment and management of adnexal lesions, ultrasound (US) is still the first-line preoperative differential diagnosis method for ovarian masses (3, 4).

Several risk stratification systems have been developed to standardize the assessment of adnexal masses with US to improve accuracy and interreader agreement. The International Ovarian Tumor Analysis (IOTA) group proposed terminology and definitions to describe ultrasound features of adnexal lesions in 2008, aiming to provide a standardized tool for differentiating benign and malignant adnexal lesions (5, 6). The IOTA Simple Rules (IOTA SR) includes five descriptions for benignity (benign features) and five for malignancy (malignant features), and adnexal masses are classified as benign, malignant, and inconclusive. Previous studies showed that the IOTA SR has high performance, with a pooled sensitivity of 0.93 and specificity of 0.80 (7). However, the IOTA SR is unable to classify all adnexal masses, leaving as much as 25% inconclusive lesions; when both malignant and benign features were present, or if none of the features were present, the simple rules were inconclusive (6).

In 2014, the IOTA group developed a new scoring system named the Assessment of Different NEoplasias in the adneXa (ADNEX) model, which used three clinical variables and six ultrasound variables to calculate the risk of an adnexal lesion (benign or malignant), distinguishing four types of malignant ovarian tumors: borderline, stage I cancer, stage II–IV cancer, and secondary metastatic cancer (8–10). Additionally, other standardized guidelines or risk stratification systems were proposed such as the Gynecologic Imaging Reporting and Data System (GI-RADS), the Risk of Malignancy Index 4 (RMI4), and the logistic regression model 2 (LR2) (11–13). Nonetheless, many of these standardized models were found inferior to subjective expert assessment (14). In 2018, based on IOTA terms and data sets, the American College of Radiology (ACR) introduced the Ovarian-Adnexal Reporting and Data System (O-RADS) US risk stratification and management (15). With O-RADS US, an adnexal mass is stratified to a 1–5 category (1, physiologic; 2, almost certainly benign; 3, low risk of malignancy; 4, intermediate of malignancy; 5, high risk of malignancy) according to its sonographic features. Since the publication of O-RADS US, a number of studies evaluating this scoring system have been published. Additionally, some of them had performed head-to-head comparisons between O-RADS US with other guidelines. Although several meta-analyses or systematic reviews have summarized the diagnostic accuracy of O-RADS US, a comparison with other guidelines has not been reported systematically. Therefore, in this study, we aimed to systematically compare the performance of O-RADS US with IOTA SR and the ADNEX model.

This meta-analysis and systematic review adhered to the Preferred Reporting Items for Systematic Reviews and Meta-Analysis (PRISMA) statement (16). The primary outcome of this study was the direct comparison between O-RADS and IOTA SR along with the ADNEX model. Furthermore, the overall diagnostic performance of O-RADS for all the included studies was calculated.

In this meta-analysis, we systematically compared three guidelines for the risk stratification of ovarian carcinoma. Based on 12 studies, our findings demonstrated that all three risk stratification systems had high diagnostic performance, with the area under the HSROC of 0.97, 0.95, and 0.95 for O-RADS, the ADNEX model, and IOTA SR. No significant differences were found between O-RADS and the ADNEX model (P = 0.85) as well as IOTA SR (P = 0.15) using the respective eight and seven head-to-head comparison studies. In addition to overall accuracy, we compared the pooled sensitivity and specificity of O-RADS with the ADNEX model and the IOTA SR; however, no significant difference was found between these three guidelines. In the current study, the pooled sensitivity and specificity from 12 studies of O-RADS were 0.95 and 0.82. In two recent meta-analyses evaluating the overall accuracy of O-RADS US, the pooled sensitivity and specificity based on 15 and 10 studies, respectively, were 0.95 and 0.82 and 0.96 and 0.77 at a cutoff value of ≥4, which is comparable with our findings (32, 33). As for IOTA SR and the ADNEX model, the reported pooled sensitivity and specificity from previous meta-analyses or systematic reviews were 0.93 and 0.80 from 5 studies (7) and 0.92 and 0.82 from 10 studies (at the cutoff value of 15%) (34), respectively. In addition to overall diagnostic performance, all three guidelines reported high interreader agreement between radiologists. However, the kappa values were provided only in three studies for IOTA SR and in two studies for the ADNEX model. Therefore, it is unfeasible to perform a meta-analysis and compare the interreader agreement between studies.

The O-RADS US risk stratification and management tool is another effort for the standardization of the risk stratification of adnexal masses, which is modeled on the IOTA rules and based on IOTA data that included 5,905 patients with adnexal masses. Even though IOTA SR had high diagnostic performance, which may result in up to one-quarter of indeterminate lesions, it is suggested that clinicians with less experience need to be assisted by senior clinicians in using diagnostic models to correctly diagnose these lesions (5). In the current study, the reported inconclusive adnexal masses ranged from 5.4% to 24.7%, and nearly half of them were malignant. For unclassified lesions, IOTA SR recommends referring the patient to experts for subjective assessment of US findings, which could provide the most accurate diagnosis. In an earlier study, subjective assessment of adnexal masses using IOTA SR yielded a sensitivity of 89% and a specificity of 80% (5). To address the issue regarding the absence of experienced US examiners, the Simple Rules Risk (SRR) model was developed as a solution, which is a logistic regression model that utilizes TV-US features based on the SR. Its primary objective is to provide an estimated risk of malignancy for any type of adnexal masses, thereby eliminating inconclusive classification. However, because there were only three studies in our meta-analysis that reported the results of SRR, it is unfeasible to pool the data (23, 25, 27).

Compared with IOTA SR, the relatively lower specificity of O-RADS may lead to overtreatment of adnexal masses. However, for the O-RADS indeterminate adnexal masses (categories 3 and 4), the use of O-RADS MRI is suggested for further evaluation of these masses in order to better characterize their nature (32). Compared with the ADNEX model which used three clinical variables and six ultrasound variables, the O-RADS classification only employs ultrasound characteristics to classify ovarian tumors (24, 35). One shortcoming that should be addressed in the present O-RADS is that two variables (bilocular for cystic lesions and shadowing for solid smooth lesions) were not taken into consideration; therefore, it exhibited a higher sensitivity but a lower specificity than the ADNEX model, as reported in various studies (36). Some studies demonstrated that by considering acoustic shadowing as an indication of benign lesions, the overall diagnostic performance was improved significantly, with AUC increased from 0.91 to 0.94 (P = 0.01) (37). These findings suggested that acoustic shadowing is an important US feature for classifying ovarian tumors, especially in solid lesions, and should be included. In the updated O-RADS US v2022, the addition of the descriptors bilocular for cystic lesions and acoustic shadowing for solid smooth lesions, along with the expanded lexicon descriptors for the typical appearance of some classic benign lesions, may be beneficial for reducing overtreatment (38).

Although O-RADS, IOTA SR, and the ADNEX model all demonstrated high diagnostic performance, in clinical practice, subjective assessment of pelvic ultrasound images by clinicians with considerable experience in gynecologic ultrasound has demonstrated a high degree of accuracy in differentiating between benign and malignant pelvic lesions (7, 39). In fact, subjective assessment appears to be the best method to predict the likelihood of a pelvic malignancy (40). However, clinicians with this level of expertise may not be universally available, presenting a challenge to accurate diagnosis and patient management. Transferring the expertise of experienced ultrasound examiners to less experienced ones poses a significant challenge in the field of gynecologic US. While scoring systems and risk calculation models can potentially assist less experienced examiners in characterizing pelvic lesions, there are valid criticisms regarding the complexity of US information required by some ultrasound-based risk calculation models, particularly outside of specialist centers. One of the primary criticisms of these models is their reliance on sophisticated ultrasonic features and measurements that may be challenging to obtain consistently and accurately by less experienced examiners. Moreover, the interpretation of ultrasound findings can be subjective and may vary among examiners, leading to potential discrepancies in risk assessment and diagnostic accuracy. Considering the low incidence but high mortality rate, risk stratification of adnexal masses is a trade-off between sensitivity and specificity, which should take into consideration a number of factors such as risk tolerance for missing cancer and surgery risk (31). Therefore, the physician and the patient have to contemplate the risks and benefits of any procedure and determine the individual cutoff in specific circumstances in which the adnexal mass is evaluated.

The main strength of our study is that we systematically summarized currently available evidence on the comparison between O-RADS US with the IOTA SR and the ADNEX model. However, our study has some limitations that must be taken into consideration. First, all studies included in this meta-analysis had a retrospective study design, which was subjected to a selection bias, emphasizing the need for prospective validation. Second, substantial heterogeneity was observed among the studies, which affected the general applicability of our study. To investigate the heterogeneity, we performed metaregression analysis using several potential covariates. Nevertheless, these analyses only accounted for the partial source of heterogeneity, and a portion remains unexplained. Third, comparisons between O-RADS and the ADNEX model as well as the IOTA SR were based on nine and seven studies, respectively; thus, our conclusions and results should be regarded with caution and future large, prospective studies are needed to compare these different guidelines.

The O-RADS US, the ADNEX model, and IOTA SR showed favorable diagnostic accuracy for risk stratification of adnexal masses, and these three guidelines demonstrated comparable performance. However, O-RADS US yielded a slightly higher sensitivity but a lower specificity than the ADNEX model and IOTA SR.

The original contributions presented in the study are included in the article/Supplementary Material. Further inquiries can be directed to the corresponding author.

JH: Data curation, Resources, Writing – original draft. JW: Formal analysis, Methodology, Validation, Writing – review & editing. WH: Supervision, Writing – review & editing.