This study was in accordance with the Declaration of Helsinki. The Ethics Committee of Tianjin Medical University Cancer Hospital approved this retrospective study (Approval No. bc2021114), and informed consent was waived. All the clinical and biodatas have been anonymized. We enrolled 412 patients with ovarian tumor from Tianjin Medical University Cancer Institute and Hospital (Tianjin, China). All patients were enrolled between March 2013 to December 2016. Patients with mucinous tumor, endometrioid tumor, clear cell cancer, metastatic cancer and the tumor with poor quality ultrasound images were excluded from the study. In total, 265 patients meeting the inclusion criteria were enrolled consecutively in our study. The samples comprised 106 ovarian cystadenomas, 65 borderline tumors, and 108 ovarian malignancies, all of which were pathologically confirmed to be serous. Of the tumors we eventually included, the ultrasound images of 28 tumors were from 14 patients who had bilateral ovarian tumor (7 patients with bilateral borderline serous tumors, seven patients with bilateral ovarian serous cancer). The patient data included age, age at menarche, CA125 level (range: 5.11–5000 IU/L), location of the lesion (unilateral or bilateral), family history of cancer, and ascites. The inclusion criteria were as follows: 1) histological diagnosis of benign, borderline, and malignant ovarian serous tumors; 2) availability of preoperative US images suitable for diagnostic analysis; 3) US scanning performed before neoadjuvant therapy or surgical resection. The exclusion criteria included the following: 1) no US results or the ovarian mass was not completely visible in the image; 2) mucinous, clear cell, endometrioid, metastatic cancer (Figure 1).

In a two-step decision-making approach, two tasks were performed to train and validate the ability to distinguish between benign vs borderline and malignant (task 1) and borderline vs malignant (task 2). A clinical-feature alone model and a combined clinical-radiomic (CCR) model were built using predictive nomograms for each task. During the development of these tasks, we used a fixed 70%/30% training/validation cohort split. A 10-fold cross-validation was done to evaluate the true diagnostic potential of this method.

All ultrasound images were acquired using a Philips iU22/HD11 (California, United States) ultrasound machine with a 5–12 MHz probe and retrieved from the picture archiving and communication systems (PACS) for image segmentation and analysis at our institution (Figure 2A). The boundary of lesions manually segmented using ImageJ (https://imagej.nih.gov/ij/) by a sonographer with more than 8 years of experience. When the boundary was not determined, another experienced sonographer was consulted for a final opinion. The two sonographers were blind to the pathological and clinical information.

Eight hundred and fifty-five radiomics features, including shape, gray-scale histograms, texture, and wavelet features, were extracted automatically from each segmented region of interest using an in-house software written in MATLAB R2018b (MathWorks, Inc., Natick, Massachusetts). Detailed information on the feature extraction algorithms is provided in Supplemental Table S1.

For each task, we followed a three-step procedure to identify the reliable radiomic features. First, the Wilson test was used to identify features highly related to the biomarkers with a significance of less than 0.05 (p < 0.05). Pearson correlation matrices were used to assess the correlation between the features where a correlation coefficient greater than 0.8 was considered redundant. One of two features with a lower p-value was excluded. Next, the minor absolute shrinkage and selection operator (LASSO) regression method was used to select the most useful prognostic combination of features followed by the computation of the radiomics score (Radscore) for each patient through a linear combination of selected features weighted by their respective coefficients.

All images from the validation cohort were in random order subjected to critical evaluation by a senior (LCX, with 8 years of working experience) and a junior sonographer (LJH, with 2 years of working experience) in the ultrasound department, where each of them had carried out over 200 scans of ovarian ultrasound images per year. Both readers were blinded to the clinical information, study design, and background.

Clinical factors, including age, CA125 level, lesion location, family history of cancer, ascites, and Radscore, were evaluated using univariate analysis in the training set. Variables with p < 0.05 in the univariate analysis were included in the multivariate logistic analysis. The clinical and CCR models were built using these clinical variables with or without a Radscore for each task. These models were presented in the form of a nomogram.

In this study, the continuous variables were presented as the mean (± standard deviation), and categorical variables were recorded as numbers and percentages. The chi-square test, Fisher’s exact test, or Wilcoxon sum-rank test were used to identify categorical variables for the univariate analysis. Binary logistic regression analysis was used for multivariate analysis. Based on the factors mentioned above, the multivariate logistic regression model was adopted to establish two nomograms for diagnosing ovarian neoplasms: clinical-feature alone model vs CCR model. The performance of the nomogram was evaluated based on diagnostic accuracy, sensitivity, and specificity of receiver operating characteristic (ROC) curves and calibration curves. The difference in the area under the curve (AUC) between the training and validation datasets was tested using the p-value of integrated discrimination improvement (IDI) and Delong’s (D) test, and the 95% confidence intervals (CI) were calculated.

All statistical analyses were conducted using the R software (version 6.1, R Foundation for Statistical Computing, Vienna, Austria). A two-tailed difference was considered significant at p < 0.05.

The clinical features of patients in the training and validation cohorts for the two tasks were summarized in Tables 1, 2. We observed a significant difference in the CA125 level, lesion location, and ascites between benign and non-benign serous ovarian lesions in the training cohort (Table 1). As shown in Table 2, age, CA125 level, and ascites significantly differed between the borderline and malignant serous ovarian tumors.

In task 1, LASSO was used to evaluate the diagnostic capability of 17 potential informative predictors (Supplementary Figures S1A, C), and the outputs were to Radscore calculation formula (Supplemental Material). We observed that the differences in the Radscore values between the benign and non-benign serous ovarian tumors in the training and validation cohorts were statistically significant (p < 0.001, Supplementary Figures S2A, B). The ROC curve analysis of the radiomics model showed AUCs of 0.907 (95% CI 0.863–0.950) and 0.877 (95% CI, 0.798–0.957) in the training and validation sets, respectively revealed no significant differences (D = 0.633; p = 0.5278). Next, we evaluated the diagnostic capability of the two sonographers to draw our comparative analysis. Figure 2B; Table 3; Supplementary Table S2 showed the diagnostic performance of the junior sonographer, senior sonographer, and radiomics model, respectively. A statistically significant difference between the junior sonographer and the radiomics model (D = 3.611; p < 0.001) was observed. However, there was no statistically significant difference between the performances of the senior sonographer and the radiomics model (D = 1.473; p = 0.141).

In Task 2, 22 potential informative predictors were explored using the LASSO method (Supplementary Figures S1B, D). Differences in the Radscore value between the borderline and malignant serous ovarian tumors in the training and validation cohorts were statistically significant (p < 0.001, Supplementary Figures S2C, D). The ROC curves of the radiomics model showed AUCs of 0.891 (95% CI 0.833–0.950) and 0.839 (95% CI 0.725–0.952) in the training and validation cohorts, respectively, with no significant difference between them (D = 0.607; p = 0.546). Figure 2C; Table 3; Supplementary Table S2 showed the diagnostic performance of the junior sonographer, senior sonographer, and radiomics model, respectively. There was a statistically significant difference between the performances of the junior/senior sonographer and the radiomics model (senior: D = 3.5, p < 0.001; junior: D = -4.640, p < 0.001).

By comparing the results from the 10-fold cross-validation run of the models built above to the results obtained on the fixed training/validation split, we found that the performance estimates were comparable for both the tasks with no indication of substantial overfitting (Supplementary Table S3).

Next, we utilized the features mentioned above for each task to perform multivariate logistic regression analysis to construct the two models for diagnosing ovarian neoplasms, thereby leading to the generation of two nomograms, the clinical-feature alone model (Figure 3D and Figure 4D) and the combined clinical-radiomic (CCR) model (Figure 3A and Figure 4A).

For task 1, Figure 3, Figure 5 and Table 3 showed the calibration curve and performance of the clinical-alone and CCR models. The ROC curves of the clinical-alone model showed AUCs of 0.817 (95% CI 0.765–0.868) and 0.855 (95% CI 0.786–0.924) in the training and validation cohorts, respectively (Figures 5A,C), with no significant difference between them (D = -0.88079; p = 0.3796). The ROC curves of the CCR model showed AUCs of 0.937 (95% CI 0.905–0.969) and 0.914 (95% CI 0.851–0.976) in the training and validation cohorts, respectively (Figures 5A,C), with no significant difference between them (D = 0.6394; p = 0.524). The calibration curve indicating the prediction from the two models (solid line) closely followed the 45-degree line in the training and validation cohorts, suggesting good diagnostic accuracy (Figures 3B,C for the CCR model and Figures 3E,F for the clinical alone model).

For task 2, the CCR performed satisfactorily in the training (AUC 0.924 [95% CI 0.876–0.971]) and the validation (AUC 0.890 [95% CI 0.794–0.987]) cohorts, respectively (Figures 5B,D), with no significant difference between them (D = 0.607; p = 0.546). The ROC curves of the clinical-feature alone model showed AUCs of 0.815 (95% CI 0.740–0.890) and 0.829 (95% CI 0.706–0.950) in the training and validation cohorts, respectively, with no significant difference between them (D = −0.189, p = 0.85). The calibration curve suggested good diagnostic accuracy for the CCR model (Figures 4B,C), which was slightly worse for the clinical-feature alone model (Figures 4E,F).

As shown in Table 3 and DCA curves (Figure 6), the CCR model showed a relatively better predictive performance than the clinical-feature alone model for two tasks (task 1: IDI = 0.154, 95% CI: 0.078–0.231, p < 0.001; task 2: IDI = 0.815, 95% CI: 0.066–0.303, p = 0.002). The decision curves indicated that using the clinical features combined radiomics nomogram to predict types of serous ovarian cancer adds more benefit than the clinical-feature alone model.