This retrospective pilot study was approved by our Institutional Review Board with a waiver of informed consent. From January 2017 to December 2021, we retrospectively retrieved all MRI pelvic examinations referring to ovarian or adnexal lesions from our Institutional Picture Archiving and Communication System (PACS) and obtained 1867 results. Every report was analyzed by the researchers. Then we identified patients who met the following inclusion criteria (1): pathologically confirmed as SCSTs or EOCs; (2) MRI was performed within 1 month before pelvic or laparoscopic surgery at our institution; (3) no chemotherapy, radiotherapy, or previous gynecological operation prior to MRI examination. The exclusion criteria were: (1) poor MR image quality, so that the focus cannot be clearly observed or delineated; (2) multiple nodules that the primary focus cannot be identified due to a mutual fusion and extensive adhesion pattern or any other reason. Finally, we consecutively reviewed a total of 148 patients with 173 tumors (81 SCSTs in 73 patients and 92 EOCs in 75 patients) as the primary cohort. Details of histopathology are presented in Table 1. Then the primary dataset was randomly split into the development and testing cohorts with a fixed ratio of 8:2 in each category, resulting in 137 tumors (64 SCSTs and 73 EOCs) for the development cohort and 36 tumors (17 SCSTs and 19 EOCs) for testing cohort. A flowchart of the patient selection process is shown in Figure 1.

Clinical characteristics such as patient age, menstrual status, endocrine level, cancer antigen 125 (CA125), and risk of ovarian malignancy algorithm (ROMA), were obtained from the hospital information system.

MR examinations were performed on the 3.0 T system (SIGNA Pioneer, GE Healthcare, and Skyra, Siemens Healthcare) with the phased-array abdominal coil. The conventional MR sequences included T1 weighted imaging (T1WI) in the axial plane, T2 weighted imaging (T2WI) in the axial and sagittal plane, fat-suppressed T2WI (FS-T2WI) in the axial and coronal plane, diffusion-weighted imaging (DWI) with the b value of 1000 s/mm² in the axial plane, and multiphase contrast-enhanced fat-suppressed T1WI in the axial, sagittal, and coronal plane. Detailed information about the acquisition parameters is presented in Table S1.

Two radiologists (reader 1 and reader 2, with 7 and 15 years of experience in MRI, respectively) independently recorded the conventional imaging features while blinded to the histological results. The recorded features include (1) Maximum diameter (MD) (measured at the lesion slice with the maximum diameter of the tumor in the three-dimensional measurements); (2) Visibility of hemorrhagic component; (3) Solid and cystic components (predominantly cystic, cystic-solid, and predominantly solid corresponding to less than 1/3, 1/3-2/3, and more than 2/3 solid component, respectively); (4) Signal intensity (SI) of the solid components on T2WI (Hypo-, iso-, or hyperintense was relative to the external myometrium; A few purely cystic lesions were not recorded); (5) Apparent diffusion coefficient (ADC) value (measured manually on the DWI derived ADC maps; regions of interest were placed at target areas of the tumor, and areas such as necrosis, hemorrhage, vascular structures were avoided as much as possible; three measurements were obtained and averaged). Discrepancies were resolved by a consensus, or a third radiologist (reader 3 with more than 20 years of experience in gynecologic imaging) would serve as an arbitrator.

In order to reduce the discrepancies related to various scanning parameters and eliminate the internal dependence of image radiomics features on voxel size, all images were set to a fixed voxel size (1mm×1mm×1mm) with linear interpolation algorithm.

The 3D segmentation was performed on open-source software (ITK-SNAP version 3.8.0, http://www.itksnap.org). The volume of interest (VOI) for each lesion on each slice was manually delineated on FS-T2WI by a reader 1. Then, reader 3 confirmed all the regions. To examine the reproducibility of extracted radiomics features and to obtain more robust radiomics features, the intra-class correlation coefficient (ICC) was used to assess the intra- and inter-observer agreement of VOI delineation. So, four weeks later, 30 patients (15 SCSTs and 15 EOCs) were randomly selected. Reader 1 then re-delineated the VOI for intra-consistency testing, in the meantime, reader 2 outlined the VOI according to the same procedure to test inter-consistency. The feature with an ICC > 0.75 was selected for further analysis (12). ICC can be obtained from the following equation:

where MS_R is mean square for rows, MS_C is the mean square for columns, MS_E is mean square for error and n represents the number of subjects.

The extraction of radiomics features was conducted in the Radcloud software (Huiying Medical Technology Co., Ltd, Beijing, China). A total of 1,409 features were extracted from the FS-T2WI sequences using the pyradiomics function package (https://pyradiomics.readthedocs.io/). These features could be divided into the following three categories. First, first-order statistical features, such as peak value, mean, and variance, quantitatively describe the voxel intensity distribution of the lesion area in MR images through common basic indicators. Second, two-dimensional morphological features, describe the two-dimensional shape and size of the lesion. Third, texture features, such as Gray Level Co-occurrence Matrix (GLCM), Gray Level Run Length Matrix (GLCM), and Gray Level Size Zone Matrix (GLSZM), quantify the heterogeneity of the lesion texture. In addition, several filters, such as exponential, logarithm, square, square root, and wavelet (including wavelet-LHL, wavelet-LHH, wavelet-HLL, wavelet-LLH, wavelet-HLH, wavelet-HHH, wavelet-HHL, and wavelet-LLL) filters, were applied to calculate the first-order statistical features and texture features of the transformed image.

Before the selection of radiomics features, normalization processing was performed for all extracted features, and the features were normalized to the normal distribution by mean and variance scaling.

We run a 5-fold cross validation to select features, where each fold we did feature selection and model building on 80% of the development data (the training cohort), and evaluated on the remaining 20% (the validation cohort). In each fold, we implement the dimensionality reduction process in three steps. Features with a variance value of >0.8 were first selected. Then SelectKBest was applied to select the features with a p-value less than 0.05. Finally, the Lassolars algorithm was used to screen the optimal radiomics features, among which the top 10 features with the best contribution would be selected. The least absolute shrinkage and selection operator (LASSO) is a regression analysis method that can perform both variable selection and regularization to improve the identification accuracy and interpretability of the model. Regression algorithm least angle regression (LARS) provides variables through the linear combination of high-dimensional data. It is related to positive stepwise regression. Lassolars algorithm used in this paper is a combination of Lars algorithm and lasso model, which can automatically select the optimal parameters λ and have better performance than LASSO algorithm alone (13). Finally, we counted the frequency of each feature selected in the 5 folds, the features that get selected three or more times repeatedly were considered stable features, and different models were built using various combinations of those selected features.

Univariate analysis and multivariate logistic regression analysis were successively performed based on development data to screen the clinical and conventional MR features. Based on the selected radiomics features, clinical features, and conventional MR parameters, five prediction models (clinical model, conventional MR model, traditional model, radiomics model and mixed model) were constructed by using the LR classifier. LR classifier includes classification, function establishment, solving optimal model parameters through optimization iteration, and verifying the model performance. In addition, the nomogram was also constructed to visualize the results of the logistic regression. The nomogram develops scoring criteria based on the magnitude of the regression coefficients for all independent variables. It scores each value level of each independent variable, giving an overall score and finally calculating the probability of disease risk for each patient through a conversion function between the score and the outcome probability. The calibration curve was drawn to assess the agreement between the predicted results and actual presence. The detailed process of radiomics analysis is presented in Figure 2.

The performance of the model was estimated by using receiver operating characteristic (ROC) curves and confusion matrix analysis, and the area under the curve (AUC), sensitivity, specificity, and accuracy were calculated. The DeLong test was used to compare the performance of different models. The decision curve analysis (DCA) was used to evaluate the clinical utility of the prediction models by quantifying the net benefits at different threshold probabilities in the dataset.

Statistical analysis was performed using SPSS version 25.0 (IBM). Quantitative variables are shown as mean ± standard deviation. Categorical variables were assessed by Chi-square tests or Fisher exact test, and differences in continuous variables were assessed by t-test or Mann-Whitney U test. P< 0.05 was considered statistically significant. Model construction was executed using R software 3.5.3 (https://cran.r-project.org/) and Python 2.7 software (https://www.python.org/). The packages of “pyradiomics” (https://pyradiomics.readthedocs.io/), “scikitlearn” (https://scikit-learn.org/), and “matplotlib” (https://matplotlib.org/) were used for feature selection, model building, and plotting in this study.

The comparisons of the clinical data and MR parameters between SCSTs and EOCs groups in the primary and development cohorts are summarized in Table 2. Age, ROMA index, serum CA125, ADC value, MD, SI on T2WI, solid and cystic components showed significant differences between the SCSTs and EOCs groups, while no significant difference was observed in menstrual status, endocrine level, and hemorrhagic component. Among the variables with significant differences, only ROMA index (P< 0.001), ADC value (P = 0.004), solid and cystic components (P = 0.043) remained as independent predictors on the multivariate logistic regression analysis.

Of all the extracted radiomics features, the median ICC was 0.90, and 1277 of 1409 features (91%) were robust and were selected for subsequent analysis, with ICC > 0.75. Lassolars algorithms on feature selection for each fold are shown in Figure S1. The selected radiomic features and the corresponding coefficients in each fold are shown in Figure 3 and Table S2. We did model building in every fold, the AUCs for the 5-fold cross validation are reported in Table S3 and Figure S2, and the mean AUCs were 0.883 ± 0.018 in the training cohort and 0.849 ± 0.021 in the validation cohort. Finally, there were three features got selected in 3 of the 5 folds, three features got selected in 4 of the 5 folds, and three features got selected in every fold. The frequencies of the radiomics features are summarized in Table 3.

Based on the selected clinical variable ROMA index, a clinical model was established. Based on parameter ADC, solid and cystic components, a conventional MR model was established. Then, based on the combination of the above clinical factors and conventional MR parameters, a traditional model was established. The AUCs of the clinical model, conventional MR model, and traditional model were 0.729 (95%CI: 0.645-0.812), 0.737 (95%CI: 0.654-0.819), 0.755 (95%CI: 0.677-0.834) in the development cohort, respectively, and 0.680 (95%CI: 0.498-0.862), 0.693 (95%CI: 0.515-0.872), 0.735 (95%CI: 0.569-0.902) in the testing cohort, respectively (Figure 4 and Table 4).

Three different radiomics models (Models 1-3) were developed using the following combinations of the radiomics features: 3 features (got selected in every fold), 6 features (got selected 4 or more times repeatedly) and 9 features (got selected 3 or more times repeatedly). Among them, Model 3 showed the best performance, which was determined as the final radiomics model, with the AUCs of 0.915 (95% CI: 0.869-0.962) and 0.867 (95% CI: 0.732-1.000) in the development and testing cohorts, respectively (Figure 4 and Table 4). However, no significant differences were observed between Model 3 and the other two models. The AUCs of the three models and the comparisons in terms of diagnostic performance among them were shown in Table 5. Rad-score based on 9 features were weighted by their respective coefficients, the calculation formula for the Rad-score is provided in the Supplementary Materials page 7.

Finally, we established a mixed model based on the Rad-score, clinical characteristics (ROMA), and conventional MR parameters (ADC, solid and cystic components). The AUCs of the mixed model were 0.934 (95% CI: 0.892-0.976) and 0.875 (95% CI: 0.743-1.000) in the development and testing cohorts, respectively (Figure 4 and Table 4).

The difference between the traditional model and radiomics model was statistically significant in the testing cohort (traditional model AUC vs. radiomics model AUC: 0.735 vs. 0.867, Delong test P< 0.001); however, there was no evidence of a difference in the testing set between the radiomics model compared with the mixed model (radiomics model AUC vs. mixed model AUC: 0.867 vs. 0.875, Delong test P = 0.561).

A radiomics nomogram was constructed by using the selected variables from multivariate logistic regression and Rad-score to provide a visualized outcome measure (Figure 5A). The total score for this nomogram was calculated using the formula: Nomo-score = -1.5402 + 11.4118 × Rad-score -0.0335 × ROMA + 1.3928 × ADC + 0.3532 × Components. The calibration curves demonstrated good diagnostic consistency between the predictions of the radiomics nomogram and the actual observations of the samples (Figure 5B).

DCA revealed that the radiomics model and the mixed model provided a better net benefit than the traditional model across the majority of the range of reasonable threshold probabilities (Figure 6).