This retrospective study was approved by the institutional review board of our institution, and the written informed consent was waived. The enrolment process of patients for this study is shown in Figure 2. In total, 106 patients who underwent contrast-enhanced CT scannings for clinically and pathologically diagnosed CPA (n = 50) or APA (n = 56) from January 2014 to November 2018 were retrospectively reviewed. The diagnosis of APA and CPA was established by these criteria: (i) common clinical characteristics and laboratory findings including an elevated aldosterone/renin ratio together with positive confirmatory tests in APA and an elevated serum cortisol, failure to suppress cortisol with dexamethasone, and normal aldosterone levels in CPA, respectively; (ii) presence of an adrenal mass confirmed via CT before surgery; (iii) a confirmed pathological diagnosis of the adrenal mass as an adrenal adenoma after surgery; and (iv) a postoperative cure or considerable improvement. The exclusion criteria were as follows: (i) insufficient clinical data; (ii) receiving treatment before surgery; (iii) calcified lesions in tumors; and (iv) motion artifacts disturbed the lesion characterization severely. Finally, 9 patients with CPA and 14 patients with APA were excluded. The dataset was divided into two portions called training set and testing set, 70% of which were used as training set, and the remaining 30% were used as test set.

All CT imaging was acquired using the same multidetector CT system (Somatom Sensation 64; Siemens Healthcare, Erlangen, Germany) following a standardized protocol. A three-phase scanning was performed on each patient (plain scan, arterial phase, and venous phase). The CT scanning parameters were as follows: tube voltage of 100 kV; tube current of 75 mAs, and slice thickness of 5 mm. Images were reconstructed using a B60f filter with a slice thickness of 1 mm and a slice increment of 1 mm as axial images. Contrast-enhanced CT images were obtained after intravenous administration of iohexol (300 mg/mL at a rate of 3.0 mL/s, followed by a 30-mL saline flush). Arterial phase imaging and portal phase imaging were initiated at 30 and 70 s after the injection of iohexol. The total contrast volume was 1.5 mL/kg.

All CT images (DICOM format) were loaded into a computer workstation for region of interest (ROI) segmentation, which were displayed with the appropriate window level and window width. Two radiologists with 5-year experience in interpreting CT imaging (Dr. Liu and Zhong) were recruited to manually delineate the two-dimensional (2D) ROI around the tumor outline slice by slice to form 3D volume of interest (VOI) on the CT plain scan, arterial phase, and venous phase images using an open-source image processing platform ITK-SNAP (version 3.7) (12), where horizontal, coronal, and sagittal views were represented simultaneously for visualization. Image magnification and 3D view techniques have been used to facilitate precise segmentation (Figure 3).

Conventional imaging analysis was included the following information: (a) tumor size; (b) mean CT attenuation of tumor in precontrast, arterial, and portal venous phase; and (c) the presence of ipsilateral or contralateral adrenocortical atrophy. The mean CT attenuation was used to describe average value of tumor density over the CT pixels and automatically obtained by drawing ROI around tumor contour on workstation. The presence of adrenocortical atrophy was defined as the maximum thickness of a unilateral adrenal gland more than a 50% reduction compared to the other side.

The radiomics features were extracted from each VOI segmentation derived from multiparameter CT images, which are divided into four feature groups: (I) intensity, (II) shape, (III) texture, and (IV) wavelet features (11, 13). Supplementary Tables 1–3 summarize these features in this study. Mathematical definitions of all radiomics features, as well as the extraction methods, have been described (14). The texture features were computed by averaging their values over all 13 directions. Wavelet features are the transformed domain representations of the intensity and textural features, which were computed on different wavelet decompositions of the original image using a Daubechies wavelet transformation. Finally, the combination of four categories of features derived from multiphase CT images was incorporated into the radiomics feature set.

Radiomics feature reproducibility was evaluated prior to feature selection by computing the intraclass correlation coefficient (ICC). Each radiomics feature with ICC more than 0.8 was set to consider robust to acquisition variation (15) (see Supplementary), which were retained based on the hypothesis that non-robust features would be too sensitive to noise to be predictive of clinical outcomes. Feature selection as an important problem for pattern classification has become an apparent need in the radiomics application. To find optimal characterization condition and achieve minimal classification error in machine learning, feature relevancy needs to be eliminated. The extracted radiomics features were selected using principal component analysis (PCA), ReliefF algorithm, least absolute shrinkage and selection operator (LASSO), recursive feature elimination, and mutual information. We chose these methods mainly because of their popularity, simplicity, and computational efficiency. All features have been normalized to zero mean and unit variance so as to avoid being affected by the differences in respective feature scales for classification model building. Furthermore, publicly available implementations were readily available for these methods, which increases their reusability.

For the model development, four different algorithms such as linear discriminant analysis, logistic regression (LR), random forest, and support vector machine (SVM) have been adopted. Each classifier has been tested and verified using the feature sets obtained by the different feature selection methods to construct the stable and optimal machine learning model. In the training set, efficient data partitioning such as 5-fold cross-validation was employed to tune and optimize the model parameter to achieve good assessment of the model performance (16). The area under the receiver operating characteristic (ROC) curve (AUC), sensitivity, specificity, and accuracy were used as metrics to assess the performance of the machine learning models. All classifier algorithms were implemented by our in-house scripts in MATLAB (version 2017b, MathWorks, Natick, MA, USA).

The nomogram was used to represent the integrated model for distinguishing APA from CPA. The radiomics signature was constructed by the selected features sorted by their coefficient values in LASSO. Then, the nomogram based on the multivariate logistic analysis was developed by using the combination of radiomics signature and clinic–radiological features as a quantitative diagnostic tool to provide physicians with an individual prediction probability of APA. Calibration curves accompanied by the Hosmer–Lemeshow test were used to assess the model performance. AUC, accuracy, PPV, and NPV were calculated to quantify the diagnostic performance of nomogram. The 1,000-bootstrap repetitions were carried out for internal validation to achieve a relatively corrected performance where the training cohort was randomly chosen with a replacement from the original dataset.

Continuous variables, expressed as mean value ± standard deviation or median with interquartile range as appropriate, were analyzed using Student t-test or Mann-Whitney U-test, respectively. Categorical/dichotomous variables, expressed as counts (percentage), were analyzed using a χ²-test or Fisher exact test as appropriate. Multiple and pairwise comparisons of AUCs were accomplished using the DeLong non-parametric approach. Univariate and multivariate logistic regression models were employed to select the independent clinical features and construct clinic–radiological model. Statistical analysis was performed with R version 3.6.1 (http://www.r-project.org). A two-sided p < 0.05 was considered to represent statistically significant.

The demographic data and radiological characteristics between APA and CPA are presented in Table 1. Sex ratio and age distribution did not differ significantly between these two groups (p > 0.05). In conventional CT findings analysis, there were significant differences between CPA and APA groups in tumor size, mean CT attenuation value of precontrast phase and portal venous phase, and the presence of adrenocortical atrophy (p < 0.05). In the APA group, the tumor showed smaller size and lower mean CT attenuation compared to CPA group, while the ipsilateral or contralateral adrenocortical atrophy was more commonly seen in CPA group (Table 2).

In total, 58 patients including 29 patients with CPA and 29 patients with APA comprised the training cohort, and 25 patients including 12 patients with CPA and 13 patients with APA comprised the test cohort. The proportions of training cohort and the test cohort were 70 and 30%, respectively, and no significant differences of clinical characteristics or CT findings were found between the training and test cohorts (p > 0.05). Table 2 showed the significant differences between CAP and APA in the training and test cohorts.

The longest diameters and the ratio of longest diameter to shortest diameter of raw ROIs between two groups were distributed without statistical significance (p > 0.05), each of which was selected from the slice demonstrating the largest cross-sectional area on CT images. ICCs on the basis of radiologist I's first-extracted features and those of radiologist II were employed to evaluate the consistency between different physicians. The ICCs on the basis of radiologist I's first and second feature extraction were calculated to evaluate the stability and reproducibility of each feature. According to the criterion of excluding the radiomics features with ICC below 0.8, a total of 39 radiomics features were considered as robust shown in Supplementary Table 4.

Feature selection determines the minimum set of relevant indicators needed by a machine learning model. The above robust radiomics features are further screened by retaining those that differed significantly between the two groups. Twenty classification strategies using combinations of four machine learning and five feature selection methods, respectively, have been tested, and the AUCs for differential diagnosis between CPA and APA in the test dataset are shown in Figure 4. For the combination of multiple sequences, it is shown that the LR combined with LASSO performs better and more stable than the other models, which yielded a sensitivity of 0.935, a specificity of 0.823, and an accuracy of 0.887 [AUC = 0.882, 95% confidence interval (CI) = 0.819, 0.945], followed by SVM classifier with ReliefF, yielding an accuracy of 0.842 (AUC = 0.854, 95% CI = 0.811–0.897) in the test cohort.

The discriminative performance of the LR-LASSO models using the radiomics features from multiple sequences and single sequence was investigated. The ROC curves for all single sequences such as CT plain, arterial phase, and venous phase scanning are shown in Figure 5A, and the ROC curves for the combination of multiple sequences are shown in Figure 5B. For single sequence, the performance of plain and venous phase scanning is similar, and the highest AUC was 0.834 (95% CI = 0.779–0.889). For two sequences, the performance of the combination of plain and venous phase scanning was highest with an AUC of 0.876 (95% CI = 0.808–0.944). For three sequences, the model performed best and yielded the highest AUC of 0.882 (95% CI = 0.819–0.945). The AUCs among the three single sequences were not statistically significant, while the DeLong test showed that the AUCs for the different combinations of the multiple sequences were significantly better than those of single sequences, and the AUC for the combinations of the three sequences was the highest.

The above results revealed that the multiparametric CT radiomics-based LR-LASSO model would be most suitable to effectively differentiate CPA from APA. The clinic–radiological characteristics such as age, gender, tumor size, and CT value were determined to establish the clinical model. The combined model that incorporated radiomics signature and clinic–radiological characteristics was developed and presented as a radiomics nomogram (Figure 6A). The clinical model yielded an AUC of 0.829 (95% CI, 0.796–0.863) in the training cohort and 0.732 (95% CI, 0.671–0.793) in the test cohort. When clinic–radiological characteristics were combined, the radiomics nomogram yielded an AUC of 0.931 (95% CI = 0.869–0.993) in the training cohort and 0.902 (95% CI, 0.822–0.982) in the test cohort. Table 3, Figure 7 presented the detailed discrimination indicators of the three models. The calibration curves of the radiomics nomogram for differential diagnosis between CPA and APA showed good agreement between the model outcome and gold standard test in the training (Figure 6B) and test (Figure 6C) cohorts (p = 0.849 and 0.814, respectively; Hosmer-Lemeshow test). The net reclassification improvement (NRI) test showed the integrated model achieved considerably better discrimination ability than the clinic–radiological model (p = 0.012) and radiomics model (p = 0.012) in the training cohort. The performance of the integrated model was comparable to that of the radiomics model (p = 0.989; NRI test), but was superior to that of clinic–radiological model (p = 0.001; NRI test). The illustration in the supplementary material presented two cases pathologically diagnosed as CPA and APA, respectively, and the probability values predicted by the nomogram.