PubMed, Embase and Web of Science Core Collection were queried to identify relevant studies published until July 17, 2023, utilizing terms such as radiomics, lung adenocarcinoma, lung squamous cell cancer with Boolean logic operation. The search details were listed in Supplementary Table S2 . The reference lists of included studies and relevant reviews were manually examined to identify any potential studies that may have been overlooked. There was no language limit.

We included studies that met the following criteria: (1) radiomics-based model for LUAD and LUSC classification; (2) radiomics features extracted from pre-treatment lung imaging irrespective of the modality of imaging; (3) patients were pathologically confirmed as LUAD or LUSC.

We excluded (1) articles that were not original full-text, such as reviews, letters, or commentaries, as well as conference abstract; (2) studies that did not provide information on LUAD and LUSC classification; (3) studies that used features other than radiomics to differentiate between LUAD and LUSC; (4) studies that used a sample that had already been used in another study; (5) studies that lacked sufficient information to assess their quality.

Two reviewers (LS, with 6 years of experience in chest image analysis, AND JZ, a broad-certified radiologist with 19 years of experience) individually examined tiles, abstracts, and assessed the full texts to determine eligibility. The disagreements were resolved by consensus.

Basic information of each study including the surname of the first author and the publication year were extracted. The study’s characteristics such as sample size and study design were provided. The index test information consisted of the imaging modality used, segmentation method, software used for segmentation and radiomics feature extraction, the number of radiomics features extracted, feature selection method, classification model, and any non-radiomics features included in the model. Furthermore, the outcomes encompassed true positive, false positive, false negative and true negative, along with any other statistical data that might be used for calculation. If multiple classification objectives were presented in a study, only the information differentiating LUAD from LUSC based on radiomics features was extracted. If a study where multiple radiomics models were mentioned, the model with the highest area under the curve (AUC) was selected.

Data extraction was implemented independently by two reviewers (LS, ZW with 4 years of experience in chest image analysis). The disagreements were resolved by discussion.

The quality of the included studies was evaluated using the Improved Quality Assessment of Diagnostic Accuracy Studies (QUADAS-2) and Radiomics Quality Score (RQS). The purpose of QUADAS-2 is to evaluate the quality of primary diagnostic studies, which includes 4 key domains: patient selection, index test, reference standard, and flow and timing (15). The results of QUADAS-2 were recorded using Revman 5.4. The signaling questions of the 4 key domains were modified to tailor to our study (16). If all signaling questions of a domain were answered “yes”, the domain was considered at low risk of bias.

RQS, consisting of 16 items, was suggested as a means to enhance the radiomics workflow and has been extensively employed in evaluating the methodological quality of radiomics studies in systematic review (9). After evaluating the studies based on each item, a total score will be calculated for each study and displayed on a scale ranging from -8 to 36, which can be converted into a percentage. Scores below 0 will be considered as 0, while a score of 36 will be equivalent to 100%.

The quality of the included studies was evaluated by two separate reviewers (LS and JZ). Quality discrepancies were resolved through reassessment and discussion.

The agreement between two reviewers on each item of RQS and each signaling question of QUADAS-2 was by expressed by a modified Fleiss kappa statistic (17). The inter-rater agreement of total RQS was measured using the interclass correlation coefficient (ICC) (18). R (version 4.2.2) was utilized for the computation. A significance level of less than 0.05 was deemed statistically significant.

Studies were pooled to estimate sensitivity and specificity along with 95% confidence intervals (CIs) using random-effect model. To reflect the synthesized diagnostic accuracy, the summary receiver operating characteristic (SROC) curve was constructed and the AUC was calculated. The chi-squared test was utilized to analyze the statistical heterogeneity among studies and the results were presented as the I ² statistic. Significant heterogeneity was observed when P<0.1 and I ²>50%. Subgroup analysis was performed to detect the cause of heterogeneity. To evaluate the model’s stability, sensitivity analysis was conducted by plotting measures of goodness-of-fit, bivariate normality, influence analysis and outlier detection. An investigation of publication bias was conducted using Deeks’ funnel plot, and a P-value was computed using Deeks’ asymmetry test. Two-tailed P<0.05 was considered statistically significant.

The initial search and duplicate removal yielded 1860 unique records. After reading titles and abstracts, 1743 were eliminated. The remaining 117 full-texts were screened for eligibility. The number of records and reasons for removing are listed in Figure 1 . Finally, 40 records were included in this systematic review, out of which 21 contained enough data to generate the diagnostic confusion matrix and were subsequently included in the meta-analysis. Neither their reference lists nor relevant reviews provided additional eligible studies.

The characteristics of studies included in this review were listed in Table 1 (19–58). Studies were published between 2011 and 2023. All but one of the included studies enrolled patients retrospectively. The range of patients in the studies that were included varied from 30 to 1419. The imaging modalities used included CT, PET/CT, MRI and PET/MRI. Two studies (51, 58) constructed radiomics models based on images from two or more imaging modalities. Twenty-five studies segmented regions of interests manually, 4 automatically, and 11 semi-automatically. Most studies segmented the regions of tumors only, while two studies (35, 41) also segmented the peritumoral regions of interests. ITK-SNAP (8/40) was the most used segmentation software, next the LIFEx (3/40). Matlab (8/40) was used frequently to extract radiomics features, followed by pyradiomics (7/40). The extracted features included first-, second-, and higher-order features, along with shape features. Logistic regression (11/40) and SVM (10/40) were the most two commonly used classifiers. Sixteen studies included non-radiomics features in multivariable models such as gender, tumor location, and smoking status.

The summarized QUADAS-2 results are showed in Figure 2 . In the domain of patient selection, 2 studies did not provide the information of patient, 5 did not randomly or consecutively enrolled patients, and the exclusion criteria in 7 studies were inappropriate. With regards to the domain of index test, 4 studies did not describe the imaging acquisition well, 6 did not describe the segmentation methods in detail, 6 did not describe the feature extraction software. Independent validation was missed in 17 studies. All included studies used the reference standard which could correctly classify between LUAD and LUSC. The interval between imaging and reference standards was not reported in 5 studies. The high concern of applicability was observed in the aspects of index test (12/40) and patient selection (7/40). The answers to 9 signaling question of included studies are showed in Supplementary Table S3 . The absolute agreement of 9 signaling questions ranged from 86.3% to 100%.

The total RQS of included studies ranged from -5 to 19, with median of 12 ( Table 2 ). The median RQS proportion was 33.3%, with a maximum of 52.8%. The RQS guideline did not define low or high quality clearly. However, 17 studies scored less than 30%. No studies included in the analysis performed multiple segmentation or cost-effective analysis. Feature reduction was conducted in 90% studies prior to modeling. More than half of the studies (23/40) used independent validation sets. Only 6 studies reported calibration statistics. Six studies reported the potential clinical utility. Twenty-nine studies published neither code nor data. The scores for each item and total scores of each study are presented in Table 2 . The Fleiss kappa statics varied from 84.4% to 100% for each item and ICC of 98.1% (95%CI: 96.4%~99.0%) for the overall RQS showed a satisfactory level of agreement between the two reviewers, as shown in Table 2 .

The pooled analysis was performed according to radiomics studies based on various imaging techniques ( Supplementary Table S4 ). Analysis could not be performed on only one study that utilized PET/MRI images. Table 3 and Figures 3 – 6 presented the combined sensitivity, specificity, and SROC-AUC for CT images, PET images, PET-CT images and MRI images. The radiomics model utilizing PET-CT images exhibited the greatest combined effects magnitudes, with a sensitivity of 0.87(95%CI: 0.72~0.94), specificity of 0.88(95%CI: 0.80~0.93) and SROC-AUC of 0.93(95%CI: 0.91~0.95), correspondingly. The radiomics model, which utilized CT and PET images, demonstrated favorable diagnostic performance with an SROC-AUC of 0.86 (95%CI: 0.82~0.89) and 0.85(95%CI: 0.82~0.88). Additionally, it exhibited moderate performance when applied to MRI images, achieving an SROC-AUC of 0.79 (95%CI: 0.75~0.82).

No heterogeneity was observed among the 4 studies that utilized MRI radiomics features, as indicated by I² value of 0% for the combined sensitivity and specificity ( Figure 6 ). With regard to the pooled analysis based on other imaging modalities, there was substantial heterogeneity among studies with I² >50% ( Figures 3 - 5 ).

Out of the 11 studies that developed radiomics models using CT images, 5 studies incorporated non-radiomics features in their models, resulting in higher sensitivity (0.82 compared to 0.76), specificity (0.95 compared to 0.77) and SROC-AUC (0.89 compared to 0.83) ( Table 3 , Supplementary Figure S1 ) than the 6 studies did not include non-radiomics features ( Table 3 , Supplementary Figure S2 ).

The sensitivity analysis showed two studies had impact on the pooled results of radiomics studies utilizing CT images ( Figure 7 ). After removing the two studies (20, 36), the combined sensitivity, specificity, and SROC-AUC were 0.81(95%CI: 0.75~0.85), 0.87(95%CI: 0.78~0.93), and 0.87(95%CI: 0.84~0.90), correspondingly ( Table 3 , Supplementary Figure S3 ).

The sensitivity analysis detected one outlier study (45) with the highest diagnostic performance (sensitivity of 0.99 and specificity of 0.96) among the included studies based on PET-CT images ( Figure 8 ). By omitting the study, the combined sensitivity, specificity, and SROC-AUC were 0.82(95%CI: 0.78~0.85), 0.86(95%CI: 0.79~0.91) and 0.86(95%CI: 0.83~0.89), respectively ( Table 3 , Supplementary Figure S4 ). The study also influenced the heterogeneity. When it was removed, the heterogeneity indicator I² of sensitivity, and specificity, decreased from 84.7% to 21.2%, 77.6% to 46.1%, respectively.

The sensitivity analysis of pooled analysis based on PET images ( Supplementary Figure S5 ) and MRI images ( Supplementary Figure S6 ) did not find any studies impacting the pooled results.

Figure 9 displays Deeks’ funnel plot and the result of Deeks’ asymmetry test, which showcases the publication bias of the studies that developed radiomics classification models based on CT images. The figure did not exhibit evident asymmetry with P value of 0.83. The assessment of publication bias was not conducted for other subgroups due to the limited number of studies, which could result in an inconclusive funnel plot (59).