A comprehensive literature search was conducted in the PubMed, Embase, and Cochrane Library databases up to 8 July 2023, incorporating MeSH terms and free-text keywords related to “Artificial Intelligence,” “Machine Learning,” “Deep Learning,” “Radiomics,” “Computed Tomography Angiography,” and “Plaque.” The purpose of this search strategy was developed to identify all studies assessing the diagnostic performance of AI-assisted CTA for plaque analysis. The strategy included a wide range of relevant literature without restrictions based on publication date or language to avoid bias. The process was refined iteratively to ensure a comprehensive overview of the evidence base on AI-assisted CTA for plaque analysis.

Studies were selected for inclusion after removing duplicates if they met the following criteria: (1) patients with coronary or carotid plaque were included in study; (2) artificial intelligence algorithms based on CT images was applied to evaluate the diagnostic accuracy; (3) at least 10 patients were included; and (4) the true-positive (TP), false-positive (FP), false-negative (FN), and true-negative (TN) rates could be calculated from the data.

Studies were excluded if they were: (1) personal communications, editorials, letters, abstracts, conferences, or case reports; (2) not using AI-assisted CTA images; (3) not investigating humans, but for experimental animals; or (4) non-English publications. Two reviewers (PJ and MF) independently screened all identified studies against the eligibility criteria, extracted the relevant data, and assessed the study quality. Any disagreements were resolved by discussion with a third reviewer (JZ).

Two reviewers independently extracted the following key data from eligible studies using standardized forms: (1) study details: publication year, country, design, sample size, title, population demographics, imaging used; (2) AI algorithm information: imaging modality, feature extraction techniques, algorithm type and name; (3) diagnostic performance metrics: area under the receiver operating characteristic curve (AUROC), sensitivity (SEN), specificity (SPE), TP, FP, FN, and TN.

The quality assessment of all eligible studies was performed independently using the revised Quality Assessment Tool for Diagnostic Accuracy Studies (QUADAS)-2 (27). This comprehensive assessment encompassed four key domains, namely “Patient Selection,” “Index Test,” “Reference Standard,” and “Flow and Timing” to ascertain potential biases, with the first three domains also undergoing scrutiny for concerns relating to their applicability. Quality ratings were assigned as “high,” “low,” or “unclear.” Discrepancies in the assessment process were resolved through consensus between two reviewers (PJ and MF). Furthermore, the bias risk for each included study was evaluated using the Review Manager 5.3 software to facilitate an in-depth analysis of the quality of diagnostic articles.

The meta-analysis and statistical analysis were conducted following rigorous adherence to the established Cochrane guidelines. The presence of the threshold effect was assessed using the spearman correlation coefficient, with a p-value <0.05 serving as an indicator of its presence (28). Moreover, the evaluation of statistical heterogeneity was carried out using the I² statistics (26), The value of I² exceeding 50% was conventionally considered to indicate substantial heterogeneity, while a value of I² below 50% suggested a lower degree of heterogeneity (29). Pooled estimates of SEN, SPE, positive predictive value (PPV), negative predictive value (NPV), and diagnostic odds ratio (DOR) were computed using a random effects model. The derivation of these estimates involved the integration of the raw data on TP, FP, TN, and FN values reported in each study. The certainty of these pooled estimates was further substantiated through the calculation of 95% confidence intervals (95% CIs) (30). In addition, the AUROC was constructed, a graphical representation that encapsulates the diagnostic performance of the AI-assisted system across various threshold settings. The curve allowed for the calculation of the area under the curve (AUC), a pivotal metric was used to quantify the overall diagnostic accuracy. Notably, different ranges of AUROC values (0.5–0.7, 0.7–0.9, and 0.9–1) were utilized to categorize the diagnostic accuracy as low, moderate, and high, respectively (31). Funnel plots were used to assess articles for publication bias. The entire meta-analysis procedure was carried out utilizing the Meta-DiSc 1.4 and “MIDAS” modules within the STATA 17 software (version 17.0 IC; Stata Corporation, College Station, TX, USA). Furthermore, a comprehensive subgroup analysis was implemented to elucidate potential sources of heterogeneity, encompassing various factors, such as arterial region (carotid or coronary artery), study design (prospective or retrospective), the geographic origin of the literature (Asia, Europe, or America), AI methodology (deep learning or machine learning), and study center type (multicenter or single-center). This approach allowed for a thorough exploration of potential variations in the diagnostic accuracy of AI-assisted systems, providing valuable insights into the underlying factors influencing the observed heterogeneity.

The literature search and screening process are presented in Figure 1. A total of 728 potentially eligible articles were initially identified through the database search. After the exclusion of 157 duplicates, 571 articles were subjected to screening based on titles and abstracts, resulting in the selection of 43 studies for full-text review. Ultimately, 11 articles pertaining to AI-assisted CTA diagnostic technology for plaque assessment were incorporated into this meta-analysis (20–24, 32–37), and the comprehensive details of the literature search and screening process are depicted in Figure 1.

Table 1 and Supplementary Table S2 provide a detailed overview of the characteristics of the 11 identified studies comprising 21 trials, with publication dates between 2019 and 2023. The cumulative number of participants across all studies was 1,484, with a wide age range of 33–88 years. Five studies containing 15 trials investigated vessel stenosis caused by plaque, of which 8 trials focused on 50% stenosis and 7 trials on 70% stenosis. Five studies containing five trials investigated calcified plaque and one remaining study investigated plaque vulnerability. Nine studies containing 16 trials investigated coronary plaque and 2 remaining studies containing 5 trials examined carotid plaque. Four studies containing 12 trials were multicenter studies while the remaining 7 studies containing 9 trials were single-center studies. In terms of the applied algorithm, four studies focused on deep learning, five studies used machine learning, and artificial intelligence was used in two studies. Semi and automatic segmentation were applied in nine studies, whereas manual segmentation of images was reported in two studies. The systematic review showed that AI-assisted CTA diagnostic technology for the assessment of plaque had SEN rates in the range of 64.00%–100.00%, SPE rates in the range of 43.90%–99.80%, and AUC in the range of 69.00%–96.00%.

We employed the radiomics quality score (RQS) (38) to assess the quality of the eligible studies. Supplementary Table S3 displays the mean score for each criterion of the RQS across all included studies. The mean RQS score of the studies was determined to be 24.7% [8.9 points, standard deviation (SD) ± 7.9 points], with a median score of 22.2% (8.0 points) and a range of 2.8%–72.2% (1–26 points). Notably, an upward trend in the RQSs of the included studies over time was observed, as depicted in Supplementary Figure S3.

To evaluate the risk of bias and applicability concerns, we applied the QUADAS-2 tool (Figure 2). Concerning patient selection, two (18.18%) studies were deemed to possess a high risk of bias, whereas two (18.18%) studies were classified as low risk and seven (63.64%) studies were characterized by an unclear risk of bias. In the index test domain, seven (63.67%) studies were rated as low risk, while four (36.36%) studies were designated as having an unclear risk of bias. Regarding the reference standard domain, 10 (90.91%) studies were identified as low risk, with 1 (9.09%) study demonstrating an unclear risk of bias. As for flow and timing, four (36.36%) studies were classified as low risk and seven (63.67%) studies were regarded as having an unclear risk of bias.

The analysis revealed a notable risk of bias in the domain of patient selection, with the absence of randomization serving as a primary contributing factor. Overall, the included studies were deemed suitable for subsequent analyses in light of its quality.

The Spearman correlation coefficient was 0.418 (p = 0.201 > 0.05), suggesting that there was no discernible threshold effect for using AI-based CTA to evaluate atherosclerotic plaque. High heterogeneity was shown by the I² values, which were 95.61% and 98.02% for pooled SEN and SPE, respectively (Figure 3). For evaluating atherosclerotic plaque using AI-based CTA, the pooled SEN and SPE were 0.90 (95% CI 0.85–0.93) and 0.93 (95% CI 0.87–0.96), respectively (Figure 3). The results showed that the AUROC was 0.96 (95% CI 0.94–0.97) (Figure 4 and Supplementary Table S4), which indicated a high diagnostic performance. In identifying ≥50% stenotic vessels, the SEN, SPE, and AUROC were 0.90 (95% CI 0.84–0.94), 0.89 (95% CI 0.76–0.96), and 0.95 (95% CI 0.93–0.96). In identifying ≥70% stenotic vessels, the SEN, SPE, and AUROC were 0.87 (95% CI 0.78–0.93), 0.98 (95% CI 0.91–0.99), and 0.96 (95% CI 0.94–0.97), which indicated high diagnostic accuracy. In identifying calcified plaque, the SEN, SPE, and AUROC were 0.94 (95% CI 0.73–0.99), 0.87 (95% CI 0.81–0.92), and 0.92 (95% CI 0.90–0.94), respectively (Supplementary Figure S2).

As shown in Figure 5, Deek's funnel plot showed a p-value of 0.12, suggesting no obvious publication bias was found in all eligible studies.

Table 2 presents the detailed outcomes of the subgroup analyses conducted to investigate the potential sources of heterogeneity. A slight drop in I² was observed in SEN (from 95.61% to 76.57%) and SPE (from 98.02% to 94.80%) in the carotid plaque group after grouping according to whether carotid or coronary plaque was detected. After grouping according to the study design, a drop in I² was observed in SEN (from 95.61% to 84.61%), suggesting the prospective study may reduce the probability of heterogeneity. Studies with a multicenter study design yielded a higher AUROC (0.96 vs. 0.86) and DOR (175 vs. 50) than those with a single-center study design. Further, utilizing machine- and deep-learning models to assist the diagnosis of atherosclerotic plaque has a higher SPE (0.92 vs. 0.81) and AUROC (0.96 vs. 0.90).