A total of 1,100 patients with AC- LVO admitted between June 2016 and April 2018 at the Second Hospital of Hebei Medical University, North China, were registered in the derivation cohort; 927 of them who presented with AIS related with AC-LVO and asymptomatic AC-LVO were retrospectively reviewed. In addition, 471 patients with first-ever ischemic stroke (including disabling and non-disabling stroke) were selected. For the external validation, we collected data of patients with AC-LVO from Hebei Province People's Hospital, China between September 2016 and April 2021.

Anterior circulation-LVO was defined as complete occlusion of at least one intracranial internal carotid artery (ICA) or middle cerebral artery (MCA) visualized on computed tomography angiography (CTA) or magnetic resonance angiography (MRA). ICA occlusion refers to the complete occlusion of the C1–C7 segment of the internal carotid artery based on CTA or MRA. MCA occlusion refers to the occlusion of the MCA involving at least the M1 segment (for more details please see in Supplementary Figure I). Asymptomatic AC-LVO was defined as the absence of a transient ischemic attack (TIA), amaurosis fugax, and ischemic stroke attributed to anterior circulation large vessel (11, 12). In accordance with previous studies, disabling and non-disabling ischemic strokes were defined by the initial clinician as National Institutes of Health Stroke Scale (NIHSS) > 5 and ≤ 5 on admission, respectively (13).

Patient characteristics that were collected on admission for the development of Models 1 and 2 include (1) demographic data of the patients such as the age, sex, body mass index (BMI), current smoking and drinking status, comorbidity (hypertension, coronary atherosclerotic heart disease, atrial fibrillation, diabetes mellitus, and hyperlipidemia), history of transient ischemic attack (TIA); (2) clinical variables such as serum apolipoprotein B (Apo B) and homocysteine on arrival; (3) imaging variables such as occluded vessels (unilateral MCA, unilateral ICA, and multiple arteries), posterior circulation large vessel severe stenosis (≥ 70%) /occlusion, anterior cerebral artery (ACA) occlusion, and Alberta Stroke Program Early CT Score (ASPECTS). Data on 14 variables were included in Model 1, and on 12 in Model 2. Specifically, normal blood flow status of the vertebrobasilar arteries via the posterior communicating artery plays a major role in primary collateral compensation after anterior circulation large vessel occlusion. Therefore, posterior circulation large vessel stenosis/occlusion was introduced into Model 1. Posterior circulation large vessel refers to the intracranial vertebral artery, basilar artery, or segment P1 of the posterior cerebral artery.

Processing of the data was performed using Python. First, records containing outliers, which were identified by boxplot, were excluded. Furthermore, the median imputation method was used to impute missing values in derivation cohorts. Finally, the categorical variables were converted into numerical values with dummy encoding, and the continuous features were standardized by removing the mean and scaling to unit variance.

Machine learning is a discipline that constructs models base on data, which is a part of artificial intelligence. Machine learning extracts the characteristics and abstracts the model of the data, discovers the information in the data, and then analyzes and predicts it. First, an algorithm and some parameters of the model which were supplied with training data were selected arbitrarily. During training procedures, the model automatically adjusts some trainable parameters stage by stage to achieve better performance optimization. After the training, all the model parameters are fixed. Importantly, the true effectiveness of the model was evaluated using test data that were completely separate from the training data.

We selected logistic regression without regularization (LR), regularized logistic regression (RLR), random forest (RF), and support vector machine (SVM) as machine learning algorithms that are commonly used.

Logistic regression, a classic classification algorithm in machine learning, was regularized using a combination of L1 and L2 loss in this study. Here, the target was determined by Y:

We selected binary cross-entropy loss as the cost function, where y is the ground truth, y^∧ is the predicted score of the model, and R represents the regularization. The loss functions L1 and L2 are defined as follows:

In the training process of the model, standardization of numerical variables was carried out to accelerate the convergence process and speed of the model.

Random forest is an extended variant of bagging, which uses a decision tree as the base learner and introduces the selection of random attributes in the training process of the decision tree (14). The main parameters that can affect the model performance in RF include the number of trees in the forest, maximum depth of the tree, minimum number of samples required to split an internal node, minimum number of samples required to be at a leaf node, and function to measure the quality of a split. In this study, the values in the dataset were discretized, and the parameters were optimized with a grid search during the training process.

An SVM classifies data by calculating the maximum-margin hyperplane, which adds a regularization term in the solving process to optimize the structural risk. The strength of SVM is that it can process complex datasets with many variables or dimensions (15). The validity of SVM depends mainly on the selection of the kernel function, parameters of the kernel, and soft margin parameter C. Otherwise, in this study, each combination of parameter selections was checked using cross-validation, and only parameters with optimal accuracy were selected.

Moreover, LR, RLR, RF, and SVM can estimate the contribution of each feature to the model by calculating the absolute value of the standardized regression coefficient, information gain / Gini coefficient, and weight coefficient.

In this study, for model derivation, we adopted 5-fold cross-validation, which is a standard way of optimizing the model with inner test data and has been used in a previous study (16). During modeling, the grid search algorithm which is a greedy algorithm was combined to tune and optimize the model hyperparameters. For each group of hyperparameters, we selected 5-fold cross-validation to determine the optimal ones, after which we calculated the means of sensitivity, specificity, accuracy, and AUC to evaluate the performance of each model (Figure 1).

The derivation and validation models were conducted using Python 3.6. The model algorithms, cross-validation, and grid search were based on the Scikit-Learn library of Python in the PyCharm. Matplotlib 3.3.3, NumPy 1.19.5, pandas version 1.1.5, and Scikit-Learn toolkit version 0.21.0 were used to train the machine learning models.

After internal training and testing, the performance of the model was evaluated using external validation data. Subsequently, the AUCs were compared among machine learning algorithms.

Clinical variables are presented as mean ± SD or median with interquartile range, depending on the distribution of the variables. To compare the group differences, continuous variables were compared using the Student's t-test or Mann-Whitney U test, and categorical variables were compared with the χ2 test or Fisher's exact test. These two prediction models were discriminated against using AUC. Calculation of AUC, sensitivity, specificity, precision, negative predictive value (NPV), and accuracy criteria were performed with R statistical software version 3.0.2. The area under the precision-recall curve (PRC) and F1-score were calculated with MedCalc. For the derivation and validation cohorts, a comparison of AUC among the machine learning methods was performed using the DeLong test with Bonferroni correction. Two-sided P < 0.05 were considered statistically significant.

Figures 2A,B illustrate the flow diagram of the enrolled patients. For the derivation cohort, 1,100 patients with AC-LVO were hospitalized at the study institution. After excluding 173 patients with prior stroke or TIA as non-acute ischemic cerebrovascular events, 713 patients with AIS related with AC-LAO and 214 with asymptomatic AC-LAO were finally included in the analysis for model 1. Among the 214 patients with asymptomatic AC-LVO, 119 (56%) were hospitalized for head discomfort such as heaviness of the head and fullness in the head. The other reasons for hospitalization in patients with asymptomatic AC-LVO included coronary artery disease, subarachnoid hemorrhage, migraine, cerebral large artery disease detected by routine physical examination, unruptured intracranial aneurysms, diabetic peripheral vascular disease, central nervous system infection, lower extremity atherosclerotic occlusive disease, intracranial space-occupying lesions, epilepsy, Parkinson's disease, cerebral atrophy, subclavian artery steal blood syndrome, cough syncope, and cardiac syncope. The general screening of large artery disease was performed with transcranial Doppler and carotid artery ultrasound in these patients. Further computed tomography angiography (CTA) or magnetic resonance angiography (MRA) examinations were conducted and AC-LVO was identified. Among the 713 patients with AIS, 242 with prior stroke were excluded, and 471 patients with first-ever ischemic stroke (254 with disabling and 217 with non-disabling strokes) were included in the analysis for Model 2. For the external validation cohort, 150 eligible patients with AC-LVO were included in Model 1. Of the 99 patients with AIS, 82 who presented with the first episode were included in the analysis for model 2. The baseline characteristics of the included patients are presented in Tables 1, 2, and Supplementary Tables I–IV.

The performance metrics of each approach for Models 1 and 2 in the derivation cohort are shown in Tables 3, 4, respectively. The receiver operating characteristic (ROC) curve (indicating the predictive performance of our LR/RLR/RF/SVM model) for each algorithm in the two models and the comparison among these machine learning algorithms are shown in Figures 4A,C. In model 1, the AUCs of RF and SVM were significantly higher than those of the LR and RLR, when using the DeLong test with Bonferroni correction (RF vs. LR, P < 0.0001; RF vs. RLR, P < 0.0001; SVM vs. LR, P < 0.0001; SVM vs. RLR, P < 0.0001; Figure 3A). Similar results were obtained for accuracy and F1-score. In model 2, while the differences in AUCs among the four machine learning algorithms were not significant (Figure 3C), the RF showed the most perfect classification accuracy (71.8%) compared to that of the other machine learning approaches.

The ROC curves for Models 1 and 2 in the external validation cohort are shown in Figures 4B,D. In Model 1, RF exhibited the worst performance among the machine learning models (Table 5). The AUCs in LR, RLR and SVM were significantly higher than that in RF, when using the Delong test with Bonferroni correction (LR vs. RF, P = 0.0048; RLR vs. RF, P = 0.0048, SVM vs. RF, P = 0.0006; Figure 3B). In Model 2, there was no difference in AUCs among the four machine learning algorithms (Figure 3D). The AUC of each algorithm was as follows: LR 0.68 (95% CI 0.56–0.8), RLR 0.76 (95% CI 0.66–0.87), RF 0.71 (95% CI 0.59–0.83) and SVM 0.77 (95% CI 0.66–0.87) (Table 6).

After calculating the importance of each feature, the top five selected variables of Models 1 and 2 were ranked by their discriminative performance (Figures 5, 6). For LR and RLR, the absolute value of the standardized regression coefficient was calculated in both models. For RF, the important features for information gain and Gini coefficient were ranked in Models 1 and 2 respectively. For SVM, the absolute value of the weight was used to rank the variables only in model 2 due to the introduction of the kernel function in Model 1. The absolute values of the important metrics for the features were normalized, ensuring the comparability in feature importance ranking. In Model 1, homocysteine, occluded vessels and BMI appeared together in the top five rankings of all machine learning algorithms. In addition, coronary atherosclerotic heart disease was an important feature in both LR and RLR. Age and Apo B appeared to be important variables in RF. In Model 2, ASPECT, age and BMI were common variables for all machine learning algorithms. Prior TIA was included in LR, RLR, and RF. Hypertension, current smoking, and gender appeared in RLR, RF, and SVM, respectively. Furthermore, occluded vessels coexisted in LR and SVM.