The internal and external validation sets used in this study are overviewed in Figure 1. One dataset was based on the Korean Atrial Fibrillation Evaluation Registry in Ischemic Stroke Patients (K-ATTENTION), which compiled medical information from AF-related stroke patients admitted within 7 days of symptom onset at 11 tertiary stroke centers in South Korea. K-ATTENTION has previously been used for model training, variable selection, and internal validation. The other dataset, used for external validation, comprised patients with AF-related stroke events extracted from the Korea University Stroke Registry (KUSR), collected from three Korean university hospitals (Anam, Ansan, and Guro branches). Common features among clinical, radiological, and laboratory findings acquired during early hospitalization were extracted from the two datasets to develop the models (11). Features with a loss of information pertaining to 3-month functional outcomes measured using the modified Rankin Scale (mRS) were excluded.

Approval was obtained from the institutional review boards of all participating centers. This study complied with the transparent reporting of a multivariable prediction model for individual prognosis or diagnosis reporting guidelines (13).

The outcomes of interest were short-term outcomes following 90 days of the index stroke. Primary outcomes were defined as unfavorable functional outcomes with an mRS ≥ 3. The secondary outcome was mortality, defined as all-cause death within the 90-day period.

Approximately 40% of the patients from the K-ATTENTION registry were randomly selected and stratified by outcomes and variable subgroups (Figure 1). These data were used as the internal validation set, whereas the remaining data were allocated for training. Models were trained using a 10-fold cross-validation strategy.

Variables with a missing rate exceeding 20% were excluded, and missing values were imputed using multivariate imputation by chained equations (MICE) (14). Outliers were detected using an isolation forest (15) and replaced with the closest normal values from the training set. Supplementary Table S1 from the Supplementary Information lists all variables included in the analysis along with their missing rates.

The contribution of each variable to model prediction was evaluated using the Shapley additive explanations (SHAP) method (16). Positive and negative SHAP values indicate positive and negative effects, respectively, on the prediction score. A greedy backward selection method, namely, recursive feature elimination (17), was used to select the best feature set to maximize cross-validation performance by evaluating SHAP in every recursion. The mean absolute SHAP value for each variable was calculated as a ranking criterion representing the importance of each variable. The light gradient boosting machine (LightGBM) (18), a gradient-boosted tree-based model that can handle categorical variables, was used for SHAP evaluation.

The SHAP method was used to investigate the local interpretability of the developed LightGBM model. Model predictions were visualized for true-positive, true-negative, false-positive, and false-negative cases.

A logistic regression with L2 regularization was used as the baseline comparator. We tested two representative machine learning models, namely, LightGBM and the multi-layer perceptron (MLP). MLP is a feedforward neural network with fully connected layers. Hyperparameters were tuned using Bayesian optimization (19) to maximize the predictive performance of cross-validation. All details pertaining to hyperparameter settings are described in the Supplementary Methods and Supplementary Table S2. Models were calibrated using isotonic regression on the validation data during cross-validation. All methods used in this study were implemented in Python version 3.9.7 using Scikit-learn version 1.1.2.

The model performance of internal validation was evaluated in different subgroup cohorts. Age (≤64, 65–74, and ≥ 75 years), sex, hypertension, diabetes mellitus, type of AF (sustained and paroxysmal AF), and stroke recurrence (first-ever and recurrent stroke) were defined as subgroups.

Descriptive statistics were expressed as numbers (percentages), means (standard deviations), or medians (interquartile ranges). The Shapiro–Wilk test was used for normality, and Levene’s test was used for homoscedasticity. The chi-squared test, independent t-test, or Mann–Whitney U-test was used for comparison.

The area under the receiver operating characteristic curve (AUROC) was used as the primary outcome metric, as well as a guiding metric in all model training processes, including variable selection (recursive feature elimination) and hyperparameter tuning (Bayesian optimization). The AUROC was calculated and compared between models using the DeLong method (20). Furthermore, we evaluated the area under the precision–recall curve (AUPRC) of the models as an overall performance measure.

The detailed performance of each model was evaluated in terms of sensitivity, positive predictive value, and negative predictive value with net reclassification improvement (NRI) at low false positive rates (FPRs) of 5, 10, and 20%. A positive NRI value indicates superior reclassification performance by the new model compared to that of the reference model (21). We evaluated sensitivity at fixed FPR levels in the subgroups using the model that exhibited the best overall performance. Calibration errors were evaluated before and after calibration using the calibration curve and Brier score, which is the mean-squared error of predicted probability (22). We evaluated the model’s capability of risk stratification for the prediction of 3-month mortality. Four quartile strata were obtained from the prediction scores of the best-performing model, and survival curves were plotted using Kaplan–Meier estimation and compared using the log-rank test. The cutoff thresholds for investigating local interpretability were determined at an FPR level of 20%.

p-values were adjusted using the Benjamini–Hochberg method for multiple comparisons. The significance level was set at a p-value of <0.05.

A total of 2,307 and 898 patients were included in the K-ATTENTION and KUSR groups, respectively (Figure 1). Table 1 presents a comparison between the two registries, including the general characteristics of the study participants, revealing an unfavorable functional outcome rate of 49.8% and the mortality rate of 11.6% 3 months following the index stroke. These results are compatible with unfavorable functional outcomes at 3 months poststroke (p > 0.05), whereas the models significantly differed in terms of 3-month mortality (p < 0.05). The patients in the two registries were comparable in terms of sex, initial National Institute of Health and Stroke Scale (NIHSS) score, hypertension, and history of vascular diseases, including coronary and peripheral artery diseases. However, patients in the KUSR group were generally older and had higher initial blood pressure and body mass index (BMI) values than those in the K-ATTENTION group. Furthermore, patients in the KUSR group were more likely to exhibit a severe pre-stroke functional status, persistent AF, and a history of congestive heart failure and diabetes mellitus, whereas they had a lower previous history of stroke. The initial imaging findings revealed significant differences in lesion lateralization, diffusion-weighted imaging (DWI) lesion patterns, and concomitant intracranial and extracranial artery stenosis.

An evaluation of model performance is presented in Figure 2, indicating that the machine learning models outperform logistic regression in predicting both outcomes. In the prediction of unfavorable functional outcomes, MLP obtained an AUROC value of 0.890 on the internal validation set, representing a significant improvement over the AUROC value of 0.874 obtained by logistic regression. In the external validation set, both LightGBM (0.873) and MLP (0.859) achieved significantly higher AUROC values than those of logistic regression (0.834).

In the prediction of mortality, both LightGBM (0.839) and MLP (0.842) attained significantly higher AUROC values than logistic regression (0.803) on the internal validation set. On the external validation set, the AUROC values of LightGBM (0.805) and MLP (0.797) were also higher than that of logistic regression (0.790) although not significantly so. Furthermore, the machine learning models exhibited higher AUPRC values than those of logistic regression for predicting both outcomes in each validation set.

The cross-validation performance and calibration curves with the Brier scores of the models are presented in Supplementary Table S3 and Supplementary Figure S1, respectively. MLP was used to obtain four quartile strata, and mortality risk stratification was evaluated with pairwise comparisons between the survival curves of the strata using the log-rank test (Supplementary Figure S2). For the internal validation set, all p-values for pairwise comparisons were less than 0.0001, except for those between the first and second quartiles. For the external validation set, p-values for the pairwise comparisons were 0.0002 or less, except for those between adjacent quartiles.

Model performance in predicting unfavorable functional outcomes was consistent across all subgroups, with the lowest AUROC and AUPRC values in patients with recurrent stroke and those aged <65 years, respectively. Both machine learning models exhibited comparable or superior performance to that of logistic regression, with MLP significantly outperforming logistic regression in most subgroups (p < 0.05) except for patients aged <65 and > 74 years and those with recurrent strokes. No significant differences were observed between the two machine learning models or between logistic regression and LightGBM.

Supplementary Figure S3 shows the results of a performance comparison in the prediction of 3-month mortality across subgroups. Performance was also consistent across subgroups, with machine learning models exhibiting superior performance for all subgroups except patients aged <65 years.

Low-FPR sensitivity results in the subgroup cohorts for predicting unfavorable functional outcomes and mortality are presented in Supplementary Figures S4, S5, respectively. Detailed performance and NRI at low FPRs for the prediction of unfavorable functional outcomes and mortality are presented in Supplementary Tables S4, S5, respectively (see Figure 3).

Out of the 43 variables, 34 were selected in the variable selection process, with the 10 most important variables summarized in Figure 4A. The most important variables were the initial NIHSS score, followed by DWI lesion pattern, pre-stroke mRS, and hs-CRP.

Partial SHAP dependence plots for the four representative variables among the top 10 are displayed in Figure 4B, with those for the other six variables presented in Supplementary Figure S6. Thus, ≥ 7.4 points in the initial NIHSS score, having a specific pattern of lesion, including single corticosubcortical lesion, confluent and an additional lesion in one vascular territory, or multiple lesions in multiple vascular territories, ≥ 8,700 cells/μL in white blood cell (WBC) count, ≥ 318.3 mg/dL in fibrinogen, ≤ 22.5 kg/m² in BMI, ≥ 3 in pre-stroke mRS, bilateral or diffuse multifocal lesion lateralization, ≤ 12.9 g/dL in hemoglobin, ≥ 74.3 years in age, and ≤ 51.8 mL/min in creatine clearance contributed to a higher risk of the unfavorable functional outcome. The SHAP values associated with the 10 most important variables exhibited a pattern comparable to linear association. WBC revealed a sigmoid pattern, fibrinogen displayed a J-shaped pattern, and other variables revealed complex linear, sigmoid, and J-shaped patterns.

Supplementary Figure S7 presents the local interpretability of LightGBM for the prediction of unfavorable outcomes with individual cases on an external validation set.

Out of the 16 selected variables, the 10 most important are summarized in Figure 4C. The most important variable was the initial NIHSS score, followed by age, concomitant intracranial/extracranial steno-occlusion, fasting glucose, and creatinine clearance.

Partial SHAP dependence plots for four representative variables are displayed in Figure 4D, with those for the other six variables presented in Supplementary Figure S8. In summary, ≥ 8.2 in the initial NIHSS score, ≥ 74.5 years in age, ≤ 56.6 mg/dL or ≥ 122.6 mg/dL in fasting glucose, and ≤ 22.1 or ≥ 30.3 kg/m² in BMI, ≥ 4 in pre-stroke mRS, ≥ 87.2 IU/L in ALP, ≤ 4.0 mg/dL, or ≥ 7.0 mg/dL in uric acid, multiple lesions in multiple vascular territories, presence of concomitant intracranial or extracranial stenosis, and ≥ 8,300 cells/μl in WBC count predicted mortality. Most association patterns between SHAP values and variables were near-linear, sigmoid, J-shaped, or combinations of the three. BMI and uric acid levels revealed U-shaped patterns.

The local interpretability of LightGBM for the prediction of mortality is demonstrated in Supplementary Figure S9 by presenting individual cases through model prediction on the external validation set.