Physicians are concerned about the prognosis following a diagnosis of acute ischemic stroke (AIS) and must establish a long-term treatment plan (1). Traditional risk-scoring models have been used to predict clinical outcomes after AIS (2–4), but these models face challenges when learning the input of complex multi-dimensional functions. The optimal weight for one input can easily change depending on other input values (5, 6). On the other hand, machine learning (ML) algorithms have the advantage of being able to maximize information to enhance predictive accuracy while predicting complex clinical outcomes influenced by various situations and conditions.

However, the clinical application of ML models has been hindered by the interpretability challenge due to the black box problem (7, 8). In medicine, the lack of interpretability can make it difficult for clinicians to trust the predictions of the model and incorporate them into their decision-making (9). The SHapley Additive exPlanations (SHAP) is an attribution method used to provide interpretation and an intuitive understanding of the contribution of each variable to the outcome (10).

In this study, we aimed to develop and validate an ML-based model that predicts the functional outcome of patients with AIS. Furthermore, we aimed to provide interpretation and an intuitive understanding of the model using model-agnostic attribution methods.

A total of 3,687 patients were included in the derivation dataset, of which 1,123 (30.4%) had an unfavorable outcome. The external validation dataset had 360 patients: 250 in external dataset A and 110 in dataset B, with 93 (37.2%) and 32 (29.1%) unfavorable outcomes, respectively (Supplementary Figure S3). All missing values for the derivation dataset common variables were within 10% (Supplementary Figure S4). Hence, multivariate imputation-chained equations were applied to all variables (14). Baseline demographics and clinical characteristics according to the favorable and unfavorable outcomes of the dataset on which the imputation was performed are listed in Supplementary Table S3.

In this study, we demonstrated the application of machine learning (ML) algorithms to predict the functional outcomes following an acute ischemic stroke. Our model provided an explanation not only based on individual patient data but also utilized the entire dataset to visualize the contribution and directionality of specific input features through SHAP.

The features that contributed most significantly to our study were initial NIHSS, END, and age, which are well-known risk factors for stroke functional outcomes and have been consistently reported in previous studies (2, 17–20). In the case of a strong linear relationship between two variables where the outcome variable is binary, the predicted probabilities can manifest as a sigmoid pattern (21, 22). Interestingly, in our study, initial NIHSS showed a sigmoid curve relative to the 3-month binary functional outcome (Figure 2).

In the prediction of unfavorable outcomes, END exhibited clear interactions with most variables. When the initial NIHSS was < 3 points, the absence of END indicated a negative association with unfavorable outcomes. However, in cases where NIHSS was 3 or higher, the absence of END contributed more to unfavorable outcomes. While this finding may seem counterintuitive, it can be understood when considering that the most common cause of END is a progressive stroke, such as lacunar infarction (23). The mechanism of this progression may well be related to the concept of branch atheromatous disease, suggesting that a hypoperfused area in the perforating arteriole region may deteriorate after admission, leading to the occurrence of END, particularly in patients who were admitted with lower NIHSS scores.

This notion aligns with the results from a previous cohort study conducted on patients with lacunar infarction, where END was associated with worse functional results at 90 days, and low NIHSS scores at admission along with low perfusion lesions predicted END (18). Our findings corroborate these results. Furthermore, the same cohort study suggested that patients who received IVT treatment demonstrated improved functional outcomes at 90 days, and a conclusion is also reflected in our study.

As shown in Figure 3D, the presence of END within 12 h of admission demonstrated a positive interaction with unfavorable outcomes, whereas it exhibited a negative interaction in patients who arrived later. This discrepancy could be attributed to the fact that patients arriving at the hospital quickly are more likely to have END identified. However, for those arriving later, even if END has occurred, it may not be recorded as such in the data if the event happened outside the hospital, thus potentially explaining this observed interaction.

The other features, while not as widely recognized as risk factors for 3-month functional outcomes as the ones previously mentioned, showed consistency in our model interpretation with existing research results (Figure 2). The post-hoc analysis of the Enhanced Control of Hypertension and Thrombolysis Stroke Study (ENCHANTED), which indicated that hyperglycemia and increased WBC count independently relate to poor functional outcomes (24), was reproduced in our study. Interestingly, lower WBC counts at admission were associated with better functional outcomes regardless of treatment (25), and the previously observed association of increased WBC with poor functional outcomes seemed to be reproduced in our results as a U-shaped impact pattern.

Consistent with previous studies suggesting a potential link between increased BUN and unfavorable outcomes (26), our study also found an association between increased BUN and unfavorable outcomes. BUN exhibited a V-shaped pattern, which seems to be a plausible non-linear result when combined with previous findings that a decrease in BUN is associated with better neurological improvement (27).

Our study found that a decrease in hemoglobin contributed significantly to unfavorable outcomes. Previous studies have also found significant associations between abnormal hemoglobin levels and poor outcomes (28, 29).

The duration of symptoms to hospital admission within 3 h tended to decrease the likelihood of unfavorable outcomes but showed a positive association thereafter. However, this influence slightly decreased as the duration increased. This likely reflects the impact of being able to use rtPA for those who arrived within 3 h.

In Figure 1, EVT showed relatively low importance for the model, but the dependency plot revealed its association with a favorable outcome in patients with an initial NIHSS value of 6 or more (Supplementary Figure S7c). This aligns with guidelines suggesting that performing EVT on patients with an initial NIHSS score of 6 or higher correlates with better prognoses (30). However, it is necessary to interpret with caution, considering that features with low contributions might have limited clinical significance for the model.

We looked at the interpretation between the classical generalized linear models and the interpretation of ML-based SHAP. Referring to the estimates from the multivariable logistic regression model based on the dataset used in this study, there was no considerable difference in the contributions of major features derived from the SHAP of the XGB model (Figure 1, Supplementary Figure S9). The overall consistency in the results from both approaches in our dataset can be attributed to the following reasons. As shown in Figure 2, variables considered crucial in this study (initial NIHSS, Age, and END) did not exhibit complex non-linear or multi-dimensional relationships. Datasets with such simple inter-variable relationships are likely to result in no substantial difference in predictive performance between traditional and ML models. Indeed, the AUC of the logistic regression model was not notably different from the XGB model. The AUC for the logistic regression was 0.792 (0.776–0.807) for the internal test set, 0.795 (0.737–0.853) for external validation set A, and 0.893 (0.829–0.956) for set B. These AUC values from the logistic regression did not show a statistically significant difference when compared to the AUC of the XGB model (p = 0.793, 0.738, and 0.247 for the internal test set, external validation sets A and B, respectively). However, in datasets with more complex relationships and interactions between variables, the interpretation results and performance of these two models might differ. Ultimately, understanding the characteristics and structure of the data and choosing the appropriate model may be pivotal in clinical research and practical decision-making.

Nevertheless, interpretable methods such as SHAP provides added value. Through the use of SHAP, we can visually understand not only the intensity of the contribution of risk factors to outcomes but also the non-linear contribution of specific risk factors and the interactions between variables. These are insights that were not discernible in previous traditional statistical methods. Through such analysis, we gain a deep understanding of how key features contribute to model predictions and how changes in their values impact those predictions. For example, we can not only reproduce the results of previous studies but also decipher hidden implications within those results. Furthermore, we can identify lesser-known risk factors. In this way, interpretable methods can provide insights about characteristics observed in clinical settings, potentially improving model performance and facilitating its use in medical practice.

Our research has several distinct advantages over previous studies. Earlier research efforts particularly showed limitations in their generalizability. This was mainly due to the limited amount of data relative to the number of features and the absence of external validation (31–33). However, our study overcame these limitations by utilizing a much broader sample and conducting both internal cross-validation and external validations.

Furthermore, while past research lacked clear methodologies or approaches to the interpretability of ML models, our research sought to address this gap by employing SHAP. Through this, we were able to delve deeper into the interactions and relationships among the variables used in our study. We believe that our research will significantly contribute to expanding the current knowledge on post-stroke management in clinical decision support systems powered by AI.

Our study has several limitations. First, due to the multicenter retrospective nature of the investigation, the performance results may be inadequate for determining the robustness of the ML model for clinical utility. Well-designed prospective cohort studies are required to provide clear evidence for clinical use.

Second, the performance of our model in some external datasets was superior to that in internal validation. There could be multiple explanations for this phenomenon: (1) If the training data contain many noises or outliers, the model may overfit by learning these noises. In this case, if the test data are cleaner or better reflect the general pattern, the performance on these datasets may be better; (2) The performance results could also vary depending on the evaluation metric used. Particularly, if the distribution of training data and test data differ, the same model can have varying performance depending on the dataset. In such a case, the test dataset could receive a higher score in a particular evaluation metric; and (3) If the model is robust, that is, if it can capture specific patterns or trends not present in the training data, it can enhance performance in the test data. This happens because the model can respond well to new patterns that were not observed in the training data. If the model is not overfit and is appropriately generalized, it can have a high predictive capability for new data.

Third, our study primarily employed a cross-sectional research design, focusing on predicting short-term outcomes. While this offers immediate insights, it captures data at a single point in time, potentially limiting our ability to track changes over time. Additionally, this approach might not provide a comprehensive perspective on how these outcomes might evolve or persist in the long run. A longitudinal study on the long-term outcomes of stroke should be undertaken as it could shed light on the temporal patterns and variations, granting deeper insights into the cause-and-effect relationships.

Fourth, while SHAP helps infer the importance of features for a given model, there may be inherent problems in the SHAP inferences if the model is poorly developed or trained. However, we developed our model using a large dataset and also carried out external validation.

In conclusion, machine learning algorithms, specifically the tree-based model, can be used to predict the 3-month functional outcome in patients with acute ischemic stroke. Through the utilization of the SHAP method, we can gain an in-depth understanding of how critical features contribute to model predictions and how their changes influence these predictions. Future work should focus on refining the model, exploring additional predictive features, and validating these findings in prospective cohort studies.

The data analyzed in this study is subject to the following licenses/restrictions: access to the data from the National Information Society Agency (https://aihub.or.kr/) has been granted, but the data is only available for review by domestic residents who have received approval. Requests to access these datasets should be directed to SP, risepsh@gmail.com.

The studies involving humans were approved by Institutional Review Board (IRB No. 2021-09-025-006) of Haeundae Paik Hospital. The studies were conducted in accordance with the local legislation and institutional requirements. The Ethics Committee/institutional review board waived the requirement of written informed consent for participation from the participants or the participants' legal guardians/next of kin because the design of the study was retrospective.

JL: data curation and manuscript editing. KMP: data curation. SP: study design and concept, data curation and analysis, and manuscript editing. All authors contributed to this article and approved the final manuscript.