Front. Neurol.Frontiers in NeurologyFront. Neurol.1664-2295Frontiers Media S.A.10.3389/fneur.2022.875491NeurologyOriginal ResearchPredicting 30-Day Readmission for Stroke Using Machine Learning Algorithms: A Prospective Cohort StudyChenYu-Ching12ChungJo-Hsuan1YehYu-Jo1LouShi-Jer13LinHsiu-Fen45LinChing-Huang6HsienHong-Hsi7HungKuo-Wei8YehShu-Chuan Jennifer19ShiHon-Yi1391011*1Department of Healthcare Administration and Medical Informatics, Kaohsiung Medical University, Kaohsiung, Taiwan2Department of Public Health, College of Medicine, National Cheng-Kung University, Tainan, Taiwan3Graduate Institute of Technological and Vocational Education, National Pingtung University of Science and Technology, Pingtung, Taiwan4Department of Neurology, Kaohsiung Medical University Hospital, Kaohsiung, Taiwan5Department of Neurology, College of Medicine, Kaohsiung Medical University, Kaohsiung, Taiwan6Division of Neurology, Kaohsiung Veterans General Hospital, Kaohsiung, Taiwan7Department of Internal Medicine, St. Joseph Hospital, Kaohsiung, Taiwan8Division of Neurology, Department of Internal Medicine, Yuan's General Hospital, Kaohsiung, Taiwan9Department of Business Management, National Sun Yat-Sen University, Kaohsiung, Taiwan10Department of Medical Research, Kaohsiung Medical University Hospital, Kaohsiung, Taiwan11Department of Medical Research, China Medical University Hospital, China Medical University, Taichung, Taiwan
Edited by: Hari Kishan Reddy Indupuru, University of Texas Health Science Center at Houston, United States
Reviewed by: Iqram Hussain, Seoul National University, South Korea; Anwar P. P. Abdul Majeed, Universiti Malaysia Pahang, Malaysia; Thomas T. H. Wan, University of Central Florida, United States; Haoyue Zhang, University of California, Los Angeles, United States
*Correspondence: Hon-Yi Shi hshi@kmu.edu.tw
This article was submitted to Stroke, a section of the journal Frontiers in Neurology
This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner(s) are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.
Background
Machine learning algorithms for predicting 30-day stroke readmission are rarely discussed. The aims of this study were to identify significant predictors of 30-day readmission after stroke and to compare prediction accuracy and area under the receiver operating characteristic (AUROC) curve in five models: artificial neural network (ANN), K nearest neighbor (KNN), random forest (RF), support vector machine (SVM), naive Bayes classifier (NBC), and Cox regression (COX) models.
Methods
The subjects of this prospective cohort study were 1,476 patients with a history of admission for stroke to one of six hospitals between March, 2014, and September, 2019. A training dataset (n = 1,033) was used for model development, and a testing dataset (n = 443) was used for internal validation. Another 167 patients with stroke recruited from October, to December, 2019, were enrolled in the dataset for external validation. A feature importance analysis was also performed to identify the significance of the selected input variables.
Results
For predicting 30-day readmission after stroke, the ANN model had significantly (P < 0.001) higher performance indices compared to the other models. According to the ANN model results, the best predictor of 30-day readmission was PAC followed by nasogastric tube insertion and stroke type (P < 0.05). Using a machine learning ANN model to obtain an accurate estimate of 30-day readmission for stroke and to identify risk factors may improve the precision and efficacy of management for these patients.
Conclusion
Using a machine-learning ANN model to obtain an accurate estimate of 30-day readmission for stroke and to identify risk factors may improve the precision and efficacy of management for these patients. For stroke patients who are candidates for PAC rehabilitation, these predictors have practical applications in educating patients in the expected course of recovery and health outcomes.
Globally, stroke is not only the second leading cause of death, but also the disease with the second largest healthcare burden as estimated in disability-adjusted life-years (1). Previous studies have estimated that as many as 21% of stroke patients are readmitted within 30 days and have found that unplanned Medicare readmission in 2004 estimated in excess of $17 billion in costs (2–4). Furthermore, the mortality rate for 30-day readmission after stroke is more than 2.5 times greater than index admissions and highest among those readmitted for recurrent stroke (2). Additionally, one current study found that ~25.4% of the venous thromboembolism (VTE)-related hospital readmissions occurred within the first 30 days of discharge and they also estimated the mean cost for a hospital readmission with a primary diagnosis of VTE was $18,681; for readmissions with a primary diagnosis of deep vein thrombosis and pulmonary embolism, mean costs were $14,719 and $23,305, respectively (5). Reducing readmission rates among hospitals has become a goal of national healthcare reform.
This prospective study evaluated the use of machine learning algorithms for predicting 30-day readmission after stroke, univariate analysis and feature importance analysis. This study presented a novel opportunity to evaluate the use of post-acute care (PAC) history, demographic characteristics, clinical characteristics, and functional status outcomes as predictors of 30-day readmission in patients with stroke. The results of this study could be used to improve precision and efficacy in managing these patients. These results not only validate the use of similar prediction models for clinical practice in other countries, they also indicate that both PAC and analysis of functional status outcomes should be routinely be integrated in the care for stroke patients.
Although prior works to stratify risk of stroke outcomes have utilized basic statistical models, such as logistic regression been proposed recently, models for predicting readmission have had three major shortcomings. Firstly, recently proposed machine learning models have shown superior area under the receiver operating characteristic (AUROC) curve compared to conventional regression models in predicting 30-day readmission (range: 0.729–0.834 vs. 0.714–0.828, respectively) (6–8). Secondly, proposed forecasting models require use of health insurance claims data, which would not be available in a real-time clinical setting (9). Thirdly, previous studies predicted the risk of readmission do not comprehensively consider baseline patient characteristics, including post-acute care (PAC) history, demographic characteristics, comorbidities, and functional status score (10–12). However, literature on their use for predicting 30-day readmission for stroke is relatively sparse. The current studies regarding to 30-day readmission for patients with cerebrovascular diseases by using machine learning are summarized in Table 1 (6–9, 13–15).
The studies in predicting 30-day readmission for patients by using machine learning.
Advanced machine learning (ML) methods along with natural language processing (NLP) features out performed logistic regression for all-cause readmission [areas under the curve (AUC), 0.64 vs. 0.58; P < 0.001] and stroke readmission prediction (AUC, 0.62 vs. 0.52; P < 0.001)
1. GBM provided the highest AUC (0.68), specificity (0.95), and positive predictive value (PPV) (0.33) when compared to the other models 2. In terms of AUC, specificity, and PPV, the LR had poor performance compared to XGBoost and GBM models
The full logistic regression model with 25 predictors had a C-index of 0.67 as compared with the best-performing ML model (Random Forest) with only 10 predictors and a C-index of 0.73
1. The boosted tree-based ML algorithms had the highest AUC with XGBoost compared to the L1 logistic regression (0.685 vs. 0.591) in predicting 30-day readmission 2. Calibration plots for XGBoost showed that predicted readmission was aligned with observed risks and that low predicted risks were associated with fewer actual outcomes highlighting higher negative predicted values at lower predicted risks
Wang et al. (USA) (14)
47,498 eligible heart failure with reduced ejection fraction patients
1. The best AUCs of deep learning (DL) models without a buffer window in predicting heart failure hospitalizations and worsening heart failure events in the total patient cohort were 0.977 and 0.972, respectively 2. The best AUCs in predicting 30-day readmission in all adult patients were 0.597 and 0.614, respectively 3. For all outcomes assessed, the DL approach outperformed traditional machine learning (ML) models
Amritphale et al. (USA) (15)
16,745 patients with carotid artery stenting
1. Logistic regression (LR) 2. Support vector machine (SVM) 3. Deep neural network (DNN) 4. Random forest (RF) 5. Decision tree (DT)
1. The artificial intelligence machine learning DNN prediction model has a C-statistic value of 0.79 in predicting the patients who might have all-cause unplanned readmission within 30 days of the index carotid artery stenting discharge 2. The DNN model showed a significant higher receiver operating characteristic (ROC; 0.802 vs. 0.680, 0.670, 0.607, and 0.586, respectively) and precision-recall (0.383 vs. 0.140, 0.140, 0.380, and 0.269, respectively) than the LR, SVM, RF, and DT in predicting 30-day readmission among patients with carotid artery stenting
To reduce 30-day readmission after stroke and subsequent mortality, identifying factors that predict readmission is crucial. Determining the risk factors for 30-day readmission may be useful for developing policies for preventing readmission after stroke. Therefore, the aims of this study were to compare forecasting accuracy in the artificial neural network (ANN), K nearest neighbor (KNN), random forest (RF), support vector machine (SVM), naive Bayes classifier (NBC) and Cox regression (COX) models and to explore significant predictors of readmission within 30 days after stroke. The key contributions of this study can be summarized as follows:
Advances in artificial intelligence have been applied in clinical practice. However, machine learning algorithms have not been used to predict 30-day readmission for patients with stroke mainly because of the high complexity of prediction algorithms relative to diagnostic algorithms.
The proposed machine learning algorithms exhibit strong potential for use in predicting readmission within 30 days after stroke.
A feature importance analysis was also performed to determine the significance of the selected input variables.
Materials and MethodsStudy Design and Patients
The subjects of this prospective cohort study were 1,476 patients with a record of an ICD-9-CM (433.01, 433.10, 433.11, 433.21, 433.31, 433.81, 433.91, 434.00, 434.01, 434.11, 434.91 and 436 for ischemic stroke; 430 and 431 for hemorrhagic stroke), ICD-10 (I60–I62 were used to identify hemorrhagic stroke; I63 was used for ischemic stroke), and a history of admission to the PAC ward at one of four hospitals (three regional hospitals and one district hospital) or to a traditional non-PAC ward at one of two medical centers in south Taiwan between March, 2014, and September, 2019. The enrollment criteria were patients hospitalized for their first-ever stroke who were examined within 30 days with computed tomography (CT) or magnetic resonance imaging (MRI) and a Modified Rankin Scale (MRS) score of 2 to 4. Scores for the MRS range from 0 to 6, and a high MRS score indicates a high severity of disability. Patients were excluded if PAC beds were unavailable at the participating hospitals or if they had been transferred to PAC wards at other hospitals. In this scale, absence of symptoms is scored as 0. No significant disability, slight disability moderate disability moderately severe disability, and severe disability is scored as 1, 2, 3, 4 and 5, respectively (16). Another 167 stroke patients were recruited from October to December, 2019 (Figure 1). Figure 2 also depicts the conceptual framework of the proposed method for predicting readmission within 30 days after stroke. The study protocol was approved by the institutional review board at Kaohsiung Medical University Hospital (KMUH-IRB-20140308), and written informed consent was obtained from each participant.
Flowchart of the study.
Conceptual framework of the proposed method for predicting readmission within 30 days after stroke.
Instruments and Potential Predictors
Functional disability was measured using the 10-item Barthel Index (BI) (17). The BI measures functional disability in terms of inability to perform certain daily life activities (e.g., dressing, performing self-care, and walking up and down stairs). A BI score of 10 indicates complete independence. In stroke patients who had dysphagia, functional oral intake was assessed with the Functional Oral Intake Scale (FOIS) (18), in which swallowing function is classified on a scale from 1 (nil by mouth) to 7 (total oral diet without restriction). Cognitive status was quantitatively assessed with the Mini-mental State Examination (MMSE) (19). The MMSE includes tests for orientation, memory, attention, calculation, language, and construction functions where higher scores indicate better functional status (total score range, 0–30). The Instrumental Activities of Daily Living (IADL) scale is most useful for assessing current function and improvement or deterioration in function over time (20). When the IADL scale is administered in women, all eight domains for function are scored. In men, the domains of food preparation, housekeeping, and laundering are not scored. The EuroQoL Quality of Life Scale (EQ-5D-3L) measures the total health state of the subject based on a self-assessment of 5 items: mobility, self-care, usual activities, pain or discomfort, and anxiety or depression (21). Each EQ-5D-3L item is scored as 1 (no problem), 2 (some problem), or 3 (extreme problem). The 14-item Berg Balance Scale (BBS) is used to measure functional balance (22). Each item is rated from 0 (poor) to 4 (good), and the maximum score is 56. The Chinese versions of all instruments used in this study have been validated and used extensively in both clinical practice and research (17, 23).
A research assistant collected the following data from medical records after index discharge: PAC program (PAC group or non-PAC group), patient attributes (age, gender, education, and BMI), clinical attributes [stroke type, NG tube, Foley catheter, hypertension, diabetes mellitus (DM), hyperlipidemia, atrial fibrillation, previous stroke, acute care LOS, and rehabilitation ward LOS]. In multivariate analysis, the potential predictors were the independent variables, and 30-day readmission was the dependent variable.
Machine Learning Algorithms
Machine learning algorithms are effective tools for identifying and classifying readmission within 30 days after discharge in patients with stroke. Previous studies have successfully used machine learning to classify stroke according to characteristics such as cardiac source and gait in various scenarios (24, 25). In the present study, machine learning algorithms used to predict 30-day readmission in patients with stroke included ANN, KNN, RF, SVM, NBC and COX models.
Statistical Analysis
The unit of analysis in this study was the individual patient with stroke. Statistical analysis was performed in the following steps. In the first step, the statistical significance of continuous variables was tested by one-way analysis of variance, and that of categorical variables was tested by Fisher exact analysis. Univariate analyses were performed to identify significant predictors (P < 0.05). In the second step, data for the study cohort of 1,476 subjects were randomly divided into two datasets: a training dataset containing data for 1,033 subjects (70%), which was used for model development, and a testing dataset containing data for 443 subjects (30%), which was used for internal validation. A validation dataset containing data for another 167 patients enrolled after September, 2019, was used for external validation. To identify the optimal hyper-parameters for the machine learning algorithms, we applied Bayesian optimization using the expected improvement as the acquisition function (26). To perform the hyperband method of optimization and to test different combinations of hyper-parameters, we used Optuna version 2.10.0 (27). A total of 1,000 trials were conducted, and the parameters with the greatest area under the receiver operating characteristic curve were saved. Additionally, since data used for model fitting tended to overestimate model performance on unseen subjects, we coupled 10-fold cross-validation (28) with the logistic loss metric to measure the generalizability of the model to unseen subjects during model selection. A total of six machine-learning classifiers were constructed in the training dataset and tested in the validating dataset. A confusion matrix is used to describe and visualize the performance of the machine learning algorithm classifier and also to provide insight on what the model misclassifies. In the present study, the performance of the machine learning algorithms for the best classification task was evaluated in terms of confusion matrix-based performance measuring metrics including sensitivity, specificity, positive predictive value (PPV), negative predictive value (NPV), and accuracy. In order to evaluate and select the most accurate machine learning algorithms, we used a confusion matrix and calculated the percentage of sensitivity, specificity, and accuracy of each forecasting model. In addition, the performance of the machine learning algorithms in the present study was also evaluated by the receiver operating characteristic (ROC) curve and the area under the ROC curve (AUROC). The independent variables fitted to the forecasting models were significant predictors of 30-day readmission, and the dependent variable was 30-day readmission. After model training, model outputs were collected for each testing dataset. In the third step, bootstrapping, a machine learning technique, which involves taking random samples from the dataset with re-selection of 1,000 resamples was used to compare different machine learning algorithms employing the performance indices and the 95% confidence intervals. We used paired t-test to identify performance indices that significantly differed between the two models.
In the fourth and final step, feature selection method was calculated by using an algorithm to obtain an importance score for each potential predictor in the dataset (29). Feature importance analysis provides information about how each feature contributes to model prediction accuracy. The final weight of each feature is calculated by averaging the decrease in model accuracy after random permutation of the feature values within a testing set. Permutation of an important feature should decrease the score whereas permutation of a feature that is not very important to model prediction accuracy should increase the score. To obtain robust results with our small dataset, the train-test split was performed with a repeated stratified K fold cross validation. This technique has two advantages: first, it is model-agnostic; second, it can be performed repeatedly with different feature permutations. All statistical analyses were performed using the STATISTICA 13.0 software package (StatSoft, Inc., Tulsa, OK, USA). All statistical tests were two-sided; a P-value < 0.05 was considered statistically significant.
ResultsStudy Characteristics
Table 1 shows that 1,283 patients (86.9%) joined the per-diem PAC program and the remaining patients selected the fee-for-service non-PAC program. The patients with stroke had a mean age of 65.5 years (standard deviation, SD 13.0 years), and most (62.5%) patients were male. During the study period, 120 patients with stroke were readmitted within 30 days. In univariate analysis, PAC program, age, gender, education, body mass index (BMI), stroke type, nasogastric (NG) tube, Foley, hypertension, diabetes mellitus (DM), hyperlipidemia, atrial fibrillation, previous stroke, acute care length of stay (LOS), rehabilitation LOS and functional status score before rehabilitation were significantly associated with 30-day readmission (P < 0.05). These significant predictors were included in the forecasting models (Table 2).
Baseline characteristics of the study population (N = 1,476).
Variables
Mean ±SD or N (%)*
Post-acute care
No
193 (13.1)
Yes
1,283 (86.9)
Patient attributes
Age (years)
65.5 ± 13.0
Gender
Female
554 (37.5)
Male
922 (62.5)
Education (years)
8.9 ± 2.1
Body mass index (kg/m2)
24.0 ± 2.6
Clinical attributes
Stroke type
Ischemic
1,224 (82.9)
Hemorrhagic
252 (17.1)
Nasogastric tube
No
1,187 (80.4)
Yes
289 (19.6)
Foley catheter
No
1,342 (90.9)
Yes
134 (9.1)
Hypertension
No
449 (30.4)
Yes
1,027 (69.6)
Diabetes mellitus
No
906 (61.4)
Yes
570 (38.6)
Hyperlipidemia
No
967 (65.5)
Yes
509 (34.5)
Atrial fibrillation
No
1,354 (91.7)
Yes
122 (8.3)
Previous stroke
No
1,250 (84.7)
Yes
226 (15.3)
Acute care length of stay (days)
15.2 ± 9.0
Rehabilitation length of stay (days)
44.9 ± 21.2
Readmission in 30 days
No
1,356 (91.9)
Yes
120 (8.1)
Functional status scores before rehabilitation
BI score
39.0 ± 23.7
FOIS score
5.5 ± 2.1
EQ5D score
10.4 ± 1.9
IADL score
1.2 ± 1.1
BBS score
15.6 ± 15.8
MMSE score
19.4 ± 8.9
Data are frequencies (percentages), as indicated, for categorical variables and mean ± standard deviation for continuous variables of baseline characteristics.
SD, standard deviation; BI, Barthel Index; FOIS, Functional Oral Intake Scale; EQ-5D, EuroQoL Quality of Life Scale; IADL, Instrumental activities of Daily Living Scale; BBS, Berg Balance Scale; MMSE, Mini-Mental State Examination.
Comparison of Forecasting Models
Significant predictors of 30-day readmission did not significantly differ between the training and testing datasets; therefore, samples were compared between the training and testing datasets to increase reliability of the validation results (Table 3). We used grid search to find the best hyperparameters for the neural networks. We searched for the following hyperparameters: the number of hidden layers (in the range of 1–6), the number of hidden neurons in each layer (in the range of 1–512), activation functions (“relu,” “logistic sigmoid”), and learning rate (in the range of 0.01–0.001). We used adam optimizer, constant learning rate, and the regularization rate of alpha = 0.01. The SVM model was configured with linear kernel, and regularization parameter C = 1.0. The RF model is an ensemble learning method combined of multiple decision tree predictors that are trained based on random data samples and feature subsets. We configured the RF algorithm with two trees in the forest. Hyperparameter optimization was then performed to improve the performance of the compact model, and the machine learning algorithms with the greatest AUROC values in 1,000 trials were obtained. Table 4 lists the final hyperparameter settings. The data in Table 5 indicate that the ANN model compared to KNN, RF, SVM, NBC, and COX models had significantly (P < 0.001) higher sensitivity, specificity, PPV, NPV, accuracy, and AUC values. Similar results also were shown in dataset for testing simultaneously. The receiver operating characteristic (ROC) curve results in Figure 3 show that the ANN model had significantly higher ROC values compared to other forecasting models (P < 0.001).
Univariate analysis of selected risk factors for 30-day readmission in patients with stroke (N = 1,476).
Variables
Statistics
P-value*
Post-acute care (yes vs. no)
52.074
<0.001
Patient attributes
Age (years)
7.890
0.005
Gender (female vs. male)
23.657
<0.001
Education (years)
10.870
<0.001
Body mass index (kg/m2)
7.944
0.005
Clinical attributes
Stroke type (ischemic vs. hemorrhagic)
32.053
<0.001
Nasogastric tube (yes vs. no)
49.361
<0.001
Foley catheter (yes vs. no)
5.590
0.018
Hypertension (yes vs. no)
4.564
0.033
Diabetes mellitus (yes vs. no)
7.324
0.007
Hyperlipidemia (yes vs. no)
5.777
0.016
Atrial fibrillation (yes vs. no)
6.114
0.013
Previous stroke (yes vs. no)
6.899
0.009
Acute care length of stay, days
30.008
<0.001
Rehabilitation length of stay, days
26.508
<0.001
Functional status score before rehabilitation
BI score
37.494
<0.001
FOIS score
26.508
<0.001
EQ5D score
16.712
<0.001
IADL score
22.726
<0.001
BBS score
14.903
<0.001
MMSE score
34.665
<0.001
One-way analysis of variance and Fisher exact analysis were performed to assess for associations between the variables and 30-day readmission.
BI, Barthel Index; FOIS, Functional Oral Intake Scale; EQ-5D, EuroQoL Quality of Life Scale; IADL, Instrumental Activities of Daily Living Scale; BBS, Berg Balance Scale; MMSE, Mini-Mental State Examination.
Hyper-parameters and final settings in all machine learning algorithms.
Algorithms
Hyper-parameters
Settings
Artificial neural network (ANN)
Hidden layers
6
Hidden neuron
512-256-128-64-32-1
Learning rate*
0.001
K nearest neighbor (KNN)
Neighbors
5
Support vector machine (SVM)
Cpenalty
1.0
Gamma
1/[n_features * X.var()]
Naive Bayes classifier (NBC)
Alpha
1.0
Random forest (RF)
Estimators
100
Splitmin
2
leafmin
1
Cox regression (COX)
–
–
Optimizer algorithm using Adam.
Comparison of 1,000 pairs of forecasting models for predicting 30-day readmission in patients with stroke (N = 1,476).
Model
Sensitivity
Specificity
PPV
NPV
Accuracy
AUC
Training dataset (n=1,033)
ANN (95% CI)
0.73 (0.65, 0.82)
0.98 (0.96, 0.99)
0.88 (0.84, 0.92)
0.77 (0.70, 0.84)
0.92 (0.89, 0.95)
0.94 (0.91,0.97)
KNN (95% CI)
0.59 (0.50, 0.68)
0.86 (0.82, 0.90)
0.56 (0.47, 0.65)
0.64 (0.56, 0.72)
0.83 (0.78, 0.88)
0.76 (0.68, 0.84)
RF (95% CI)
0.70 (0.64, 0.76)
0.92 (0.87, 0.97)
0.79 (0.75, 0.83)
0.71 (0.64, 0.78)
0.88 (0.84, 0.92)
0.85 (0.80, 0.90)
SVM (95% CI)
0.49 (0.39, 0.59)
0.96 (0.93, 0.99)
0.76 (0.68, 0.84)
0.62 (0.54, 0.70)
0.89 (0.85, 0.93)
0.74 (0.66, 0.82)
NBC (95% CI)
0.48 (0.38, 0.59)
0.96 (0.93, 0.99)
0.50 (0.40, 0.60)
0.69 (0.61, 0.77)
0.81 (0.75, 0.87)
0.73 (0.65, 0.81)
COX (95% CI)
0.51 (0.42, 0.61)
0.97 (0.95, 0.99)
0.77 (0.69, 0.85)
0.71 (0.63, 0.79)
0.85 (0.80, 0.90)
0.88 (0.83, 0.93)
P-value*
<0.001
<0.001
<0.001
<0.001
<0.001
<0.001
Testing dataset (n=443)
ANN (95% CI)
0.70 (0.62, 0.78)
0.97 (0.95, 0.99)
0.89 (0.85, 0.93)
0.82 (0.76, 0.88)
0.93 (0.90, 0.96)
0.89 (0.85, 0.93)
KNN (95% CI)
0.53 (0.44, 0.62)
0.88 (0.84, 0.92)
0.60 (0.51, 0.69)
0.71 (0.63, 0.79)
0.71 (0.63, 0.79)
0.81 (0.75, 0.87)
RF (95% CI)
0.69 (0.62, 0.76)
0.94 (0.92, 0.96)
0.85 (0.82, 0.88)
0.79 (0.76, 0.82)
0.88 (0.84, 0.92)
0.87 (0.83, 0.91)
SVM (95% CI)
0.53 (0.44, 0.62)
0.93 (0.90, 0.96)
0.75 (0.67, 0.82)
0.78 (0.71, 0.85)
0.82 (0.73, 0.89)
0.80 (0.74, 0.86)
NBC (95% CI)
0.50 (0.40, 0.60)
0.93 (0.90, 0.96)
0.63 (0.54, 0.72)
0.79 (0.72, 0.86)
0.83 (0.76, 0.90)
0.84 (0.78, 0.90)
COX (95% CI)
0.54 (0.45, 0.64)
0.96 (0.94, 0.98)
0.88 (0.83, 0.93)
0.61 (0.53, 0.69)
0.87 (0.82, 0.92)
0.87 (0.82, 0.92)
P-value*
<0.001
<0.001
<0.001
<0.001
<0.001
<0.001
ANN, artificial neural network; KNN, K nearest neighbor; RF, random forest; SVM, support vector machine; NBC, naive Bayes classifier; COX, Cox regression; PPV, positive predictive value; NPV, negative predictive value; AUC, area under the curve; CI, confidence interval.
The P-value is the statistical significance of the forecasting models and performance indices calculated using a Chi-squared test.
Performance indices of forecasting models used to predict 30-day readmission in patients with stroke when using (A) training dataset, (B) testing dataset. The box plot shows the median (centers) and interquartile range (borders). In analyses of accuracy and AUROC, the ANN model had significantly higher values compared to other forecasting models (P < 0.001). AUROC, area under the receiver operating characteristics; ANN, artificial neural network.
Significant Predictors in the ANN Model
Figure 4 shows the feature importance analysis results for the ANN model. The VSR value for predicting 30-day readmission in stroke patients was highest for PAC (permutation importance = 0.761) followed by NG tube (0.552), stroke type (0.448), BI score before rehabilitation (0.423), IADL score before rehabilitation (0.418), MMSE score before rehabilitation (0.409), BBS score before rehabilitation (0.408), FOIS score before rehabilitation (0.404), EQ5D score before rehabilitation (0.401), and others.
A permutation importance analysis of artificial neural network model in predicting 30-day readmission in patients with stroke. BI, Barthel Index; IADL, Instrumental Activities of Daily Living; MMSE, Mini-Mental State Examination; BBS, Berg Balance Scale; FOIS, Functional Oral Intake Scale; EQ-5D, EuroQoL Quality of Life Scale.
Sensitivity Analysis
Next, the validating dataset of 167 subjects was used to compare the predictive accuracy of the models. Table 6 also compares the performance indices obtained in external validation of the ANN, KNN, RF, SVM, NBC and COX models. For predicting 30-day readmission, the ANN model consistently achieved significantly higher performance indices (P < 0.001).
Comparative performance indices of forecasting models when using 167 new validating datasets to predict 30-day readmission in patients with stroke.
Models
Sensitivity
Specificity
PPV
NPV
Accuracy
AUC
ANN (95% CI)
0.74 (0.66, 0.82)
0.97 (0.95, 0.99)
0.89 (0.85, 0.94)
0.87 (0.82, 0.92)
0.93 (0.90, 0.96)
0.94 (0.91, 0.97)
KNN (95% CI)
0.50 (0.40, 0.49)
0.87 (0.83, 0.91)
0.61 (0.52, 0.70)
0.70 (0.62, 0.78)
0.80 (0.74, 0.86)
0.83 (0.78, 0.88)
RF (95% CI)
0.70 (0.66, 0.74)
0.95 (0.91, 0.98)
0.84 (0.80, 0.88)
0.85 (0.81, 0.89)
0.90 (0.87, 0.93)
0.90 (0.86, 0.94)
SVM (95% CI)
0.51 (0.41, 0.61)
0.96 (0.94, 0.98)
0.76 (0.69, 0.83)
0.79 (0.72, 0.87)
0.88 (0.84, 0.92)
0.81 (0.76, 0.86)
NBC (95% CI)
0.50 (0.40, 0.60)
0.93 (0.90, 0.96)
0.61 (0.52, 0.70)
0.80 (0.73, 0.87)
0.84 (0.79, 0.89)
0.80 (0.75, 0.85)
COX (95% CI)
0.58 (0.49, 0.67)
0.92 (0.89, 0.95)
0.84 (0.78, 0.90)
0.69 (0.61, 0.77)
0.88 (0.84, 0.92)
0.88 (0.84, 0.92)
P-value*
<0.001
<0.001
<0.001
<0.001
<0.001
<0.001
ANN, artificial neural network; KNN, K nearest neighbor; RF, random forest; SVM, support vector machine; NBC, naive Bayes classifier; COX, Cox regression; PPV, positive predictive value; NPV, negative predictive value; AUC, area under the curve; CI, confidence interval.
The P-value is the statistical significance of the forecasting models and the performance indices calculated using a Chi-squared test.
Discussion
Accuracy in predicting 30-day readmission in patients with stroke was compared among five forecasting models. For a given set of clinical inputs, the ANN model clearly had superior forecasting accuracy compared to the other four. Notably, our prospective study collected longitudinal data from six different medical institutions, which provided a real-world depiction of current treatment for patients with stroke. In contrast, previous works have used data from a single medical center (10–13). Moreover, using registry data obtained from six hospitals mitigated the potential for referral bias or bias caused by analyzing the practices of a single physician or a single institution (30, 31).
Recent works have demonstrated the superior performance of machine learning-based models for predicting stroke outcomes (24, 25). One advantage of using an ANN model is that it enables appropriate and accurate processing of inputs that are incomplete or inputs that introduce noise (9, 32). Another advantage of ANN models, whether linear or non-linear, is their good performance in/effectiveness for analyzing large-scale medical databases constructed using data that are highly correlated but not normally distributed. The high robustness of the ANN model has been demonstrated in many clinical applications, particularly predicting prognosis in various diseases (32). In performance comparisons of the five models in this study, expanding the number of potential predictors apparently improved the performance of the ANN model in systematic analysis of outcome in various diseases.
Our current results indicate that ANN models can use clinical outcome data for predicting 30-day readmission after stroke. Prospective prediction performance and cross-validation performance were adequate when subjects were familiar with the task and when information from the previous test session was made available. However, larger scale studies are still needed to validate this approach.
A permutation importance analyses of the weights of significant predictors of 30-day readmission for stroke revealed that the best predictor was PAC. This finding is consistent with earlier reports that, in comparisons of independent predictors, PAC is the best predictor of stroke outcome, including overall treatment cost, functional status after stroke, and duration of hospital stay before transfer to rehabilitative ward (30, 33). In a quasi-experimental study of stroke patients, Wang et al. (30) investigated the longitudinal impact of PAC on functional status. The authors concluded that multidisciplinary rehabilitative PAC delivered on a per-diem basis substantially improved functional status compared to standard rehabilitation. Another study performed in a nationwide stroke cohort compared mortality and numerous functional domains between a PAC group and a well-matched non-PAC group (34). The PAC group had significantly lower 90-day hospital readmissions and stroke-related readmissions compared to the non-PAC group.
Dennis et al. (35) reported that, compared to nasogastric feeding, percutaneous endoscopic gastrostomy was associated with higher risk of death or poorer outcomes at 6 months after stroke. However, Ho et al. (36) noted that prolonged (i.e., longer than 2 weeks) nasogastric tube feeding was significantly associated with pneumonia and mortality. In the current study, NG tube insertion before rehabilitation was significantly associated with 30-day readmission (P < 0.001). During the study period, no patient with stroke required NG tube insertion after rehabilitation.
Compared to other stroke types, hemorrhagic stroke is reportedly associated with higher severity and with higher overall mortality in the first 3 months after stroke (37, 38). The current study further revealed that hemorrhagic stroke has a higher 30-day readmission rate for ischemic stroke.
This prospective observational cohort study of patients with stroke in Taiwan analyzed data from patients treated at six healthcare institutions. The predictive accuracy of the ANN model developed in this study outperformed the other four models in identifying predictors of 30-day readmission. Three implications of this study are noted. First, the proposed ANN model may be useful for guiding the clinical care of patients with stroke. Second, healthcare administrators and managers at medical institutions should facilitate prompt and appropriate PAC for patients with stroke. Third, the Taiwan National Health Insurance Administration should include PAC in its guidelines for clinical treatment of stroke in order to achieve a broad nationwide improvement in care for these patients. However, further studies are needed to confirm the clinical relevance of the proposed ANN model in terms of its efficacy in predicting prognosis and optimizing medical management for patients with stoke.
For further validation of the significant association observed between PAC and 30-day readmission for stroke, Table 7 compares six relevant studies performed in the United States or Taiwan (39–44). The six studies shared the following features: (1) a relatively large sample size, (2) a mean age of 65 years or more, (3) use of statewide or national datasets, and, most importantly, (4) investigation of 30-day readmission in patients with stroke. As in these previous works, out study demonstrated a significantly lower 30-day stroke readmission rate in a multidisciplinary PAC group compared to a non-PAC group (P < 0.001).
Reported associations between post-acute care (PAC) for stroke and 30-day readmission.
Authors (country)
No. of subjects
Mean age
Data source
Findings
Present study (Taiwan)
1,476
65.5
Prospective cohort study from six hospitals
Post-acute care (PAC) program was the best predictor of 30-day readmission
Kim et al. (U.S.) (39)
51,863
80.4
Medicare provider analysis and review files
Using Instrumental Variable analysis to control for endogeneity bias, an increase in institutional PAC use was associated with a decrease in 30-day readmission rate by 0.19 percentage points
Kosar et al. (U.S.) (40)
2,044,231
80.2
Medicare provider analysis and review database
In most rural counties, 30-day readmission rates were 0.3 (95% CI, −0.6 to −0.1) percentage points lower in a non-PAC group compared to a PAC group
Raman et al. (U.S.) (41)
1,613
74.4
State inpatient database, California
Clinical predictors of 30-day readmission included comorbidities (e.g., liver disease, hypertension) and discharge to a PAC facility
Li et al. (U.S.) (42)
7,851,430
65~100
Medicare beneficiaries
An increase in quarterly PAC use was significantly (P <0.001) associated with a decrease in 30-day risk-standardized readmission rates for acute myocardial infarction, heart failure, and hip/femur fracture
Ramchand et al. (U.S.) (43)
4,850
53.1
National readmissions database
It showed that discharge to inpatient postacute care facility (adjusted odds ratio 1.61, 95% CI 1.07–2.41) was significantly associated with a higher likelihood of 30-day readmission after discharge
Hsieh et al. (Taiwan) (44)
6,839
69.4
National Health Insurance claims datasets
The 30-day readmission rates were 15.5% for the PAC group vs. 30.4% in the non-PAC group
This study has several limitations inherent in a large database analysis. First, the validity of the comparisons in the study is limited by the exclusion of complications associated with stroke rehabilitation outcomes. Second, the analysis was limited to 30-day readmission, which reduces the subset of patients with stroke in which the ANN model is clinically applicable. Third, imbalance between positive and negative outcomes, i.e., class imbalance, is a common problem in analysis of medical data and has not been satisfactorily addressed (45, 46). Further studies are needed to investigate the use of ensemble algorithm for solving the class imbalance problem. Additionally, whether the timing or duration of the stroke treatment is a relevant prognostic predictor of readmission deserves further study. Nevertheless, the results can still be considered valid given the robustness and statistical significance of the results.
Conclusions
Based on the comparison results in this study, we conclude that the ANN model is superior to the other forecasting models in terms of accuracy in predicting 30-day readmission for stroke after a hospital discharge. The ANN model outperformed the other models in terms of both accuracy and AUROC curve. Using a machine-learning ANN model to obtain an accurate estimate of 30-day readmission for stroke and to identify risk factors may improve the precision and efficacy of management for these patients. Predictors of stroke can be discussed when educating PAC candidates in the expected course of recovery and health outcomes. Although the practical applicability of database studies such as this have been convincingly demonstrated in the literature, future studies can expand the range of clinical variables included in the analysis, which could obtain additional results and potentially improve prediction accuracy. Such data could be vital for developing, promoting, and improving health policies for treating patients with stroke.
Data Availability Statement
The raw data supporting the conclusions of this article will be made available by the authors, without undue reservation.
Ethics Statement
The study protocol was approved by the Institutional Review Board at Kaohsiung Medical University Hospital (KMUH-IRB-20140308) and written informed consent was obtained from each participant.
Author Contributions
Y-CC and H-YS: conceptualization, data curation, formal analysis, investigation, methodology, resources, software, validation, visualization, writing—original draft, and writing—review and editing. J-HC, Y-JY, S-JL, H-FL, C-HL, H-HH, K-WH, and S-CY: data curation, formal analysis, investigation, methodology, resources, software, validation, and visualization. All authors contributed to the article and approved the submitted version.
Funding
This study was supported by funding from the NSYSU-KMU JOINT RESEARCH PROJECT (NSYSU-KMU 109-P001 and 110-P017), NPUST-KMU JOINT RESEARCH PROJECT (NPUST-KMU 109-P010, 110-P001, and 111-P010), and the Ministry of Science and Technology (MOST 104-2410-H-037-006-SS2, MOST 106-2410-B-037-076, and MOST 108-2410-H-037-006-SS3) in Taiwan. The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.
Conflict of Interest
The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.
Publisher's Note
All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.
ReferencesOttenbacherKJKarmarkarAGrahamJEKuoYFDeutschAReistetterTA. Thirty-day hospital readmission following discharge from postacute rehabilitation in fee-for-service medicare patients. JAMA. (2014) 311:604–14. 10.1001/jama.2014.824519300NouhAMMcCormickLModakJFortunatoGStaffI. High mortality among 30-day readmission after stroke: predictors and etiologies of readmission. Front Neurol. (2017) 8:632. 10.3389/fneur.2017.0063229270149FehnelCRLeeYWendellLCThompsonBBPotterNSMorV. Post-acute care data for predicting readmission after ischemic stroke: a nationwide cohort analysis using the minimum data set. J Am Heart Assoc. (2015) 4:e002145. 10.1161/JAHA.115.00214526396202LiottaEMSinghMKostevaARBeaumontJLGuthJCBauerRM. Predictors of 30 day readmission after intracerebral hemorrhage: a single-center approach for identifying potentially modifiable associations with readmission. Crit Care Med. (2013) 41:2762–9. 10.1097/CCM.0b013e318298a10f23963121AminADeitelzweigSBuciorILinJLingohr-SmithMMengesB. Frequency of hospital readmissions for venous thromboembolism and associated hospital costs and length of stay among acute medically ill patients in the US. J Med Econ. (2019) 22:1119–25. 10.1080/13696998.2019.161886231084383LinebackCMGargROhENaidechAMHollJLPrabhakaranS. Prediction of 30-day readmission after stroke using machine learning and natural language processing. Front Neurol. (2021) 12:649521. 10.3389/fneur.2021.64952134326805DarabiNHosseinichimehNNotoAZandRAbediV. Machine learning-enabled 30-day readmission model for stroke patients. Front Neurol. (2021) 12:638267. 10.3389/fneur.2021.63826733868147XuYYangXHuangHPengCGeYWuH. Extreme gradient boosting model has a better performance in predicting the risk of 90-day readmissions in patients with ischaemic stroke. J Stroke Cerebrovasc Dis. (2019) 28:104441. 10.1016/j.jstrokecerebrovasdis.2019.10444131627995SarajlicPSimonssonMJernbergTBäckMHofmannR. Incidence, associated outcomes, and predictors of upper gastrointestinal bleeding following acute myocardial infarction: A SWEDEHEART-based nationwide cohort study. Eur Heart J Cardiovasc Pharmacother. (2021) pvab059. 10.1093/ehjcvp/pvab05934423350ZhongWGengNWangPLiZCaoL. Prevalence, causes and risk factors of hospital readmissions after acute stroke and transient ischemic attack: a systematic review and meta-analysis. Neurol Sci. (2016) 37:1195–202. 10.1007/s10072-016-2570-527129874WenTLiuBWanXZhangXZhangJZhouX. Risk factors associated with 31-day unplanned readmission in 50,912 discharged patients after stroke in China. BMC Neurol. (2018) 18:218. 10.1186/s12883-018-1209-y30587162LinHJChangWLTsengMC. Readmission after stroke in a hospital-based registry: risk, etiologies, and risk factors. Neurology. (2011) 76:438–43. 10.1212/WNL.0b013e31820a0cd821209374SharmaVKulkarniVMcAlisterFEurichDKeshwaniSSimpsonSH. Predicting 30-day readmissions in patients with heart failure using administrative data: a machine learning approach. J Card Fail. (2022) 28:710–22. 10.1016/j.cardfail.2021.12.00434936894WangZChenXTanXYangLKannapurKVincentJV. Using deep learning to identify high-risk patients with heart failure with reduced ejection fraction. J Health Econ Outcomes Res. (2021) 8:6–13. 10.36469/jheor.2021.2575334414250AmritphaleAChatterjeeRChatterjeeSAmritphaleNRahnavardAAwanGM. Predictors of 30-day unplanned readmission after carotid artery stenting using artificial intelligence. Adv Ther. (2021) 38:2954–72. 10.1007/s12325-021-01709-733834355QuinnTJMcArthurKDawsonJWaltersMRLeesKR. Reliability of structured modified Rankin scale assessment. Stroke. (2010) 41:e602. 10.1161/STROKEAHA.110.59054721030706LeungSOChanCCShahS. Development of a Chinese version of the Modified Barthel Index– validity and reliability. Clin Rehabil. (2007) 21:912–22. 10.1177/026921550707728617981850CraryMAMannGDGroherME. Initial psychometric assessment of a functional oral intake scale for dysphagia in stroke patients. Arch Phys Med Rehabil. (2005) 86:1516–20. 10.1016/j.apmr.2004.11.04916084801AgrellBDehlinO. Mini mental state examination in geriatric stroke patients. Validity, differences between subgroups of patients, and relationships to somatic and mental variables. Aging. (2000) 12:439–44. 10.1007/BF0333987411211953VittenglJRWhiteCNMcGovernRJMortonBJ. Comparative validity of seven scoring systems for the instrumental activities of daily living scale in rural elders. Aging Ment Health. (2006) 10:40–7. 10.1080/1360786050030794416338813RabinRde CharroF. EQ-5D: a measure of health status from the EuroQol group. Ann Med. (2001) 33:337–43. 10.3109/0785389010900208711491192ChouCYChienCWHsuehIPSheuCFWangCHHsiehCL. Developing a short form of the Berg Balance Scale for people with stroke. Phys Ther. (2006) 86:195–204. 10.1093/ptj/86.2.19516445333WangCYMiyoshiSChenCHLeeKCChangLCChungJH. Walking ability and functional status after post-acute care for stroke rehabilitation in different age groups: a prospective study based on propensity score matching. Aging. (2020) 12:10704–14. 10.18632/aging.10328832482912ParkSJHussainIHongSKimDParkHBenjaminHCM. Real-time gait monitoring system for consumer stroke prediction service. 2020 IEEE Int Conf Consum Electron. (2020) 2020:1–4. 10.1109/ICCE46568.2020.904309827295638HussainIParkSJ. Big-ECG: Cardiographic predictive cyber-physical system for stroke management. IEEE Access. (2021) 9:123146–64. 10.1109/ACCESS.2021.3109806KamogashiraTFujimotoCKinoshitaMKikkawaYYamasobaTIwasakiS. Prediction of vestibular dysfunction by applying machine learning algorithms to postural instability. Front Neurol. (2020) 11:7. 10.3389/fneur.2020.0000732116997LiLJamiesonKDesalvoGRostamizadehATalwalkarA. Hyperband: A novel bandit-based approach to hyperparameter optimization. J Mach Learn Res. (2017) 18:6765–816. 10.48550/arXiv.1603.06560WongTT. Performance evaluation of classification algorithms by k-fold and leave-one-out cross validation. Pattern Recogn. (2015) 48:2839–46. 10.1016/j.patcog.2015.03.009AltmannAToloşiLSanderOLengauerT. Permutation importance: a corrected feature importance measure. Bioinformatics. (2010) 26:1340–7. 10.1093/bioinformatics/btq13420385727WangCYHsienHHHungKWLinHFChiouHYYehSCJ. Multidiscipline stroke post-acute care transfer system: propensity-score-based comparison of functional status. J Clin Med. (2019) 8:1233. 10.3390/jcm808123331426354JahanRSaverJLSchwammLHFonarowGCLiangLMatsouakaRA. Association between time to treatment with endovascular reperfusion therapy and outcomes in patients with acute ischemic stroke treated in clinical practice. JAMA. (2019) 322:252–63. 10.1001/jama.2019.828631310296BaxtWG. Application of artificial neural networks to clinical medicine. Lancet. (1995) 346:1135–8. 10.1016/S0140-6736(95)91804-37475607WangCYChenYRHongJPChanCCChangLCShiHY. Rehabilitative post-acute care for stroke patients delivered by per-diem payment system in different hospitalization paths: a Taiwan pilot study. Int J Qual Health Care. (2017) 29:779–84. 10.1093/intqhc/mzx10229025039PengLNLuWHLiangCKChouMYChungCPTsaiSL. Functional outcomes, subsequent healthcare utilization, and mortality of stroke postacute care patients in Taiwan: a nationwide propensity score-matched study. J Am Med Dir Assoc. (2017) 18:990.e7–12. 10.1016/j.jamda.2017.06.02028804011DennisMSLewisSCWarlowC. FOOD Trial Collaboration. Effect of timing and method of enteral tube feeding for dysphagic stroke patients (FOOD): a multicentre randomised controlled trial. Lancet. (2005) 365:764–72. 10.1016/S0140-6736(05)17983-515733717HoCHLinWCHsuYFLeeIHHungYC. One-year risk of pneumonia and mortality in patients with poststroke dysphagia: a nationwide population-based study. J Stroke Cerebrovasc Dis. (2018) 27:1311–7. 10.1016/j.jstrokecerebrovasdis.2017.12.01729395642AndersenKKOlsenTSDehlendorffCLeeIHHungYC. Hemorrhagic and ischemic strokes compared: stroke severity, mortality, and risk factors. Stroke. (2009) 40:2068–72. 10.1161/STROKEAHA.108.54011219359645RothGADwyer-LindgrenLBertozzi-VillaAStubbsRWMorozoffCNaghaviM. Trends and patterns of geographic variation in cardiovascular mortality among us counties, 1980-2014. JAMA. (2017) 317:1976–92. 10.1001/jama.2017.415028510678KimYThirukumaranCTemkin-GreenerHHillEHollowayRLiY. Institutional postacute care use may help reduce readmissions for ischemic stroke patients. Med Care. (2021) 59:736–42. 10.1097/MLR.000000000000156833999571KosarCMLoomerLFerdowsNBTrivediANPanagiotouOARahmanM. Assessment of rural-urban differences in postacute care utilization and outcomes among older US adults. JAMA Netw Open. (2020) 3:e1918738. 10.1001/jamanetworkopen.2019.1873831913495RamanNAl-RobaidiKJadhavAThirumalaPD. Perioperative stroke and readmissions rates in noncardiac non-neurologic surgery. J Stroke Cerebrovasc Dis. (2020) 29:104792. 10.1016/j.jstrokecerebrovasdis.2020.10479232280000LiCYKarmarkarALinYLKuoYFOttenbacherKJ. Hospital readmissions reduction program and post-acute care: implications for service delivery and 30-day hospital readmission. J Am Med Dir Assoc. (2020) 21:1504–8. 10.1016/j.jamda.2020.05.01832660855RamchandPThibaultDPCrispoJALevineJHurstRMullenMT. Readmissions after mechanical thrombectomy for acute ischemic stroke in the united states: a nationwide analysis. J Stroke Cerebrovasc Dis. (2018) 27:2632–40. 10.1016/j.jstrokecerebrovasdis.2018.05.03530042034HsiehCYTsaoWCLinRTChaoAC. Three years of the nationwide post-acute stroke care program in Taiwan. J Chin Med Assoc. (2018) 81:87–8. 10.1016/j.jcma.2017.09.00329033109Soto-VarelaARossi-IzquierdoMDel-Río-ValeirasMFaraldo-GarcíaAVaamonde-Sánchez-AndradeILirola-DelgadoA. Modified timed up and go test for tendency to fall and balance assessment in elderly patients with gait instability. Front Neurol. (2020) 11:543. 10.3389/fneur.2020.0054332595593WuJTangY. Revisiting the immune balance theory: a neurological insight into the epidemic of COVID-19 and its alike. Front Neurol. (2020) 11:566680. 10.3389/fneur.2020.56668033178109