The protocol of this systematic review was prospectively registered in PROSPERO (Registration Number: CRD42021233211).

We searched the following databases and registries from their inception to January 2021: PubMed, Cochrane Library, Embase, ClinicalTrials.gov, Chinese Biomedical Literature Service System (SinoMed), China National Knowledge Infrastructure (CNKI), Chinese Scientific Journals Database (VIP), WanFang database, and Chinese Clinical Trial Register (ChiCTR). The detailed search strategies for all databases are presented in Supplementary Table S1.

On 14 November 2021, we updated the search using the same search strategies.

We excluded trials with the following features, in which: 1) other TCM treatments were applied in either the intervention or control group. 2) Outcomes were solely biochemical (for example, inflammatory markers) and not patient-centric (for example, the functional independence or NDS). 3) Outcomes were solely composite (for example, total effective rate) and not original (for example, NDS). 4) Outcomes were unclear (for example, NDS without definite evaluation criteria). 5) The course of treatment was less than 10 days or unclear. 6) The sample size was less than 60. 7) The language was not Chinese or English. 8) The full text was not available.

Two reviewers (LDW and XMF) independently performed literature selection according to the predefined eligibility criteria. The records retrieved in all databases were imported into NoteExpress 3.2, and the duplicate records were deleted. The records were first screened based on the title and abstract, and in cases of uncertainty, the full texts were obtained. Any disagreement between the paired reviewers was resolved by discussing with a third reviewer (WS).

Two reviewers (LDW and YFC) independently extracted data from each trial using a predetermined data extraction form and then cross-checked data. Discrepancies were solved by discussion between the two reviewers or arbitrated by a third reviewer (WS).

We extracted the following data: 1) publication information (authors, country, and year of publication); 2) study designs (methods of randomization, allocation concealment, and blinding); 3) participant baseline characteristics and sample size; 4) details of intervention and control groups; and 5) outcomes (dichotomous data were number of events and total subjects per group; continuous data were mean, standard deviation, and total subjects per group). In case of missing data or unclear information, we contacted the original authors to clarify the information.

Two reviewers (LDW and XMF) independently assessed the risk of bias of the included trials. We used the Cochrane Risk of Bias Tool 2.0 (Sterne et al., 2019) to evaluate the following five domains: randomization process, deviations from intended interventions, missing outcome data, outcome measurements, and selective reporting. Each domain was judged as either “low risk of bias”, “some concerns”, or “high risk of bias”. If disagreements on the judgment were identified, a third reviewer (WS) was consulted.

Review Manager (RevMan 5.4) software was utilized to perform data analyses. Risk ratio (RR) was used for dichotomous data, while weighted mean difference (WMD) or standardized mean difference (SMD) was used for continuous variables, and all of which were demonstrated with effect size and 95% confidence intervals (CIs). A fixed-effects model was selected when no significant heterogeneity was identified (p ≥ 0.10, or I ² ≤ 50%). Otherwise, a random-effects model was applied. We performed subgroup analyses based on the course of treatment and the initiation time of XNJ. When the heterogeneity was substantial (p < 0.10, or I ² > 50%), sources of heterogeneity would be fully explored given the data were accurate.

We further performed sensitivity analyses based on methodological quality and forest plots. After removing different trials in turn, we successively re-examined the meta-analysis results of the remaining trials to assess whether the statistical difference and heterogeneity change. If the findings changed evidently, the full texts of these trials would be checked, and we would interpret the results carefully.

To detect publication bias, we planned to generate funnel plots for meta-analyses including at least ten trials.

Two reviewers (LDW and YFC) independently assessed the certainty of the evidence using the Grading of Recommendations Assessment, Development and Evaluation (GRADE) approach (Balshem et al., 2011), and the evidence was classified as “high,” “moderate,” “low,” or “very low”. The certainty can be downgraded for five limitations (risk of bias, consistency of effect, imprecision, indirectness, and publication bias) and upgraded for three reasons (large magnitude of an effect, dose–response gradient, and effect of plausible residual confounding).

The search yielded 1961 records. There were 1,234 duplicates, leaving 727 to be screened by title and abstract, from which 69 eligible records were retained for full-text evaluation. After careful evaluation, 30 reports were excluded. Ultimately, 38 trials met our inclusion criteria (Wu, 2009; Guan, 2011; Chen et al., 2012; Li et al., 2012; Tong and Zhu, 2012; Zhang et al., 2012; Dong and Fu, 2013; Qian and Jia, 2013; Luo, 2014; Wei and Cheng, 2014; Chen and Wu, 2015; Yang and Li, 2015; Lu et al., 2016; Wei et al., 2016; Zhang and Ai, 2016; Feng et al., 2017; Ji et al., 2017; Shen et al., 2017; Wang et al., 2017; Wang and Lu, 2017; Wu et al., 2017; Zhao, 2017; Fu, 2018; Guo, 2018; Lin et al., 2018; Ma, 2018; Xiao et al., 2018; Yin and Liu, 2018; Liu C et al., 2019; Zheng et al., 2019; Liang et al., 2020; Liu, 2020; Wu and Xu, 2020; Zhang, 2020; Zhao, 2020; Chen, 2021; Dong et al., 2021; He, 2021). Figure 1 shows details of trial selection. A list of 30 trials that appeared to meet the inclusion criteria but excluded is reported in Supplementary Table S2 along with citation and reasons for exclusion.

Thirty-eight RCTs involving 3,677 participants were included in this review, which included a total of 1841 patients in the intervention group and 1836 in the control group. Sample sizes of trials published from 2009 to 2021 ranged from 60 to 208. All trials were conducted in China and reported in Chinese. All interventions were XNJ in combination with CTs. Eight trials (Wei and Cheng, 2014; Zhao, 2017; Guo, 2018; Xiao et al., 2018; Zheng et al., 2019; Zhang, 2020; Chen, 2021; Dong et al., 2021) initiated XNJ treatment within 6 hours of stroke onset, and thirteen trials (Li et al., 2012; Zhang et al., 2012; Qian and Jia, 2013; Wei and Cheng, 2014; Lu et al., 2016; Zhang and Ai, 2016; Wang and Lu, 2017; Wu et al., 2017; Ma, 2018; Yin and Liu, 2018; Liang et al., 2020; Liu, 2020; He, 2021) initiated XNJ treatment within almost 1 day of stroke onset.

The proportion of functional independence at 14 days was reported by one trial (Ji et al., 2017). NDS was reported by all trials. Among them, 30 trials (Chen et al., 2012; Zhang et al., 2012; Qian and Jia, 2013; Wei and Cheng, 2014; Chen and Wu, 2015; Yang and Li, 2015; Lu et al., 2016; Zhang and Ai, 2016; Feng et al., 2017; Ji et al., 2017; Shen et al., 2017; Wang et al., 2017; Wang and Lu, 2017; Wu et al., 2017; Zhao, 2017; Guo, 2018; Lin et al., 2018; Ma, 2018; Xiao et al., 2018; Yin and Liu, 2018; Liu C et al., 2019; Zheng et al., 2019; Liang et al., 2020; Liu, 2020; Wu and Xu, 2020; Zhang, 2020; Zhao, 2020; Chen, 2021; Dong et al., 2021; He, 2021) adopted NIHSS, seven trials (Wu, 2009; Li et al., 2012; Tong and Zhu, 2012; Dong and Fu, 2013; Luo, 2014; Wei et al., 2016; Fu, 2018) adopted CSS, and four trials (Guan, 2011; Wei et al., 2016; Zheng et al., 2019; Liu, 2020) adopted ESS. The composition, source, and chemical characteristics of XNJ used in the included trials are presented in Supplementary Table S3. The other details are shown in Table 1.

We summarized risk of bias of the included trials in Figure 2.

Twenty trials reported adverse events. Among them, nine studies (Li et al., 2012; Tong and Zhu, 2012; Luo, 2014; Wei and Cheng, 2014; Yang and Li, 2015; Wang and Lu, 2017; Wu et al., 2017; Zhao, 2017; Xiao et al., 2018) reported no serious adverse events in either group, and the other trials reported adverse events in both groups including gastrointestinal reactions, skin rashes, abnormal liver function, transient dizziness, slight headache, arrhythmia, dyspnea, gingival bleeding, and gastrointestinal bleeding. No participants discontinued the trial drug due to adverse events.

The 14-day NIHSS of 24 trials was evaluated by the funnel plot, and the left–right asymmetry might be related to the Chinese publication of all included trials and unpublished negative results. As shown in Figure 11, the publication bias should be suspected to some extent.

Sensitivity analysis was conducted for all outcomes except for the functional independence rate. We omitted trials one-by-one in order to observe the meta-analysis result of the remaining trials. None of these exclusions altered the statistical significance of results, which indicated the robustness of our results. However, we found significant change in heterogeneity of CSS, IL-6, and MMP-9.

We used the GRADE approach to assess the quality of evidence for ten outcomes, and this ranged from “very low” to “low” with poor methodology, substantial heterogeneity, and publication bias. The certainty of evidence is summarized in Table 2.

In this review, we evaluated the efficacy and safety of XNJ as an emergency treatment for patients with AIS. We performed comprehensive literature search and identified 38 RCTs (3,677 participants). Compared with CTs alone, XNJ plus CTs was more effective for AIS in increasing the proportion of patients with independent function at 14 days (just one small sample trail), improving neurological function and restoring ability to perform daily activities.

Laboratory results showed positive effects of XNJ in improving IL-6, TNF-α, hs-CRP, and MMP-9. As a key cytokine in inflammatory response, IL-6 was found to be an independent risk predictor for AIS patients (Li et al., 2019), which may possibly be a new target in the prevention of short-term AIS death (Reiche et al., 2019). TNF-α, which plays an important role in the pathogenesis and the process of AIS, was also suggested to be a promising therapeutic target for the treatment of AIS (Wu et al., 2019). The anti-TNF therapy was found to be a feasible way to combat stroke disease via anti-inflammatory and metabolic mechanisms (Lin et al., 2021). Some studies (Huang et al., 2012; Vangilder et al., 2014) indicated that a high hs-CRP level was closely related to unfavorable long-term functional outcome and high rate of all-cause death 3 months after stroke. In addition, an elevated serum MMP-9 level in the acute phase of ischemic stroke was associated with increased risk of mortality and major disability, suggesting that serum MMP-9 could be an important prognostic factor for AIS (Zhong et al., 2017). Therefore, these outcomes are expected to be used as markers to predict the prognosis of AIS.

For safety evaluation, the results showed that the incidence of adverse reactions in the XNJ group was lower than that in the CT group. Nevertheless, it was worthy to note that both groups recorded slight gastrointestinal bleeding and arrhythmia. Although no difference in the incidence rate of these adverse reactions was observed between the two groups, it, nevertheless, suggested that close attention should be paid to the coagulation function and arrhythmia of patients after administering XNJ.

The trials we included had certain limitations. First, all participants were from China only, and no data were available from other countries. Second, with regard to interventions, we were unable to objectively evaluate the efficacy of XNJ combined with or without the first-line emergency treatments as few trials described the situation of IVT and EVTs in patients. Third, with regard to outcomes, only one trial reported the proportion of functional independence, and the dependency was assessed only 2 weeks after the therapies were initiated. Thus, there was no sufficient evidence in improving the long-term function outcome. Finally, and most importantly, key methodological issues were present, including poor randomization procedures, unblinded design, and lack of necessary follow-up. None of the included trials reported allocation concealment or blinding, and bias in selection, performance, and detection reduced the reliability of the findings.

In conducting this review, we have come across certain limitations that might undermine the conclusiveness of the aforementioned findings. One of the major limitations was the substantial heterogeneity in NDS and biochemical outcomes, which affected the credibility of results. We speculated that the heterogeneity may have been resulted from the following factors:

1) Measurement of biochemical outcomes, with large fluctuation range, was easily influenced by different factors.

When it comes to outcomes, most trials reported the total effective rate as a primary outcome, which we have chosen to exclude due to large uncertainty and non-standard evaluation of composite outcomes (Freemantle et al., 2003; Zhang Y Y et al., 2020). In spite of only one trial reporting the functional independence rate, we still used it as the primary outcome. The mRS has evolved as the primary outcome measure for acute stroke trials, and its application has demarcated effective and ineffective acute stroke therapies (Broderick et al., 2017; Powers, 2020). However, we excluded trials that used the entire ordinal distribution of the mRS as we contended that mRS measured by continuous statistical approach could affect the efficacy evaluation of overall functional independence due to a few severe cases. The incidences of death, hemorrhage, and other severe events should also be evaluated. In addition, the interventions included were different from the actual medical environment, which reduced the external validity.

As a consequence of the abovementioned problems, we put forward the following suggestions to future researchers.

1) For the exploration of heterogeneity in TCM trials, proper recording of the subjects’ syndrome (证型) will allow better explanation of heterogeneity. In the TCM theory, syndrome is an indispensable part of “syndrome differentiation and treatment (辨证论治)”.

Future research should be designed based on rigorous methodology, including allocation concealment, blinding, appropriate sample sizes, and longer follow-up assessment. Trial protocol should be registered on the website in advance, and the results should be reported according to the guidelines of SPIRIT-TCM Extension 2018 (Dai et al., 2019) and CONSORT-CHM Formulas 2017 (Cheng et al., 2017). We will be monitoring closely and updating this systematic review over time as high-quality evidence emerges.