A total of 54 patient scans were acquired from a cohort registry of patients with confirmed AIS of the MCA territory presenting to the Mayo Clinic, Rochester, United States, between July 2015 and February 2020. Patient eligibility for endovascular therapy was determined per the institutional protocol at the time of presentation. Cases not meeting the study criteria were not included. The diagnosis was confirmed by the treating team with reference to clinical and comprehensive imaging data (including CT angiography to confirm the vascular territory of the stroke).

Serial patients meeting the following criteria were included in this study: AIS affecting the MCA territory; eligible for acute endovascular reperfusion therapy; no evidence of intracranial hemorrhage; imaging data of adequate quality (e.g., free from excessive motion leading to major artifacts) with associated demographic and follow-up imaging available.

All imaging was acquired on a SIEMENS scanner (SOMATOM Definition Flash 39/Edge 15), with NCCT imaging available in appropriate reconstructions. For reader purposes, the presented data were 3–5 mm reconstructions to maximize the pathological contrast to noise ratio.

A consensus of three board-certified neuroradiologists (for whom ASPECTS scoring on the NCCT is part of their clinical practice) was used as the reference standard for analysis (9, 11). Each expert was given a demonstration of how to use the scoring platform and provided with training material. The three neuroradiologists independently scored each of the 54 CT scans with reference to the clinical information provided (including laterality and severity of symptoms and treatment success) and follow-up clinical imaging. To enhance the accuracy of the ground truth, the truthers were given additional clinical information and follow-up imaging to facilitate their reads. The ground truth was established by the consensus of the three expert neuroradiologist readers for each region scored, and when there was no complete consensus, the region was attributed to the status of most readers.

The 10 clinician readers comprised representative intended e-ASPECTS users from the United States, including neurologists and neuroradiologists, all of whom interpret NCCT scans as part of their clinical routine. The readers consisted of four neurologists and six neuroradiologists, all with US board certification. The time post-board certification varied from <1 year to 22 years, with a median of 6 years. All readers used the same viewing platform, with and without e-ASPECTS support.

Each reader scored each ASPECTS region of every scan in two sessions, at least 4 weeks apart. The readers were given only the acute NCCT image for their scoring. In the first session, half of the cases were selected at random to be presented with e-ASPECTS decision support. The other half of the cases were presented without e-ASPECTS decision support. In the second session, the decision support allocation was reversed.

The reader indicated which hemisphere was affected, and within the hemisphere, which of the ASPECTS regions was affected. Scans scored with reference to the e-ASPECTS overlay were presented alongside the non-overlayed scan using the standard e-ASPECTS interface, which is the same format as that output to a standard hospital imaging platform (PACS). Scans read without e-ASPECTS decision support were presented without reference to the e-ASPECTS region segmentation framework to best represent standard clinical practice.

The primary endpoint was the diagnostic accuracy of each rater compared to the reference standard (consensus read defined by the ground truthers), quantified using the area under the curve (AUC) of the receiver operating characteristic (ROC). This provides a composite measure of the impact on both sensitivity and specificity. The sample size calculation was derived based on the unified Obuchowski and Rockette–Dorman, Berbaum, and Metz (OR–DBM) analysis methods devised by Hillis, Obuchowski, and Berbaum (18). The sample size was powered to detect a difference in AUC of 0.095 (0.1 × 0.95). With an alpha of 0.05, the sample size calculation indicated that 54 patient cases and 10 readers would allow a power of 80%.

Secondary endpoints included sensitivity, specificity, overall percentage agreement (accuracy), and Cohen's Kappa statistic. Bland–Altman analysis was used to evaluate the impact on magnitude of the variation in agreement with the reference standard. Bias was defined as the mean difference between the reader's score with and without e-ASPECTS. An analysis of agreement at the patient level was described using a weighted Kappa statistic. Statistics were reported with reference to upper and lower 95% confidence interval limits. Statistical significance was determined by a p < 0.05.

A total of 54 representative NCCT brain scans were included in this study. Demographic data for the patients included are described in Table 1. The median age was 72 years, and the median time of imaging from the last known well was 3 h 19 min, reflective of the intended use of ASPECTS in patients presenting within the time window for reperfusion therapies (4).

Imaging characteristics were described using the consensus of the ground truthers, including total ASPECTS and the presence of commonly encountered co-existing imaging findings. The distribution of ASPECTS (Supplementary Table S1) observed in the unenriched study cohort is reflective of the population for whom use is intended. The observed median ASPECTS of 9 is consistent with the figures reported within a large meta-analysis of patients undergoing assessment of MCA territory AIS (19).

In all, 20 (37%) patients were noted to have incidental chronic white matter disease, and 12 (22%) patients had non-acute incidental infarcts visible on the CT scan. The prevalence of these coincident findings is comparable to those seen in cohorts described in the stroke literature (20). One patient was noted to have an incidental meningioma, which did not impact the image processing.

Comparison of the AUC for each reader with and without the support of the e-ASPECTS tool showed an improvement of 0.02 from 0.81 (95% confidence interval [CI] 0.76–0.86) to 0.83 (95% CI 0.79–0.87; difference = 0.02, 95% CI 0.00–0.04; p = 0.028) (Table 2).

When comparing reader performance with the ground truth, the improved AUC in the primary endpoint was driven by an increase in both sensitivity and specificity when assisted by e-ASPECTS compared to unassisted reading (see Table 2 for quantitative results). Overall percentage agreement (accuracy) also improved. Agreement of ASPECTS scoring with the reference standard was improved with e-ASPECTS compared to unassisted reading of scans: Cohen's kappa improved significantly from 0.60 to 0.65 (p = 0.013) and the case-based weighted Kappa improved significantly from 0.70 to 0.81 (p < 0.001) (Table 2).

The improvement in the overall level of accuracy is demonstrated in the Bland–Altmann plot (Figure 1), showing a reduction in the limits of agreement of total ASPECTS and a reduction in bias (Supplementary Table S2). A subgroup analysis based on the clinical training of the reader (radiologist vs. neurologist) demonstrates a consistent impact of e-ASPECTS across reader groups (Supplementary Table S3).

Analysis of individual reader ROC curves showed a greater magnitude of increases in AUC observed in users with lower initial performances and smaller changes in readers with higher unassisted performance. The range in AUC between users was also narrower with e-ASPECTS than unassisted (Supplementary Figure S1 and Figure 2), indicating a reduction in the variation of performance between different readers when e-ASPECTS outputs are available. Similarly, the Bland–Altman plots for individual readers also show greater consistency with the reference standard with e-ASPECTS at an individual level (Supplementary Figure S2).

To further explore the generalizability of the impact of e-ASPECTS, subgroup analyses were undertaken to investigate the performance of e-ASPECTS in deep vs. cortical regions in patients with high vs. low ASPECTS. The use of e-ASPECTS improved reader performance in both the deep (lentiform, caudate, and internal capsule) and cortical (insula and M1–M6) regions (Supplementary Table S4).

Furthermore, the benefit of e-ASPECTS was shown to be consistent across subgroups with low ASPECTS (≤ 6; N = 12) and high ASPECTS (>6; N = 42; see Supplementary Table S5).