A literature review was conducted using the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guidelines. PubMed, Embase, Scopus, Web of Science, and the first 100 results of Google Scholar were evaluated up to 22 April 2020. The search was limited to publications after 2000. Terms queried consisted of variations of resting-state fMRI, epilepsy, seizure, irritative zone, network mapping, lateral, local, and connectivity network, resulting in n = 14,382. Two additional resources were added by hand searching. The articles were loaded into Covidence (Veritas Health Innovation),¹ an online software designed to streamline literature reviews for meta-analyses. After duplicates were removed, 9,550 articles remained. The abstracts and full-text articles were each screened by two of ten independent reviewers, and discrepancies were reviewed by one of two independent reviewers blinded to the original votes as a tiebreaker. Within Covidence, each article was then randomly assigned to the independent interviewers.

Study inclusion criteria consisted of (1) primary research, (2) involving human patients with DRE, (3) patients who underwent rs-fMRI, and (4) the rs-fMRI results were compared to another SOZ-localizing modality. Exclusion criteria were non-peer-reviewed publication abstracts with n < 10 (if published then any number of subjects were allowed), studies written in a language other than English, unavailable subject level SOZ data, duplicate populations with another included article (with exception as detailed in the results that also have novel population), or studies that did not focus on localizing the SOZ.

Each study was analyzed for SOZ localization by a prior non-rs-fMRI established modality, SOZ localization by rs-fMRI, and surgical outcomes, if applicable. The comparative modalities include anatomical MRI, EEG, EEG-fMRI, MEG, iEEG, and surgical outcomes. When a study used multiple comparative modalities, only one primary modality was selected for analysis. The selected modality had to meet the following criteria: (1) It provided usable results for all study participants, and (2) it was described by the authors as the primary modality that influenced the surgical target location. In cases where no primary modality was specified, the comparative modalities were chosen in the following category order: (1) surgical outcomes, (2) iEEG, (3) EEG-fMRI, EEG, MEG, or anatomical MRI. In the case of the third category, the modality with the most detailed subject level data and the most specific brain location information was selected. Applying this process returned the following distribution of primary comparative modalities: iEEG/EEG (193 cases, 43%), surgical outcomes (154 cases, 34%), and anatomical MRI (80 cases, 18%).

True positives (TP) were defined as the SOZ, and rs-fMRI and comparative modality localized to the same place. False positives (FP) were defined as when the rs-fMRI localized a SOZ where a comparative modality did not detect a SOZ. False negatives (FN) were defined as when the comparative modality localized to a different location OR the comparative modality localized a SOZ while rs-fMRI did not detect any. True negatives (TN) were defined as when both rs-fMRI and the comparative modality did not localize a SOZ. For each study, the counts of the TP, FP, FN, and TN were extracted and tabulated into 2×2 tables, with rs-fMRI as the rows and the comparative as the columns (Table 1).

If surgical outcomes were available, which is considered the highest level of proof of validity of SOZ, then SOZ TP, FP, TN, and FN were defined according to Boerwinkle et al. (2017) as summarized in Table 2.

The Quality Assessment of Diagnostic Studies (QUADAS-2) checklist was used to assess the quality of included articles (Whiting et al., 2011). A 0 denotes low risk, a 1 denotes high risk, and a 2 denotes an uncertain risk. These findings are displayed in Table 3, with final expert review denoted in the caption.

The odds ratio was used to quantify the agreement between rs-fMRI and the comparative modalities. An odds ratio larger and less than 1 implies agreement and disagreement, respectively, while an odds ratio of 1 implies that the modalities are independent. Two types of models were used, namely, fixed-effects models and random-effects models (Borenstein et al., 2010). Differences in odd ratios between studies are viewed as systematic differences in fixed-effects models but as random variation in random-effects models.

The systematic literature review resulted in 25 studies qualifying for inclusion (Figure 1). Reasons and quantities for study exclusion are delineated in Figure 1. Of the three Hunyadi et al. (2014, 2015a, 2015b) publications, there is potential for unconfirmed population overlap. However, there is clear evidence of unique participant data as the comparative modalities used in each were different (e.g., EEG, EEG-fMRI, and surgical outcome). Based on these distinctions, we concluded that the subjects were unique across studies.

The inter-rater reliability data of the reviewers’ evaluation of the abstract of the publications are provided in Table 4. Inter-rater reliability of the publications reflected overall good agreement between raters, with proportionate agreement between 0.7 and 1 for all but one rater pairing at 0.33, and random agreement probability between 0.88–1.0 for all but two rater pairings at 0.55 (Table 4). Cohen’s Kappa coefficient range was −0.5 to 0.47, which is low. However, this ratio’s value was indeterminate in six rater pairings; thus, proportionate agreement was determined to be more informative of the study findings.

Of the 337 publications with full-text review, 312 were excluded, with the most common reason being study design (141) not matching with the study question, which required a known comparator on a 1:1 basis for the rs-fMRI SOZ.

The most common rs-fMRI analysis methods were ICA and SBC, utilized in nine and eight studies, respectively. For studies not meeting inclusion criteria, the analysis methods are in Table 5. The analyses were categorized as “data-driven” if the whole brain was investigated and “hypothesis-driven” if the study investigated a reduced set of regions of interest. The comparative modalities included aMRI, EEG, iEEG, EEG-fMRI, MEG, and surgical outcomes.

From the remaining 25 included studies, 452 rs-fMRI SOZ patient comparisons were evaluated. The age range was 18 months to 66 years old (Table 6). Six studies included children: Lee et al. (2014), Reyes et al. (2016), Boerwinkle et al. (2017), Chen et al. (2017), Boerwinkle et al. (2019), and Zhao et al. (2019).

Of the studies utilizing imaging modalities to define SOZ truth, two counted participants more than once (which is well accepted given the method explanation): Bettus et al. (2010) and Khoo et al. (2019). Bettus et al. (2010) evaluated the left and right hemisphere separately. Khoo et al. (2019) evaluated heterotopic nodular pairs within patients separately. Of the studies utilizing surgical outcomes to define ground truth, each surgery counted as a separate comparison event for corresponding pre-operative rs-fMRI (Boerwinkle et al., 2019). They investigated 64 surgical outcomes among 58 participants, four participants had two surgeries, and one participant had three surgeries.

Per the QUADAS-2 Bias Assessment Tool, there are four domains of bias: patient selection, index test, reference test, and flow and timing (Whiting et al., 2011), with results as delineated in Table 3. According to QUADAS-2 guidelines, to qualify for downgrading evidence, a publication must have two domains with high-risk bias. None of the included publications had bias meeting this level of concern. Further, of the areas rated as high risk, two were overturned by further expert review as described below.

For the patient selection domain, seven studies were rated as high risk due to selection of patients that was not well specified, such as being in consecutive order or provided reasoning otherwise. One had concern for population type (Whiting et al., 2011). This study validated rs-fMRI SOZ to that of SEEG in DRE due to multiple heterotopic nodules. Thus, the patient selection does match the study question by expert review, though the assigned reviewers rated it otherwise. The rs-fMRI connectivity between the heterotopias corresponded to that found by seizure and epileptiform activity by SEEG. Thus, while the two reviewers who evaluated this specific study in 28 consecutive patients were concerned if the comparison lined up with the study question, the expert review consensus is rated as low risk, although again the original finding is provided.

In the reference test domain, one study was of high risk (Whiting et al., 2011). In this study, six patients who underwent SEEG and rs-fMRI were analyzed by predefined metrics. Thus, it is not clear why our interpreters rated this study as possibly biased since the methods removed expert opinion from interpretation. Because of this, in retrospect, our expert overall rating is no concern, although the original results are provided in the table.

The fixed-effects analysis presented some indicators of SOZ agreement. (1) The study-specific common odds ratio was estimated to be 8.00 (95% CI: 4.3–14.9). Nonetheless, the usefulness of this single estimate is highly questionable because there is strong evidence against the existence of a common odds ratio (p < 0.001). (2) Out of the 10 studies with no zero margins included in the fixed-effects analysis, 8 studies estimated positive agreement, and 2 studies estimated negative agreement. (3) In the 4 largest studies, those with greater than 40 subjects, the odd ratio estimates, and the respective 95% confidence intervals were above 1. (4) Only one of the 25 included studies showed poor agreement between rs-fMRI and a comparator. Hunyadi et al. (2015a) found that rs-fMRI localized the SOZ to a different location than their comparative modality, surgical outcome, more often than it localized the SOZ to the same location (Hunyadi et al., 2015a). This Hunyadi et al. (2015a) study required that participants have an EEG-fMRI negative for localizing information, so this is a potential explanation for the lack of helpful information from a similarly BOLD-dependent modality (Hunyadi et al., 2015a). Though with it being such a relatively small study, caution to any conclusion is warranted. (5) Finally, somewhat spuriously different than the other studies, but in the opposite direction, Boerwinkle et al. (2019) had a relatively high odds ratio. A few key implementation aspects may yield some insight into this difference. One is that this study came from a repeat first author in relatively larger studies, implying possible influence of learned expertise effect. However, there is also the possibility of bias, though this is contradicted by a separate lab validating subjects from the same dataset using fully automated SOZ detection method with deep learning and artificial intelligence (Banerjee et al., 2023).

The random-effects analysis estimated a population median odds ratio of 1.007 (95% PI: 0.003–361.41). The hypothesis of homogeneity was strongly rejected (p-value = 0.001), arguing against the existence of a true common odds ratio. The prior meta-analysis estimated positive agreement in SOZ localization between rs-fMRI and the comparators (Chakraborty et al., 2020) using a random-effects model and found moderate, yet not significant (p = 0.162), heterogeneity in odd ratios. The added scope of rs-fMRI SOZ analysis methods, validation modalities, and wide range of patient demographics across studies may have contributed to the increased evidence against a common odds ratio. One source of such study-level heterogeneity is evidenced by the relatively wide array of rs-fMRI SOZ analysis methods, and even within the same investigator over time a difference potentially due to increasing experience. Notably, there could be an early emerging trend of convergence as the two most frequently used in recent larger studies were the data-driven ICA and hypothesis-driven SBC. This is not to say there is evidence of superiority between the analysis methods. Furthermore, we posit that each method has its strengths and, when used appropriately in isolation or combination, may improve SOZ localization and surgical outcomes. For example, to verify whether comparative modalities’ SOZ candidates contain the true SOZ, ICA may be an ideal initial search approach since it may hone in on a smaller cadre of region(s). Then, hypothesis-driven techniques, such as SBC or effective connectivity, may further narrow the SOZ candidates (Grande et al., 2020; Jayakar et al., 2016; Tavares et al., 2017; Chassoux et al., 2018; Mullin et al., 2016).

There were indicators of differences in SOZ localization agreement between rs-fMRI and the modality groups. (1) The fixed-effects analysis demonstrated high evidence of systematic differences in agreement between rs-fMRI across modalities with the level of agreement being higher for surgical outcome when compared to EEG (p = 0.002) and iEEG (p = 0.007). (2) In addition, the analysis presented high evidence of systematic differences in agreement between rs-fMRI and the comparator between studies sharing the same modality. These results suggest that while an overall positive agreement was determined by the fixed-effects analysis, the level of agreement varied across modalities, and even between studies using the same modality.

The random-effects model estimated a population mean sensitivity of 0.91. These results indicate that there is a high probability of the rs-fMRI localizing the SOZ given that the comparative modality also localized the SOZ. In contrast, the estimated population mean specificity was 0.09 which indicates a low probability of rs-fMRI not localizing the SOZ given that the comparative modality also did not localize the SOZ. Similarly, the study-specific predicted sensitivities were always higher than the predicted specificities. Similar results were observed in the fixed-effects analysis.

Strengths of this study include the wide inclusion of rs-fMRI SOZ analysis methods and extent of validation modality subtypes. In addition, this study is the first meta-analysis to investigate the sensitivity and specificity of rs-fMRI for seizure onset localization. However, there are several limitations to consider.

First, while the comparative localizing modalities are established as standards of care, this study evaluated concurrent validity, which measures the level of agreement between two or more instruments in assessing the same outcome as opposed to other measures of validity (Higgins and Straub, 2006). Second, nearly half of the studies included in our analysis had small sample sizes (n < 10), which contributes to low precision and non-estimable odds ratios, and may limit the reliability of our pooled sensitivity and specificity estimates. The predominance of small case studies also raises the possibility of publication bias, as studies with positive or significant results are more likely to be published, potentially skewing the overall findings. This publication bias could result in an overrepresentation of positive outcomes, which may artificially inflate sensitivity and specificity values for rs-fMRI’s accuracy in SOZ localization.

Third, heterogeneity across studies represents a significant limitation. The heterogeneity test for study-level odds ratios in both random- and fixed-effects analyses yielded statistically significant results, suggesting that the studies included in the meta-analysis exhibit heterogeneity and provide strong evidence that a study-level common odds ratio does not exist. This could be due to multiple factors including the inclusion of multiple comparative modalities, different rs-fMRI software, seizure etiology, and patient populations. Furthermore, our QUADAS-2 analysis indicated that 12 out of 25 studies had an uncertain or high risk of bias in at least one domain; these studies were retained in our comparisons as excluding them could have further limited our dataset. However, these high-risk studies may have introduced additional bias, impacting the robustness of our meta-analysis.

Furthermore, this analysis included only studies published up to 2019, which excludes recent advancements in rs-fMRI methodologies. Since 2019, newer techniques such as improved data acquisition, processing algorithms, and network-targeted approaches have emerged, potentially enhancing the clinical applicability of rs-fMRI for SOZ localization. Future meta-analyses incorporating these newer methodologies will be critical to validate and extend our findings as rs-fMRI evolves as a tool for SOZ localization. Additionally, with time the ILAE surgical outcome classification system has evolved to become the most broadly recognized system to compare surgical outcomes. Our study is limited by retrospective collection of surgical outcomes classified under the Engel approach, which may be improved in the future by transition to the ILAE approach.

While this is the largest meta-analysis to date evaluating the agreement of SOZ localization between rs-fMRI with other localizing modalities, the overall number of studies remains limited. Therefore, it is important to interpret these results as guidance for future validation studies of rs-fMRI.

Studies are needed to validate the use of rs-fMRI in localizing SOZ with comparative modalities, with special consideration to modalities that have high accuracy such as iEEG and surgical outcomes. Evaluating the convergent validity of rs-fMRI against gold-standard modalities may allow clinicians to localize SOZ while minimizing the degree of invasive iEEG verification risks. In addition, rs-fMRI is an increasingly accessible imaging modality as more healthcare centers acquire three Tesla MRI machines and the images can be sent to experts for analysis, or possibly fully automated SOZ localization (Jayakar et al., 2016). This compares to methods such as iEEG, which requires on-site expertise for electrode placement. Rs-fMRI could benefit patients in resource-deprived communities by providing them with increased access to expertise. Unfortunately, studies evaluating the convergent validity of rs-fMRI with modalities such as iEEG are limited due to the invasive nature of gold-standard procedures, limited funding and resources such as insurance reimbursement (Young et al., 2023), access to the expertise of advanced modalities, and difficulties related to the recruitment of individuals willing to participate in these studies.