Elmer (2016) conducted an fMRI investigation of “simultaneous language translation” in five participants. The article presents group and individual analyses of German-to-Italian and Italian-to-German translation, confined to a small set of anatomical regions previously reported to be involved in multilingual control. Here we take the opportunity to discuss concerns regarding certain aspects of the study.

A core claim of the article is that group analyses fail to handle individual-differences, especially regarding higher cognitive functions whose loci are putatively more variable across individuals. The utility of using individual participants' functionally-determined regions of interest for analyses has long been considered (Saxe et al., 2006; Fedorenko et al., 2010). However, Elmer does not apply this approach, but rather presents both individual and group-level analyses without formally combining them. A claim is made that this approach is especially beneficial in cases of small sample sizes, but no support exists for this. Even if the approach accommodates variability in the localization of individual participants' activations, the analysis remains an assessment of group-level consistency, and is therefore necessarily subject to the usual concerns regarding statistical power (the problems caused by small sample sizes, including how they have a deleterious impact on the literature by inflating apparent effect-sizes, are discussed in Button et al., 2013). With an estimated effect size of delta = 0.5 (generous for an fMRI contrast), the power to detect a real effect using a one-sample t-test at a two-tailed alpha = 0.01 (the uncorrected p-value presented in the article) with N = 5 is only 3%. The equivalent estimate for the N = 50 published by Hervais-Adelman et al. (2015) is 80%.

Crucially for an investigation of simultaneous interpreting (SI), the materials employed do not truly test SI. Short subject-verb-object sentences can be trivially converted between German and Italian as word-for-word calques. This potentially reduces the task to the management of co-activated lexical items, without any requirement to access higher-level linguistic processes (e.g., syntax). Also, participants in this study appear to have initiated their translations, on average, after the offset of the sentences with which they were presented (sentences averaged 1.75 s, but mean response latencies reported are > 2.5 s). Seemingly, participants were executing a consecutive task rather than a “simultaneous” one. It is therefore questionable whether the reported results relate to SI, when they may in fact relate to the verbal working memory and semantic processes associated with encoding and maintaining the input sentences, rather than language control processes.

Participants in Elmer's study were professional interpreters with expertise ranging from four to 22 years of professional practice. The claim is made that this compensates for the small sample size by estimating a putative impact of expertise, however no analysis of this is presented. Moreover, participants' language combinations are not as well-matched as claimed. If standard definitions of A, B and C languages are used, two of the five participants interpret (consecutively, not simultaneously) into German professionally (those having it as a B language) while the other three do not. This aggravates the issue of individual differences in the Italian-to-German condition.

Elmer's (2016) selection of brain areas for analysis is very restrictive. In contrast, Hervais-Adelman et al. (2015) investigation of SI implicated a broad network of regions, many of which are not considered here, potentially resulting in implicated regions being missed. To enable a more direct comparison, Figure 1 and Table 1 represent analyses analogous to those reported by Elmer (2016), executed on the data from Hervais-Adelman et al. (2015). Namely, we report the proportion of participants showing significant BOLD differences for “Interpreting into L1” vs. “Shadowing L2” at uncorrected p < 0.01 in every region of the AAL template (Tzourio-Mazoyer et al., 2002). This analysis shows that the greatest between-subjects consistency in the network (90%) is in left supplementary motor area, a region known to be heavily implicated in cognitive control (Nachev et al., 2008) and language switching (De Baene et al., 2015). Ought we, therefore, conclude that this region is the hub of simultaneous interpreting? In the absence of any evidence that can allow us to draw this inference, we would not presume to do so. We therefore question, with such a small sample and without any causal evidence, Elmer's conclusion that the reliability of pars triangularis activation indicates that it is a hub for language control. Elmer's analysis does not consider much of the broad language control network implicated in SI (see Table 1 and Hervais-Adelman et al., 2015), and yet the possibility that regions other than the selected ROIs may be equally or more frequently implicated than pars triangularis is not discussed.

We do not question that pars triangularis plays a substantial role in interpreting, but the data do not provide emphatic support for the idea that “These results challenge previous models” nor do they suggest the need for “re-definition of the language-control network” (Elmer, 2016, p.5). Although we appreciate that the paper incorporates an extensive “limitations” section, those limitations are seemingly not taken into consideration when drawing these conclusions. The paper contains some genuine issues beyond those acknowledged that we worry fundamentally undermine the conclusions: real effects are likely to have been missed due to lack of power, the participant selection introduced unnecessary sources of variability (age and expertise), the selection of materials means that the reported effects may not relate to SI but to consecutive interpretation and the constrained analysis space rules out conclusions about the broader language control network. These, coupled with the statistically-questionable claims made regarding how the small sample size and inter-subject variability can somehow be overcome, lead us to fundamentally question the conclusions of the article.

We welcome all challenges that arise from any effort to replicate and improve upon our and others' studies. However, while cognitive neuroscience finds itself in the harsh spotlight of a “reproducibility crisis” (Barch and Yarkoni, 2013), it behooves us to be cautious in our approach to publication, and it seems especially important to avoid drawing overly strong conclusions on the basis of underpowered studies.