Thirty-three University of Memphis undergraduate students participated for extra credit in a psychology course. All participants had normal or corrected vision and were native English speakers. Fifteen participants were randomly assigned to the semantic judgment condition, and 18 participants were randomly assigned to the iconicity judgment condition.

Each condition consisted of 64 iconic/reverse-iconic word pairs extracted from previous research (Louwerse, 2008; Louwerse and Jeuniaux, 2010; see Appendix). Thirty-two pairs with an iconic relationship were presented vertically on the screen in the same order they would appear in the world (i.e., sky appears above ground). Likewise, 32 pairs with a reverse-iconic relationship appeared in an order opposite of that which would be expected in the world (i.e., ground appears above sky). The remaining 128 trials contained filler word pairs that had no iconic relationship. Half of the fillers had a high semantic relation (cos = 0.55) and half had a low semantic relation (cos = 0.21), as determined by latent semantic analysis (LSA), a statistical, corpus-based, technique for estimating semantic similarities on a scale of −1 to 1 (Landauer et al., 2007). All items were counterbalanced such that all participants saw all word pairs, but no participant saw the same word pair in both orders (i.e., both the iconic and the reverse-iconic order for the experimental items).

An Emotiv EPOC headset (Emotiv Systems Inc., San Francisco, CA, USA) was used to record electroencephalograph data. EEG data recorded from the Emotiv EPOC headset is comparable to data recorded by traditional EEG devices (Bobrov et al., 2011; Stytsenko et al., 2011). For instance, patterns of brain activity from a study in which participants imagined pictures were comparable between the 16-channel Emotiv EPOC system and the 32-channel ActiCap system (Brain Products, Munich, Germany; Bobrov et al., 2011). The Emotiv EPOC is also able to reliably capture P300 signals (Ramírez-Cortes et al., 2010; Duvinage et al., 2012), even though the accuracy of high-end systems is superior.

The headset was fitted with 14 Au-plated contact-grade hardened BeCu felt-tipped electrodes that were saturated in a saline solution. Although the headset used a dry electrode system, such technology has shown to be comparable to traditional wet electrode systems (Estepp et al., 2009). The headset used sequential sampling at 2048 Hz and was down-sampled to 128 Hz. The incoming signal was automatically notch filtered at 50 and 60 Hz using a 5th order sinc notch filter. The resolution was 1.95 μV.

In both the semantic judgment and iconicity judgment conditions, word pairs were presented vertically on an 800 × 600 computer screen. In the semantic judgment condition, participants were asked to determine whether a word pair was related in meaning. In the iconicity judgment condition, participants were asked whether a word pair appeared in an iconic relationship (i.e., if a word pair appeared in the same configuration as the pair would occur in the world). Participants responded to stimuli by pressing designated yes or no keys on a number pad. Participants were instructed to move and blink as little as possible. Word pairs were randomly presented for each participant in order to negate any order effects. To ensure participants understood the task, a session of five practice trials preceded the experimental session.

Language statistics were operationalized as the log frequency of a-b (e.g., sky – ground) and b-a (e.g., ground – sky) order of word pairs (cf. Louwerse, 2008; Louwerse and Jeuniaux, 2010; Louwerse and Connell, 2011). The order frequency of all 64 word pairs within 3–5 word grams was obtained using the large Web 1T 5-gram corpus (Brants and Franz, 2006).

Twenty-four participants at the University of Memphis estimated the likelihood that concepts appeared above one another in the real world. Ratings were made for 64 word pairs on a scale of 1–6, with 1 being extremely unlikely and 6 being extremely likely. Each participant saw all word pairs, but whether a participant saw a word pair in an iconic or a reverse iconic order was counterbalanced such that each participant saw iconic and reverse-iconic word pairs, but no participant saw a word pair both in an iconic and a reverse-iconic order. High interrater reliability was found in both groups (Group A: average r = 0.76, p < 0.001, n = 64; Group B: average r = 0.74, p < 0.001, n = 64), with a negative correlation between the two groups (average r = −0.72, p < 0.001, n = 64).

A mixed effects regression was conducted on RTs with order frequencies and iconicity ratings as fixed factors and participants and items as random factors. For the semantic judgment condition, a mixed effects regression showed that statistical linguistic frequencies significantly predicted RTs, F(1, 760.86) = 24.95, p < 0.001, with higher frequencies yielding faster RTs. Iconicity ratings did not yield a significant relation with RT, F(1, 762.09) = 0.46, p = 0.5 (see the first two bars in Figure 1; Table 1).

For the iconicity judgment condition, a mixed effects regression showed statistical linguistic frequencies again significantly predicted RT, F(1, 945.78) = 5.03, p = 0.03, with higher frequencies yielding faster RTs. Iconicity ratings also yielded a significant relation with RT, F(1, 947.65) = 5.61, p = 0.02, with higher iconicity ratings yielding lower RTs (see the second two bars in Figure 1; Table 1).

Figure 1 shows that statistical linguistic frequencies explained RTs in both the semantic judgment and the iconicity judgment conditions, but the effect was stronger in the semantic judgment than in the iconicity judgment condition. Figure 1 and Table 1 also show the opposite results for perceptual simulation in that during the semantic judgment condition, the effect of perceptual simulation on RT was limited (and not significant). However, in the iconicity judgment condition, perceptual simulation was significant. The interaction for linguistic frequencies and condition (semantic versus iconic) was significant, F(2, 1005.05) = 15.88, p < 0.001, as was the interaction for perceptual simulation and condition, F(2, 1634.20) = 2.9, p = 0.05. Indeed, the overall interaction between factors (linguistic and perceptual) and condition was significant, F(2, 1540.18) = 8.10, p < 0.001.

These findings replicate the RT data in Louwerse and Jeuniaux (2010). That is, order frequency better explained RTs than the iconicity ratings did in the semantic judgment task, but iconicity ratings better explained RTs than the order frequency did in the iconicity judgment task.

As discussed earlier, we utilized previously established EEG source localization techniques in conjunction with statistical analyses to determine when and where relative effects of linguistic and perceptual processes occurred. Continuous neural activity was recorded from 14 international 10–20 sites (AF3, F7, F3, FC5, T7, P7, O1, O2, P8, T8, FC6, F4, F8, and AF4; Reilly, 2005, p. 139). Scalp recordings were referenced to CMS/DRL (P3/P4) locations. All electrode impedances were kept below 10 kΩ. As the Emotiv EPOC headset is noisier than high-end systems, to minimize oculomotor, motor, and electrogalvanic artifacts, a high-pass hardware filter removed signals below 0.16 Hz and a low-pass filter removed signals above 30 Hz (see Bobrov et al., 2011 and Duvinage et al., 2012 for similar filtering ranges with the Emotiv EPOC headset). The EEG was sampled at 2048 Hz and was down-sampled to 128 Hz. Gross eye blink and movement artifacts over 150 μV were excluded from the analysis. All data were wirelessly collected via a proprietary Bluetooth USB chip operating in the same frequency range as the headset (2.4 GHz). Data were recorded using Emotiv Testbench software (Emotiv Systems, Inc., San Francisco, CA, USA).

Data were filtered using EEGLAB (Delorme and Makeig, 2004), an open-source toolbox for MATLAB (Mathworks, Inc., Natick, MA, USA). Independent component analyses were implemented using ADJUST, an algorithm that automatically identifies stereotyped temporal and spatial artifacts (Mognon et al., 2010). Any remaining oculomotor or motor activity was visually identified and removed from the dataset.

On average, subjects took 1809 ms to process and respond to the words presented on the screen. Therefore the sLORETA package (Pascual-Marqui, 2002) was used to localize general activity at an early (97–291 ms) and a late (1551–1744 ms) time interval (as we predicted linguistic processes would precede perceptual simulation) in both conditions. The early time period began shortly after presentation of the stimuli and the late time period began shortly before the subject response. LORETA used the MNI152 template (Fuchs et al., 2002) to compute a non-parametric topographical analysis of variance comparing differences between two maps of averaged cortical activity over each time period (Strik et al., 1998). The topographies significantly differed between conditions at early, p < 0.01, and late, p < 0.01, intervals, with the maximum source for the early time period being found around the left inferior frontal gyrus (iFG; near electrode sites FC5, F7, and T7) and the maximum source for the late time period being found near the lingual gyrus (near electrode sites O1, O2, P7, and P8). As source localization with EEG poorly maps anatomical correlates to function, these sites are obviously approximations of the relevant underlying cortical regions (Nunez and Srinivasan, 2005). Note we are not attempting to pinpoint exact regions of neural activity at a given time but instead we are simply attempting to compare general estimates of neural activity in early versus late processing (i.e., we would like to determine when processing occurs in more linguistic versus in more perceptual regions over the duration of a trial).

Although neural processes are quite distributed and bilaterally activate multiple cortical regions (Bullmore and Sporns, 2009; Bressler and Menon, 2010), there is considerable agreement that specific regions (such as the left iFG and left superior temporal gyrus (STG) consistently show increased activation during language processing (Cabeza and Nyberg, 2000; Papathanassiou, 2000; Blank et al., 2002; De Carli et al., 2007). The same applies to visual perception and visual imagery processes, which bilaterally activate multiple cortical regions, in particular occipital and parietal lobes (Kosslyn et al., 1993, 1999; Alivisatos and Petrides, 1996). Further, visual imagery of words activates these same regions that process incoming perceptual information (Ganis et al., 2004). Reichle et al. (2000) used fMRI to demonstrate that when told to rely on visual imagery while processing linguistic information, subjects were more likely to show increased activation in parietal lobes. As expected, when asked to rely on verbal strategies, activation in traditional language processing regions dominated. Finally, in an fMRI study, Simmons et al. (2008) found that when asked to generate situations in which a word might occur, subjects showed increased activity in the cuneus, precuneus, posterior cingulate gyrus, retrospinal cortex, and lateral parietal cortex. However, when asked to participate in a word association task, activation occurred in language processing regions of the brain, specifically the lateral left iFG and the medial inferior frontal. During early conceptual processing (first 7.5 s of a 15 s trial), activation was similar to that of the word association task (i.e., these same language processing areas were active). This is consistent with our output from sLORETA in that during early processing, the maximum source was also the left iFG. Unlike early processing, Simmons et al. (2008) found that late conceptual processing (last 7.5 s of a 15 s trial) resulted in activation of the precuneus, posterior cingulate gyrus, and the right lateral parietal cortex (regions all closest to electrodes P7, P8, O1, and O2), the same regions active during situation generation. Although our sLORETA source localization indicated that the maximum source for our late time period was near the lingual gyrus, this region is also in closest proximity to electrode sites P7, P8, O1, and O2.

Figure 2 shows the activation for a participant averaged across all trials in 100 ms increments. A relatively localized increase in activation in linguistic processing regions began almost immediately after a stimulus was presented. Around the middle of the trial, the activation dispersed from the linguistic processing regions toward perceptual processing regions. Late in the trial, localized activation was relatively greater in perceptual processing regions. This pattern matches the conclusions drawn by Louwerse and Connell (2011) on the basis of RT data and the results obtained through sLORETA, that linguistic processes precede perceptual processes.

To complement the pattern observed in Figure 2 in both our RT data and in the sLORETA results, we performed a mixed effects regression on electrode activation. We assigned the linguistic cortical regions, as determined by sLORETA localization, a dummy value of 1, and we assigned the perceptual cortical regions, as determined by sLORETA localization, a dummy value of 2. We used electrode activation as our dependent variable, and participant, item, and receptor as random factors. The reason we used individual receptors as random factors was to rule out strong effects that could be observed for one receptor but not for others within the regions commonly associated with linguistic or perceptual processing. With this analysis, our objective was to determine to what extent linguistic or perceptual cortical regions overall showed increased activation throughout the trial. As in the previous analyses, F-test denominator degrees of freedom for the dependent variable were estimated using the Kenward–Roger’s degrees of freedom adjustment.

For the semantic judgment condition, a significant difference was observed between linguistic and perceptual cortical regions, F(1, 1153108.58) = 46.70, p < 0.001. A similar pattern was found for the iconicity judgment condition, F(1, 1464148.76) = 24.07, p < 0.001. The fact that a difference was observed is perhaps uninteresting; differences between linguistic and perceptual regions are expected. Instead, the direction of the effect is important here. Recall that linguistic regions were dummy coded as 1, and perceptual regions were dummy coded as 2. Positive t-values would indicate that perceptual regions dominate, and negative t-values would indicate that linguistic regions dominate. Based on the findings in the RT analysis reported above, we predicted that linguistic regions would dominate in both the semantic and iconicity task, and more so in the semantic judgment task than in the iconicity judgment task. This prediction is supported by the results; t-values in both the semantic and iconicity tasks were negative, as predicted with higher t-values in the semantic task, t (1153109) = −6.83, p < 0.001, than in the iconicity task, t (1464149) = −4.91, p < 0.001, replicating the RT findings.

To determine whether linguistic processes precede perceptual simulation processes, we created 20 time bins for each trial per participant, per condition (cf. Louwerse and Bangerter, 2010). Each time bin was therefore approximately 80 ms for the semantic judgment condition and 95 ms for the iconicity judgment condition. Twenty time bins allowed for the largest number of groups for examining trends of each factor while retaining sufficient data points per participant to test the time course hypotheses. Mixed effects models were again run, now with time bin as an added predictor in the model. The t-values of the mixed effects models per time bin are shown in Figure 3A, Tables 2 and 3. The figure shows that t-values in both the semantic judgment and the iconicity judgment experiments are predominantly negative in the first half of the trial (suggesting a bias toward cortical regions associated with linguistic processing), and predominantly positive toward the end of the trial (suggesting a bias toward cortical regions associated with perceptual processing). Note here that these are the relative effect sizes for the two clusters of cortical regions (FC5, F7, and T7) and (O1, O2, P7, and P8), with the effects for individual electrodes filtered out. The findings do not show low activation for the perceptual processing areas early on in the trial (as words must of course be recognized by the visual system during processing); these results merely show that, relative to the brain regions associated with linguistic processing, the effect sizes of perceptual processing regions dominate later in the trial. Also note the relative effect for brain regions associated with perceptual processing very early in the trial (time bins 1–4), perhaps in line with the early activation of perceptual simulations (Hauk et al., 2008; Pulvermüller et al., 2009).

To further demonstrate the neurological evidence for relatively earlier linguistic processes and relatively later perceptual simulation, we fitted the t-test values for the 20 time bins using exponential, power law, and growth models. The fit of the sinusoidal curve was superior to these models across the two data conditions. Figure 3B presents the fit, the standard errors, and the values for the four variables. The sinusoidal fit converged in four iterations (iconicity task) and five iterations (semantic task) to a tolerance of 0.00001.

Using the sinusoidal model and the parameters derived from the data, the following figure emerged (Figure 3B). For both the semantic judgment and the iconicity judgment conditions, linguistic cortical regions dominated initially, followed later by perceptual cortical regions. As Figure 3B clearly shows, activation in linguistic cortical regions dominated in the semantic judgment task, whereas activation in perceptual cortical regions was prominent in the iconicity judgment task. Moreover, linguistic cortical regions showed greater activation relatively early in the trial, whereas perceptual cortical regions showed greater activation relatively late in processing. The results from these analyses are in line with results we obtained through both more commonly used source localization techniques and RT analyses, but they give a more detailed view of relative cortical activation for linguistic and perceptual processes throughout each trial.