Introduction
In everyday communication and interaction, we often experience our sound environment through movement. We sway with music, move our eyebrows, head, or body while participating in a conversation, we gesture while speaking (e.g., McNeill, 1992; Janata et al., 2012). In particular, rhythmic sounds such as music featuring a regular beat or speech with a metrical structure (e.g., as in rhymes or poetry) are often accompanied by coordinated rhythmic movement (e.g., Ong, 2002; Maes et al., 2014). Parents stimulate their infants via concurrent rhythmic movements in verbal games and nursery rhymes (e.g., Stern, 1974; Opie and Opie, 1997). Older children utter words and move simultaneously during rhyme games and song by clapping their hands or stamping their feet (e.g., Opie and Opie, 1988). Past research has shown that rhythm facilitates multisensory coordination and temporal perception (Manning and Schutz, 2013, 2016). In return, aligning motor to verbal rhythms facilitates verbal processing (Falk and Dalla Bella, 2016). The mechanisms underlying the beneficial outcomes of aligned auditory-motor rhythms, particularly in the verbal domain, are still unclear. Here we aim at contributing to this issue by examining auditory-motor alignment and its effects on verbal processing in two rhythmically diverse languages, namely French and German.
There is evidence that cognitive benefits of auditory-motor rhythms can be driven by temporal expectancies. In general, when we expect something to happen at a certain time, we attend more to that particular moment than to another time (e.g., Large and Jones, 1999). Auditory rhythms such as metrical speech or music with a salient beat structure feature accent patterns of syllables and notes that recur (quasi-) periodically in time (e.g., Lehiste, 1977; London, 2004) and are perceived as highly regular. An “accent” refers to a prominent syllable or note marked by the expansion of pitch, intensity, duration or other heightened acoustic properties (e.g., articulatory clarity in speech). Thanks to the recurrence of these accent patterns, a predictable temporal structure (rhythm) emerges across several timescales. That these auditory rhythms enhance attending is supported by both behavioral and psychophysiological evidence (Jones et al., 2002; Ellis and Jones, 2010). EEG studies show that periodic sequences of tones and syllables evoke larger amplitudes and shorter latencies of attention-related brain potentials (i.e., the P3b) than sequences with aperiodic rhythmic organization (Schwartze et al., 2011; Otterbein et al., 2012). In addition, higher attending to temporally predictable sounds is underpinned by neural oscillatory activity that phase-locks to rhythmic periodicities in the auditory signal at hierarchically nested frequencies (e.g., Lakatos et al., 2008; Fujioka et al., 2010; Peelle et al., 2013). Independently of the exact neural basis of rhythmic attending, which is still debated, several studies point to behavioral advantages of temporally predictable sound sequences in both speech and music. In metrical speech, information is better processed and remembered (Quené and Port, 2005; Dilley and McAuley, 2008; Roncaglia-Denissen et al., 2013), in particular when it occurs on accented compared to unaccented syllables (Pitt and Samuel, 1990; Zheng and Pierrehumbert, 2010; Falk and Dalla Bella, 2016). In sum, these findings lend support to the conclusion that temporal expectancies driven by an auditory rhythm in speech may help enhancing the perceptual salience of prominent syllables, and, thereby, facilitate verbal processing at these points in time.
Movement coordinated with the rhythm of sound sequences can also modify the way listeners attend to and encode auditory stimuli. This has been shown with musical material (e.g., Phillips-Silver and Trainor, 2005). For example, synchronous motor activity with periodic tone sequences enhances the amplitude of attention-related brain potentials (Schmidt-Kassow et al., 2013; Conradi et al., 2016). In another EEG study, Chemin et al. (2014) showed that participants moving their hand to either a binary or ternary beat pattern of rhythmically ambiguous music encoded the rhythm of the sound through the rhythmic movement pattern. When the same participants listened to the music later, their brain responses were enhanced at exactly those points in time when their movements had previously occurred. This finding is compatible with the “active sensing” framework which posits that the motor system shapes processing of sensory information by linking auditory information to temporal predictions generated by action planning and execution (see Morillon et al., 2015, for an overview). The results of Chemin et al. (2014) also fit with the common coding theory (Prinz, 1990; Hommel et al., 2001), which states that predictions associated with motor action planning and execution are jointly coded with auditory predictions in the cognitive system (see also Maes et al., 2014, for a review).
While these theoretical approaches have been used to explain the role of rhythm and auditory-motor coupling in music, evidence for multisensory rhythmic effects in speech is still scarce. In a recent study, we investigated the effects of shared predictions generated by a temporally aligned auditory-motor rhythm on speech perception (Falk and Dalla Bella, 2016). Participants were cued to align or misalign a motor rhythm (i.e., finger tapping) with the accented syllables of a metrically structured German sentence. The sentence was then repeated without tapping and the participants detected a verb change occurring either on accented or unaccented syllables. Results showed that participants were most successful in detecting the changes when their finger taps were aligned with the accented syllables compared to misalignment, and when the changes also occurred on accented syllables. Thus, finger tapping aligned to accented syllables is a critical factor leading to improved verbal performance. These findings suggest that the benefits of aligning motor and auditory rhythms may be driven by overlapping temporal expectancies. The result would be a maximal enhancement of attentional resources at the rhythmic reference points (i.e., accented syllables) in the speech signal.
One of the open questions is whether the observed auditory-motor effects are confined to German prosody, or generalize to other languages. German is described as a “stress”-language (Jessen, 1999; Wiese, 2000). At the lexical level, each word is learned with a specific pattern of stressed and unstressed syllables. Thus, stress accents may distinguish meaning and thereby fulfill a contrastive function in German. In the context of spoken discourse, the German stress-accents also have a cumulative (i.e., head-marking) function within their prosodic domain (i.e., feet, prosodic words, and phrases, intonation phrases, etc.), resulting in complex metrical relations between subsequent accents that mark different hierarchical levels, similar to English (Liberman and Prince, 1977; Hayes, 1995). In the German stress system, de-stressing as well as heightened stress are flexibly used to signal communicative deviations from syntactic and semantic default interpretations (e.g., to signal focus; Féry and Ishihara, 2008; Féry and Kügler, 2008). Analysis of the profile of stress-accents (i.e., their position and strength) as a key to comprehend the meaning of words and nested discourse units is common in German listeners. Hence, the German prosodic system may be particularly well-suited to drive rhythmic expectancies and to lead to the ensuing benefits on verbal processing. Here we tested whether predictions driven by auditory-motor rhythms extend to French, a language that differs substantially from the prosody of German. French is most often described as lacking lexical stress (Rossi, 1980). Indeed, accentuation in French has predominantly a demarcative (i.e., boundary-marking) function within the prosodic domain of the accentual phrase (AP, Vaissière, 1991; Jun and Fougeron, 2000, 2002). Primary accents in French are assigned to the last syllable (except Schwa-syllables) of a phrasal segment, thereby marking the right edge of the AP. Secondary accents, though not obligatory, can be found toward the left edge of an AP, depending on the number of syllables and on the lexical composition of the phrase (e.g., Astésano, 2001; Welby, 2003, 2006). Hence, accent placement in French is tightly linked to grouping and phrasing, and, ultimately, to syntactic segmentation (e.g., Millotte et al., 2008; Michelas and D'Imperio, 2015). Importantly, compared to German, stress conflicts between words and phrases such as stress clash and shift (e.g., Mengel, 2000; Bohn et al., 2013) are more rare (Post, 2000). Peperkamp and Dupoux (2002) have suggested that because French lacks lexical relevance of accents and has fixed accent placement, French listeners pay less attention to accentual variations as compared to listeners of languages with lexical/variable stress. This intriguing hypothesis is supported by the observation of difficulties in encoding and memorizing stress contrasts (i.e., “stress-deafness”) in native French listeners when learning a foreign language with lexical/variable stress (e.g., Dupoux et al., 2008; Schmidt-Kassow et al., 2011; Domahs et al., 2012; but see also Michelas et al., 2016). Thus, we anticipate that French listeners will attend less to accented syllables while tapping to them than German listeners did. As a result, the benefit of aligning movement and speech rhythms may be less visible in French than in German.
We tested this hypothesis by asking French and German listeners to detect a verbal change in their native language in the context of an aligned or misaligned auditory-motor rhythm. The paradigm is the same as in Falk and Dalla Bella (2016). A second goal was to closely examine whether individual differences in motor performance could predict the success in this task for both languages.
Discussion
The aim of the present study was to test whether the beneficial effects of aligning a motor rhythm to a speech rhythm on verbal processing are found in rhythmically diverse languages (i.e., French, a non-stress language vs. German, a lexical stress language). In addition, we investigated the links between individual motor performance and benefits in speech processing. French and German native speakers were cued to tap their finger with metrical speech stimuli they heard in their native language. Tapping was either aligned with the speech accents (congruently) or non-aligned (incongruently; see Falk and Dalla Bella, 2016). After tapping, participants detected subtle verbal changes in the speech stimuli.
Our results showed that native speakers of both a lexical stress- and a non-stress language were more successful in detecting subtle verbal changes in speech, when their finger taps were congruently aligned with the accented syllables of the sentences compared to incongruent alignment and to a perceptual baseline. Thus, the benefits of aligning motor and speech rhythms on verbal processing, already shown in German (Falk and Dalla Bella, 2016), extend to a non-stress language (French). Generalization of the effect to rhythmically different languages supports the idea that motor alignment with prominent syllables in speech enhances attending to expected moments in time (e.g., Chemin et al., 2014), and thereby verbal processing. Although French and German accentuation substantially differs in prosodic embedding, function, and acoustic correlates, the present findings support the idea that both types of rhythmic organization similarly serve as anchors for linking perception and action in time through shared predictions in a metrical context. This is in line with previous results pointing to a close link between speech prominences and manual movements (Kelso et al., 1985; Rochet-Capellan et al., 2008; Parrell et al., 2014). Yet, note that the present study did not allow us to directly compare sensitivity in French vs. German listeners, as the material in the two languages was very different in terms of semantics, syntax, and phonetics/phonology.
We further investigated which aspects of the motor performance can predict individual success in the verbal task. We identified two motor predictors for verbal detection, synchronization performance with the cue and tapping rate during continuation, which differed depending on the alignments between motor and auditory rhythms. First, when rhythms were aligned (congruent condition), participants who tapped earlier to the metronome tones preceding the verbal stimulus had more success in detecting a verbal change than the other participants. Tapping in advance of a discrete, periodically recurring and, thereby, expected sound event is a common finding in tapping research (negative mean asynchrony; Fraisse and Voillaume, 1971; Aschersleben, 2002; Repp and Su, 2013). It is indicative of anticipatory behavior during synchronization (Repp, 2005; van der Steen and Keller, 2013). The possibility that expectancies emerging during the synchronization phase can influence subsequent speech processing is supported by brain research on auditory cueing with speech (e.g., Cason and Schön, 2012). In a recent perceptual EEG-study with the same French material (Falk et al., in press), sentences were cued by an auditory cue matching or non-matching the rhythm of speech (no motor condition). Akin to the present study, the authors found that participants' success in behavioral response (a word memory task) was correlated with neural activity during the cueing but not during the speech phase. In particular, participants whose neural oscillatory activity showed better phase-locking (i.e., “neural entrainment,” Lakatos et al., 2008) to the rhythm of the cue, also had higher success in the following speech memory task. Neural entrainment has been described as a process that drives neural excitability toward expected times and thereby engages higher attentional resources at these points in time (e.g., Calderone et al., 2014). This neural effect on attending is even more pronounced when motor synchronization is added (e.g., Morillon et al., 2014). Therefore, in our experiment, participants generating stronger expectancies during the synchronization phase may also be those that are more likely to maintain these expectancies during the continuation phase and having higher attentional resources to allocate. Thus, they may be more prone to get cognitive and neural benefits from rhythmically induced attending to speech when temporal expectancies are met by the speech structure (i.e., in the congruent condition). By contrast, in the incongruent condition, expectancies from the synchronization phase cannot be used to enhance attending because of the subsequent misalignment of taps and speech accent structure. The misalignment may disrupt expectancies directed to the relevant verbs and, hence, no relation between the synchronization phase and detection performance is found. Finally, it is interesting that we failed to observe a relation between continuation accuracy and verbal detection in the congruent condition. This may be linked to additional processes intervening during speech processing (e.g., comprehension, syntactic, or semantic processes) or to task-related factors (e.g., participants were not explicitly instructed to synchronize with stressed syllables while listening to speech). These possibilities deserve further investigation in future studies.
Second, when rhythms were not aligned (incongruent condition), tapping rate in the continuation phase (i.e., while participants listened to speech) predicted detection of a verb change, but differently depending on the language. Slowing down tapping in the continuation phase worsened detection in French speakers, while this was not the case in German speakers. This group difference may be linked to the fact that French speakers generally showed a considerable drift in tapping rate (a deceleration of 15 ms per ITI, on average) in the incongruent alignment condition which naturally also affected continuation accuracy. German speakers did not show any drift. Moreover, slower tapping rate in the continuation phase was associated with more positive asynchrony (i.e., lag) relative to the expected moments of the tap. In sum, French speakers, unlike German speakers, tended to lag behind the expected moment of the taps when motor and speech rhythms were incongruently aligned and this drift was linked to worse detection performance. In the following, we will discuss different explanations that can account for these findings.
A relatively trivial explanation is that pre-existing differences in musical experience between language groups may have led to worse tapping performance. However, musical experience should have produced group differences in all aspects of tapping performance, which was not the case (e.g., in the synchronization phase). Moreover, including musical experience as an additional predictor in the regression models did not change the pattern of results. Hence, differences in musical experience are unlikely to account for the observed group differences.
A more interesting explanation is that a dual task (i.e., tapping + listening to speech) during the continuation phase affected French and German speakers differently in our experiment. Generally, as already shown in Falk and Dalla Bella (2016), dual task conditions when auditory and motor rhythms are aligned lead to enhanced detection performance compared to a single task perceptual baseline. This finding indicates efficient integration of motor with speech accent information when both are temporally aligned. Similar beneficial multisensory effects of manual gesturing were reported in speech perception (Holle et al., 2012; Biau and Soto-Faraco, 2013; Ito et al., 2014). In contrast, incongruent alignment, by temporally dissociating prominent verbal and motor information, creates less ideal conditions for efficient auditory-motor integration. Although, a stable rhythmic relation may still be established between speech and motor rhythms when finger taps are cued to occur at the anti-phase between accents (see Vos and Helsper, 1992; Volman and Geuze, 2000; Repp, 2005), more difficulties in maintaining a stable relation may be encountered by participants. In our study, both higher motor variability in the continuation phase and lower continuation consistency were indicators of increased difficulty (or lower stability) in this condition. On the other hand, it appears that destabilizing the speech-motor relation in the incongruent alignment condition showed interesting differences between language groups, possibly because listeners had to readjust their coordination with the auditory stimulus. An important difference was the remarkable tapping deceleration found in French, but not in German speakers. One possibility is that, in the German material, the alternating strong-weak pattern made it easier for listeners to track strong syllables as reference points when tapping on the weak syllable between them. In the French material, the ternary pattern may have made it more difficult to find the anchor for tapping between two metrically strong syllables. However, there is another explanation that takes accentual properties of the two languages into account. Previous studies reported that a concurrent speech or musical rhythm (distractor sequence) makes listeners deviate from a synchronized tapping pattern to a metronome (target sequence; Dalla Bella et al., 2013). In this kind of task, a lagging distractor rhythm typically leads to positive asynchronies in concurrent synchronization performance (Repp, 2003; Repp and Penel, 2004). Similarly, positive asynchronies were observed in our experiment during incongruent continuation tapping, particularly for the French speakers that also showed significant deceleration. This suggests that properties inherent to the language-specific metrical pattern may have attracted the French speakers more to the upcoming speech accent(s) than the German speakers. Final accents (right-edge of the AP) could have played a major role in this process. Indeed, these final accents feature higher and steeper rises and vocalic lengthening and appear to be more prominent than initial accents (left-edge of the AP, e.g., Rolland and Lœvenbruck, 2002). In addition, in corpora of colloquial French, initial accents are more optional and seem to depend on stylistic variation in contrast to the obligatory and highly predictable phrase-final accents (e.g., Astésano, 2001). This imbalance in accentual acoustics and predictability may have led French speakers to perceive more prominently the periodicity of primary (i.e., final) accents occurring at the right edges of the AP in our study. These accents occurred at a periodicity of 1,200 ms, and thereby, on a higher metrical level compared to the 600-ms periodicity of both initial and final accents. Note that, in a previous study, French speakers were also more likely to spontaneously tap to a higher periodicity in metrical speech compared to English speakers, although it was not possible to identify the specific linguistic landmarks attracting the tapping in this study (Lidji et al., 2011). In sum, tapping deceleration for French speakers observed in the incongruent condition may have been driven by attending to the higher metrical level of final accents (i.e., to a higher periodicity). Moreover, as the verbal changes exclusively occurred on initial accents, participants with higher attraction to final accents may have displayed worse detection performance. Future studies, placing verbal changes in final and initial AP positions, will help to examine this possibility. Alternatively, including languages with different accentual properties in a future study (such as Spanish, which possesses lexical stress, but different acoustic and positional properties than French and German, e.g., Toro et al., 2009) may further clarify the potential of motor alignment tasks to inform us about the perception and adaption to accentual salience.
Finally, it is noteworthy that the congruent alignment condition did not produce a similar drift in French speakers. Previous synchronization studies using finger tapping have shown that 1:n metrical subdivisions are generally not deleterious to synchronization capacity, rather the contrary, provided that different periodicities are coupled in a hierarchical system (Large and Jones, 1999; Large and Palmer, 2002; Repp, 2008). Thereby, the congruent condition may have provided a metrical template with clear nesting of temporal reference frames during auditory-motor integration for French speakers even when they were attracted by the higher metrical level.
To conclude, differences in auditory-motor coordination between French and German speakers open interesting perspectives for further investigating differences in accent perception and metrical reference frames for multisensory processes. Overall, our results support recent models of expectancy-driven speech perception underscoring the role of the motor system. Our findings are compatible with Kotz and Schwartze's (2010, 2016) subcortical-cortico framework for speech perception. The model encompasses a few critical regions typically involved in motor control, such as the (pre)supplementary motor area and basal ganglia and cerebellar circuitries. According to the model, the motor timing elements together with the other elements of the neural network converge in their function to optimize predictive timing of verbal behavior. Thereby, the network serves to precisely coordinate speech perception and production in time, and also explains the role of auditory-motor coupling and learning during language acquisition. Thus, motor benefits in speech perception such as found in our study may be the expression of an enhanced stimulation of these underlying neural mechanisms of predictive timing.
Finally, our findings are likely to extend to those ecological settings, whereby metrical speech and auditory-motor interactions are used to foster verbal memory and learning. Children's lore, in particular, exploits metrical speech with synchronized movements (e.g., hand clapping, stamping) in a wide variety of games that enhance children's social and verbal skills (Opie and Opie, 1988). Oratory and joint speech in groups are other domains in which auditory-motor coupling with speech is naturally found which may help to convey verbal messages to a group and improve social cohesion and inter-subjectivity (e.g., Cummins, 2014). Ultimately, our results may encourage novel multisensory rhythm-based interventions that are currently under investigation for fostering language acquisition and learning in developmental populations with speech and language disorders, such as dyslexic children (Schön and Tillmann, 2015) or autistic children (Wan et al., 2011). To conclude, the present results help to advance our understanding of the language-specific bases and learning of synchronized auditory-motor rhythms and predictive timing in multisensory speech processing.