When individuals engage in social interactions, these exchanges are naturally organised into distinct segments known as turns, representing the time during which one person speaks before the other takes the floor. Early studies have looked at conversational partners’ gaze in relation to their alternating roles as speaker and listener (as in Goodwin, 1981, which we also refer to as primary and secondary speakers, respectively, since the listener in fact speaks when giving vocal feedback). These studies claim that in dyadic interactions, people tend to look at the other participant more when they are listening than when they are speaking (Argyle and Cook, 1976, among many others). Kendon (1967, 1990) provided a more precise description of the different patterns of speaker and listener gaze: Listeners tend to maintain long gazes at speakers, interrupted by brief glances away, while speakers alternate gazes towards and away from the listener with approximately equal gaze durations.

In addition to these general gaze patterns, researchers have looked more specifically at what happens at crucial conversational moments such as turn transitions. In speech, turns have generally been found to transition from one speaker to another with brief gaps and few overlaps (Levinson and Torreira, 2015). This pattern, which appears to be highly robust in the face of individual, methodological, and contextual variation, raises interest in the nature of the factors that regulate this remarkable synchronisation between interlocutors.

The multimodal nature of turn transitions is widely attested (see, for example, Kendrick et al., 2023 for the effect of manual and gaze signals on turn transitions), as is the role of eye gaze in facilitating turn-taking (see Degutyte and Astell, 2021 for an extensive review). This was established in the seminal work of Kendon (1967), who has been highly influential in understanding the role of eye gaze at the start and end of a speaker’s turn. In a corpus of spontaneous conversations, he found that over 70% of utterances began with the speaker looking away from the listener, while over 70% ended with the speaker looking at the listener. Kendon argued that at the beginning of their turn, participants in a conversation tend to avert their gaze (i.e., they look away from their interlocutor). This serves two functions: first, indicating that they are focusing internally, concentrating on formulating their thoughts, planning their speech, or recalling information before they begin to speak; and second, functioning as a signal to others that they intend to take the floor by initiating a speaking turn. Thus, by averting gaze, the participant who currently holds the conversational floor signals that they are not available as a listener.

These observations were confirmed by a number of subsequent studies on dyadic exchanges (e.g., Cummins, 2012; Duncan, 1972; Duncan and Fiske, 1977; Ho et al., 2015; Oertel et al., 2012; see also Jokinen et al., 2009 for natural three-party conversations). For instance, Novick et al. (1996) found a systematic temporal alignment between gaze and speech during turn-taking. In that study, dyads were recorded while playing guessing games using an eye tracking device. They found that at a turn transition, the speaker ends their turn looking at the listener, and the listener begins to speak with an averted gaze. Although Kendon’s observations have been confirmed by the aforementioned studies, a number of studies have attributed gaze behaviour during turn-taking to different factors (Beattie, 1978; Rossano, 2012; Rutter et al., 1978; Streeck, 2014). Beattie (1978) argued that the speaker’s gaze away during early utterance production (and during re-engagement at the end of the utterance) is solely driven by the necessity to reduce cognitive load and does not have any regulatory role during turn-taking. Alternatively, Rossano et al. (2009) and Streeck (2014) claimed that gaze does not facilitate turn-taking as such, but it does facilitate the organisation of complex actions that might require multiple turns to complete.

At the end of a turn, Kendon (1967) proposed that participants gaze towards the listener with the function of checking their availability as the next speaker and, thus, as a signal of turn-yielding. This proposal is strongly supported in later studies (Novick et al., 1996; Rutter et al., 1978; Lerner, 2003; Jokinen et al., 2009, 2013; Ho et al., 2015; Brône et al., 2017; Auer, 2018; Blythe et al., 2018; Streeck, 2014; Oertel et al., 2012; Kawahara et al., 2012; for group conversations see also Harrigan and Steffen, 1983; Kalma, 1992). However, Kendon’s findings were not supported across the board in this context either. Rutter et al. (1978) commented that Kendon’s predictions about floor changes can only occur if the listener is also looking at the speaker, that is, in the case of mutual gaze between the two participants. Moreover, Novick et al. (1996) observed two prevalent patterns of gaze direction during turn-taking, categorised as ‘mutual-break’ and ‘mutual-hold’. At the end of an utterance, the primary speaker (the participant currently holding the floor) looks at the secondary speaker (the participant currently listening), at which point the gaze is momentarily mutual until the secondary speaker breaks the mutual gaze and starts to speak, hence ‘mutual-break’, which is the most frequent pattern. Alternatively, in instances of ‘mutual-hold’, mutual gaze is held while the secondary speaker starts speaking without immediately averting their gaze. Beattie (1978), in fact, found the converse to Kendon (1967), with more immediate speaker switches when utterances terminated with no gaze than with gaze, and no differences in the effect of gaze on the number of immediate speaker switches for complete utterances, as might be expected if gaze served a floor-allocation function.

Degutyte and Astell (2021) argued that different results relative to turn boundaries and gaze might be due to different experimental designs and different types of interaction (e.g., dyadic vs. triadic/multiparty interaction, question/answer sequences vs. other types of adjacency pairs in free conversation) and methodology (see also considerations of the same kind in Spaniol et al., 2023). In addition, many individual factors can affect gaze behaviour during conversation (gender in Argyle and Dean, 1965; Myszka, 1975; Bissonnette, 1993; social status in Myszka, 1975; Foulsham et al., 2010; acquaintance status in Rutter et al., 1978; Bissonnette, 1993; and cultural background in Rossano et al., 2009).

While there is extensive research on eye gaze during turn alternations, there is less research on how eye gaze interacts with other aspects of turn alternation. One such aspect is turn-regulating vocal feedback, which is discussed in the next section.

The speaker’s planning of their contribution to a conversation is significantly influenced by a crucial variable: the listener’s reaction to their utterance, realised through vocal feedback signals. While often overlooked in conversation (Shelley and Gonzalez, 2013), these vocal feedback signals play a pivotal role in guiding verbal exchange, indicating the listener’s attitude towards what is being said and conveying intentions related to floor management. Commonly referred to as ‘backchannels’ or ‘response tokens’, these signals are argued to enhance fluency in social interactions by supporting the ongoing turn of the interlocutor (Amador-Moreno et al., 2013) and structuring dyadic conversations (Kraut et al., 1982; Sacks et al., 1974; Schegloff, 1982).

There is no consensus in the literature about their definition (Rühlemann, 2007). Fries (1952) is credited as one of the earliest to identify these ‘signals of attention’ which do not disrupt the speaker’s discourse in telephone conversations. Over time, various terms have been employed to characterise this phenomenon, including ‘accompaniment signals’ (Kendon, 1967), ‘receipt tokens’ (Heritage, 1984), ‘minimal responses’ (Fellegy, 1995), ‘reactive tokens’ (Clancy et al., 1996), ‘response tokens’ (Gardner, 2001), ‘engaged listenership’ (Lambertz, 2011), and ‘active listening responses’ (Simon, 2018). Yngve (1970) introduced the term ‘backchannel communication’ to distinguish the primary channel used by the speaker holding the floor from the one used by the listener to convey essential information without actively taking a turn. Essentially, backchannels represent tokens used to express acknowledgement and understanding, encouraging the main speaker to continue (e.g., Hasegawa, 2014, among others).

Later, Jefferson (1983) claimed that the use of such ‘acknowledgement tokens’ does not exclusively signal turn-yielding but can also cue the intention of taking the floor after acknowledging the current speaker’s turn. In particular, she observed that some token types “exhibit a preparedness to shift from recipiency to speakership” (1983:4), whereas others are more systematically used as turn-yielding signals. For English, she noticed that speakers tend to use tokens such as ‘mh-mh’ to signal the intention of acknowledging without taking the floor, whereas tokens such as ‘yeah’ when a recipient is moving into speakership. In line with this distinction, some later studies (Drummond and Hopper, 1993; Jurafsky et al., 1998; Savino, 2010, 2011, 2012; Savino and Refice, 2013; Wehrle, 2023; Janz, 2022; Sbranna et al., 2022; Spaniol et al., 2023; Sbranna et al., 2024) categorised backchannels based on two turn-taking functions, that is, passive recipiency (henceforth PR) and incipient speakership (henceforth IS). Passive recipiency involves tokens produced without the speaker taking the floor, serving as acknowledgements and continuers (i.e., backchannels, as intended by Yngve, 1970). Incipient speakership refers to tokens used by a speaker to acknowledge the interlocutor’s turn before producing a new turn themselves, realising a turn transition. This classification has also been partially supported by intonation research, which has observed a tendency for certain tokens and intonation contours to co-occur with these functions (Savino, 2010, 2011; Sbranna et al., 2022; Sbranna et al., 2024).

In this study, we adopt this operationalisation of the feedback turn-taking function, as it accounts for the role that vocal feedback signals play in the complex and multidimensional turn management system, alongside feedback signals of a different (i.e., visual) nature.

Given that feedback is meant to support the ongoing turn of the primary speaker, this speaker can actually invite the secondary speaker to produce feedback. This invitation is multimodal in nature and can be signalled in different ways, for example, by inserting a pause by intonation, or head or gaze movements. Heldner et al. (2013) proposed that the primary speaker provides “backchannel relevance spaces” to allow the listener to insert backchannels, although in their corpus of face-to-face interaction, the number of such spaces was considerably greater than the actual backchannels provided. The authors argue that not all backchannel relevance spaces are filled (either with vocal or visual feedback) as listeners choose feedback positions that actively support speakers in constructing their discourse.

Some studies on the interaction between vocal feedback and, in particular, eye gaze have shown that vocal and non-vocal feedback signals are used to show acknowledgement and understanding of the primary speaker without taking the floor—those which we refer to as PR tokens, and Heldner’s ‘backchannels’ (2013)—occur during mutual gaze between a listener and a speaker, in accordance with a number of other studies (Kendon, 1967; Bavelas et al., 2002; Eberhard and Nicholson, 2010; Cummins, 2012; Oertel et al., 2012). For example, Oertel et al. (2012) compared gaze behaviour in dyadic interactions at turn transitions during speech overlap, in silence, and in the vicinity of backchannels and found that the production of vocal feedback signals is associated with an increase in mutual gaze. At a more detailed level, Bavelas et al. (2002) found that gaze patterns used to coordinate feedback signals in dialogue are often initiated by the speaker gazing at the listener, who then looks back, resulting in short periods of mutual gaze broken by the listener looking away shortly afterwards. The authors precisely examined when and how listeners insert their feedback into a speaker’s narrative by involving participants in a storytelling task. A collaborative approach would predict a relationship between the speaker’s acts and the listener’s responses, and the authors proposed that gaze coordinates this collaboration. They found that the listener typically looks at the speaker more often than the other way around. However, at key points in their speech, the speaker seeks a response by looking at the listener, creating a brief period of mutual gaze, called “gaze window” by the authors. They observed that, during gaze windows, the listener was very likely to provide a reaction, such as a vocal ‘mhm’ or a nod, after which the speaker interrupted the gaze window by quickly looking away and continuing to speak. This pattern aligns with previous studies on the use of multimodal cues in the perception of interrogativity, which show that gaze direction towards the interlocutor and other gestural signals enhance the perception of a response being expected (Borràs-Comes et al., 2014).

Although mutual gaze has been found to be a strong predictor of feedback signals independent of their modality (Ferré and Renaudier, 2017; Hjalmarsson and Oertel, 2012; Poppe et al., 2011), a difference has been reported between vocal and non-vocal (i.e., head gestures) feedback signals, with non-vocal signals being more often produced in correspondence with mutual gaze than vocal signals (Ferré and Renaudier, 2017; Eberhard and Nicholson, 2010; Truong et al., 2011). This is probably because vocal signals do not need to be conveyed through the visual channel. Moreover, gaze was found to be sustained more often throughout a sequence that contained a visual backchannel than in a sequence containing a verbal backchannel (Ferré and Renaudier, 2017). The strong correlation between being looked at and gestural backchannels, but not vocal ones (Bertrand et al., 2007), was motivated by the idea that gaze might establish a communication mode between interlocutors. Nonetheless, in contexts where the primary speaker was not gazing at the interlocutor, both gestural and/or vocal backchannels were produced.

Ondáš et al. (2023) also looked at the temporal detail of the interval between the start of the gaze directed towards the interlocutor and the subsequent backchannel in an interview scenario and found that, in most cases, the moderator received vocal feedback from his guest within 500 ms of initialising direct eye contact. The second most probable time interval was 1,500–2000 ms. This same interval was found to be the most probable in the opposite scenario, that is, the moderator providing feedback to the guest. One possible speculation on this different behaviour is that the moderator is meant to lead the conversation, thus getting quicker reactions from the guest, who is expected to inform the moderator that they acknowledge and follow the dialogue structure. However, this result is linked to a very specific conversational format, which probably has its own dynamics that are different from other formats of conversation. Moreover, Kendrick et al. (2023), analysing the impact of visual cues in turn-timing, found that turn transitions were sped up by the use of manual gestures but not by gaze behaviour.

These studies focused on the interaction between eye gaze and backchannels, accounting only for their PR function. To our knowledge, only one recent exploratory study (Spaniol et al., 2023) has investigated the interplay of gaze patterns and turn-regulating vocal feedback using the same PR vs. IS paradigm for vocal feedback. In particular, the authors analysed gaze and vocal feedback behaviour in dyadic conversations across different communicative contexts, namely, (1) a free conversation in which participants get to know each other, (2) a Tangram game-based interaction, and (3) a free discussion about the game participants had just played together. In contrast to previous findings, they found that in the task-based context, vocal feedback was mostly provided in conjunction with averted gaze, independently of its turn-regulating function. They explained their results by the presence of a visual competitor during the conversation, that is, the task material. In the other two contexts of free conversation, three out of four dyads produced PR feedback signals more under gaze directed to the interlocutor than IS feedback signals, showing that the turn-regulating function of backchannels distributes differently on gaze behaviour patterns and that this distribution is influenced by the context of the conversation.

Another important question for the present study concerns the use of speech and gestures when the interlocutor is not visible.

Some studies have investigated the use of manual gestures in conversations, with and without visibility between participants, concluding that gestures have both communicative and internal cognitive functions. For example, Alibali et al. (2001) found that participants were still using representational gestures, that is, gestures depicting semantic content related to speech, when an opaque panel hindered mutual visibility, although to a lesser extent than in visibility conditions. Interestingly, the number of beat gestures, that is, simple rhythmic gestures lacking semantic content, remained consistent across both conditions. Bavelas et al. (2008) compared the use of gesture in face-to-face and telephone conversations and concluded that, instead of being incidental, the gestures made while talking on the phone appeared to be a deliberate adjustment to a dialogue without visual contact. Indeed, both the visibility and non-visibility groups gestured at a high rate, although in the visibility condition, participants gestured with forms and relationships with words that were more informative for their listeners. These studies point to similar conclusions: first, that gestures are adjusted to the interlocutor’s need; second, that gestures fulfil not only a communicative function but also an internal cognitive function (see also McNeill, 2005).

Based on these findings, an interesting question is whether these results apply to eye gaze. Argyle et al. (1973) sought to distinguish the various functions of gaze by utilising a one-way screen to manipulate the conditions of speakers observing or being observed. They found evidence of gaze having monitoring and signalling functions. In the first case, they observed that individuals who could see through a one-way screen exhibited increased gaze while speaking compared to those without visual access; in the second case, they observed that even the individuals who could not see their interlocutor still directed their gaze towards the other person occasionally while speaking. Indeed, several studies have found that, even in the absence of visibility, listeners are particularly sensitive to sound sources directed straight at them and are able to recognise the speaker’s head orientation, that is, the situation corresponding to mutual gaze in visible conditions (Edlund et al., 2012; Kato et al., 2010; Nakano et al., 2008).

In a study examining mutual gaze patterns in dyadic conversation under light and darkness conditions, Renklint et al. (2012) found a notable reduction in instances of mutual gaze in the absence of light. To interpret this result, the authors claimed that mutual gaze is “made for the other person to see” (aligning with the perspective of Bavelas et al., 1992:483) and that gaze extends beyond mere learned behaviour, having its own functions and meanings. Moreover, the authors noticed that in dyadic conversations, the addressee is consistently the only other participant in the conversation, and given an implicit mutual agreement about an alternation of speakers in a conversation, selecting the next speaker becomes a mere formality. This is very different from multiparty conversations, in which gaze plays a significant role in turn alternation. Thus, they suggested that in dyadic conversations, information derived from linguistic and phonetic aspects (e.g., sentence structure and prosody) provides enough cues to predict turn alternation, making mutual gaze less essential for successful communication. From these studies, it appears that (i) gaze clearly has a dual function: sensing (i.e., sampling information from a visual scene) and signalling (i.e., not just to gain information from the environment, but with an explicit and deliberate communicative goal) (Argyle and Cook, 1976; Gobel et al., 2015; Risko et al., 2016; Cañigueral and Hamilton, 2019), and (ii) in the absence of visibility, gazing at the position where the conversational partner is known to be is reduced but is not absent. This might be due to the fact that in such a condition only, the sensing function of gaze applies.

The effect of non-visibility also relates to the fluency of the speakers’ individual speech and the turn alternation system, both of which vocal feedback crucially contributes to. Beattie (1979) reported that the absence of gaze leads to an increased use of filled pauses (see also Cook and Lalljee, 1972) and interruptions (Argyle et al., 1968). Similar results were obtained by Boyle et al. (1994), who studied the effect of visibility on task resolution. The authors found that in the visibility condition, there is greater efficiency in the dialogues attributed to the exchange of visually transmitted non-verbal signals, while in the non-visibility condition, oral communication is characterised as having greater flexibility and versatility. In the latter case, participants tried to compensate for the absence of the visual channel by interrupting their partners more frequently and using more vocal backchannels to support the primary speaker. In addition, Neiberg and Gustafson (2011), comparing eye-contact to non-eye-contact conversations, found an increase in overlapped turn transitions in the absence of visibility, which they explain by a possible increase in unintentional interruptions. From these findings, it seems that visibility improves the smoothness of the dialogue flow, but, in turn, in the absence of visibility, speakers rely more on vocal resources, especially feedback signals.

We analysed interactions from an Italian corpus of dyadic task-oriented conversations (described in Savino et al., 2018). As this is the first multimodal analysis of the corpus, the corpus and setup are described in detail below.

Annotation of eye gaze was performed for rounds 1, 2, 9, 10, 21, and 22, for each dyad in both conditions. The selected game rounds correspond to the beginning (rounds 1–2), medial part (rounds 9–19), and final part (rounds 21–22) of the whole game session. This selection was made to control for any effect of (non-)familiarity with the task on participants’ communicative style. The total number of rounds included in the present multimodal analysis corresponds to 1.47 h of conversation.

To answer our research questions (RQs), we analysed the co-occurrence of vocal feedback signals with gaze directions of both the feedback producer, or “secondary speaker,” and feedback receiver, or “primary speaker” (Section 4.1). To do so, for each participant’s vocal feedback signal, we extracted the overlapping gaze direction intervals by both the feedback producer and feedback receiver (RQs 1.a and 1.b, Section 4.1.1) and measured the duration of these intervals up to the moment in which the vocal feedback signal is uttered (RQ 1.c, Section 4.1.2). This duration is operationalised as the time window from the onset of the overlapping gaze direction interval to the onset of the vocal feedback signal. We also extracted the gaze direction intervals overlapping the offset of each feedback signal to establish whether the production of vocal feedback would prompt a shift in gaze direction in any of the two participants (RQ 1.d, Section 4.1.3).

Moreover, we measured the overall amount and duration (in seconds) of gaze direction intervals (RQ 2.a, Section 4.2), as well as vocal feedback rate, operationalised as the number of feedback occurrences per minute of dialogue (RQ 2.b, Section 4.3).

The results of all analyses are presented across visibility conditions (RQ 2.c throughout the result sections for conciseness), i.e., eye-contact (EC) and non-eye-contact (NEC). The data that support the findings of this study, as well as the code used to perform the analysis, are openly available in the accompanying repository at https://osf.io/3dqzk/.

We found 1,649 gaze direction intervals in the eye-contact condition (EC; 80% of the total) and 354 gaze direction intervals in the non-eye-contact condition (NEC; 20% of the total), showing that, in line with our expectations, participants change the direction of their gaze far less often when eye contact is inhibited.

However, the relative proportions of gaze direction intervals directed to “Task” or “Addressee” displayed in Figure 7 (y-axis) do not differ greatly across conditions. In the EC condition, there is an almost equal proportion of gaze occurrences to “Task” and “Addressee” (50.5 and 49.5% respectively). In the NEC condition, where the interlocutor is behind an opaque panel, the proportion of gaze to “Task” increases only slightly (60.7%) as compared to the EC condition. On the other hand, the proportion of gaze to “Addressee” (39.3%), although lower than in the EC condition, indicates that participants gaze away from the task quite often, looking at the panel, that is, in the direction of the interlocutor’s voice, namely, where the addressee is known to be sitting.

Mean gaze duration (Figure 8) shows that the time spent looking at the “Task” is longer than that spent looking towards the “Addressee” in both conditions (4.75 s in EC and 18.2 s in NEC). However, the time spent looking at the “Task” shows high variability and can potentially be longer, especially when eye contact is inhibited (see large error bars especially for NEC). There is less variability in the time spent looking towards the “Addressee,” with a mean duration of 1.55 s in the EC condition and 0.74 s in the NEC condition, especially in the NEC condition (small error bars).

In line with our expectations, these two metrics taken together show that the lower number of gaze direction intervals in the NEC condition could be explained by the fact that participants gaze for longer periods at the task than in the EC condition, leaving little room for switching gaze direction.

Finally, Figure 9 shows the rate of vocal feedback production across pragmatic functions and visibility conditions. These values indicate that the vocal feedback rate is higher in the NEC than in the EC condition, as expected. Across feedback functions, our analysis reveals a proportionally higher rate (almost double) of PR than IS vocal feedback expressions.