Research into the use of multimodal data for analyzing learner's emotions, particularly in educational contexts, has evolved significantly since the 2000s. The idea of integrating physiological and behavioral data to understand dynamic interactions between motivation, emotions, and Willingness to Communicate (WTC) across educational environments is not new (De Bot et al., 2007) but technological developments have allowed researchers to use sophisticated tools that are increasingly affordable to applied linguists. Here, we review some key studies.

D'Mello et al. (2012) explored the use of multimodal data to monitor engagement and learning in real-time. This research utilized a combination of EFE, body posture, and interaction logs to understand how students engage during learning activities in various environments, including classrooms and online platforms. The researchers used advanced and expensive equipment which would be unaffordable to most researchers.

Another study, conducted by Tonguç and Ozkara (2020), employed a cheaper facial recognition tool with 67 students during basic information technology courses, using a camera placed in each student's computer. The materials used during the lecture were reflected on students' computer screens. EFE were analyzed and digitized to identify seven emotions/feelings: disgust, sadness, happiness, fear, contempt, anger, and surprise based on Ekman's theory of universal emotions (Ekman, 1992). This approach has been strongly criticized for being too static and essentialist. Barrett (2017) and Gendron et al. (2018) have proposed an alternative approach, the theory of constructed emotions which posits that emotions are shaped by the broader cultural context and the more specific social context in which the emotion unfolds. They also reject the idea that emotions have universal unique “fingerprints,” in other words, a smile does not automatically reflect happiness and a scowl does not always signify anger. The dynamic theory of constructed emotions is particularly appropriate when studying multilingual and multicultural individuals.

Adopting a dynamic approach, Gregersen et al. (2014) conducted a multimodal study on three high-anxiety and three low-anxiety learners of Spanish FL [based on scores on Horwitz et al. (1986) FL Classroom Anxiety Scale]. The authors combined physiological data (heart rates), idiodynamic data (anxiety ratings), and interviews about the fluctuations. Participants did a 3-minute oral presentation in Spanish. They wore heart monitors and, immediately after the task, they provided 42 anxiety ratings times on a scale from +5 to –5 while viewing their presentation on a computer. They then explained to the researchers why the spikes and dips in anxiety had occurred. Increased heart rates were positively correlated with anxiety ratings. The high anxiety participants reported difficulties in vocabulary retrieval as the main cause for their anxiety. Low anxiety participants used strategies to mitigate this. They had practiced the presentation in the preparation stage rather than attempting to memorize it. The study used various sources of data to highlight the dynamic nature of a single emotion but ignored the fact that participants may have experienced other emotions, which could also have affected heart rates.

The pioneering neurological study by Nozawa et al. (2021) was the first to peer into learners “black box,” namely their brain waves, as they were performing tasks in class. The researchers focused on two intact English FL classes in Japan with two groups of four learners each (totaling 16 students). They adopted a multimodal approach, examining the interbrain synchronization among learners working in pairs and the similarity of the flow state dynamics during collaborative learning. Prefrontal neural activities were measured using a wireless functional near-infrared spectroscopy device placed on the students' heads. Additionally, the researchers asked the learners to watch recorded videos of the classes and to evaluate their own flow levels on a scale from 1 to 7 every two minutes. The study required advanced technology and know-how to process the neurological data in a single cortical area. The authors admit that the correlation between self-reports and interbrain synchronization does not imply causation as there may be “hidden variables.” Dewaele (2023) explained there are different recent studies which try to: “catch the elusive emotional phenomena that swirl around task performance.” (p. xviii). Lambert et al. (2023) propose to resort to electroencephalography (EEG), eye tracking (ET), electrodermal activity (EDA), and automated facial expression analysis (FEA). MacIntyre (2023) revisits and develops his idiodynamic method, which allows for detailed and accurate observation of the emotional fluctuations experienced by learners while performing tasks in language classes.

These above studies highlight the diversity of methodologies and technologies used in the multimodal analysis of learners' emotions in the FL classroom. All relied on supervised learning approaches that require extensive annotation and labeled data, which can be resource-intensive, time-consuming and expensive. In contrast, our approach employs unsupervised Gaussian Mixture Model (GMM) clustering, which identifies specific moments without requiring pre-labeled data. This allows us to account for individual differences in learners' physiological and emotional patterns while reducing the need for extensive manual labeling, making the methodology more efficient and adaptable.

The introduction of positive psychology in the field of FL acquisition with the publication of MacIntyre and Gregersen (2012) made researchers aware that there had been a long-term exclusive focus on negative emotions in FL classrooms (Petiot and Visioli, 2022) and anxiety in particular. Its popularity among researchers had been boosted by Horwitz et al. (1986) whose 33-item Foreign Language Classroom Anxiety Scale (FLCAS) covered physical symptoms of anxiety, nervousness and lack of confidence in the FL class. They defined FLCA as “a distinct complex of self-perceptions, beliefs, feelings, and behaviors related to classroom language learning, arising from a uniqueness in the language learning process.” The main cause of FLCA is the inability to project an accurate image of themselves in the FL and the fear of coming across as clumsy and inauthenthic (Horwitz, 2017). FLCA grows gradually through repeated anxious experiences in the FL classroom. As such, FLCA starts as being a situation-specific state and it gradually becomes more stable and trait-like (Horwitz, 2017).

A meta-analysis by Botes et al. (2020) of 67 studies based on the FLCAS has shown that FLCA is moderately negatively linked to FL performance and progress. High FLCA was linked to lower general academic achievement, speaking, listening, reading and writing performance in the FL. Students suffering from high FLCA have lower degree of Willingness to communicate (WTC) and may even prefer to hide and remain silent in the classroom (King, 2013). This withdrawal from classroom interactions slows their progress in the FL.

Another negative emotion, frequently present in FL classrooms, is boredom (Pawlak et al., 2024). Li et al. (2023) described boredom as being characterized “by negative valence, low arousal and being achievement-related activity-focused.” It emerges when a classroom activity or task is perceived as irrelevant and when learners feel helpless and fatigued because the activity is either too easy or too difficult (Agrawal et al., 2022; Li et al., 2023). Bored learners lose their confidence and suffer from a perceived lack of control. This lowers their WTC, and undermines both their short-term and longer-term motivation and well as their overall engagement in the FL activities. Learners' boredom can also originate in the teacher's inability to hide their own boredom (Pawlak et al., 2024). Li et al. (2023) developed 32-item Foreign Language Learning Boredom (FLLB) scale consisting of 7 factors. The first factor was named Foreign Language Classroom Boredom (FLCB) and consisted of 8 items. FLCB has been used independently in later research. Dewaele and Li (2021) showed that teacher enthusiasm can counter learners' FLLB, increase their enjoyment and stimulate their engagement. Li et al. (2022) found enjoyment and boredom to be strongly negatively correlated. Unsurprisingly, a negative relationship exists between FLLB and FL achievement (Li and Dewaele, 2024).

Researchers increasingly agree that positive emotions such as Foreign Language Enjoyment (FLE) should be part of a more holistic picture and they reject the deficit view of FL learners Dewaele and MacIntyre, (2014). The authors referred to Csikszentmihalyi (1990) who noted that enjoyment (and sometimes flow) emerges when a person manages to complete a challenging task, reach a state of full concentration, perform a task with clear goals, and receiving immediate feedback on the performance. (Dewaele and MacIntyre 2014) developed the 21-item FLE scale, which was followed by a shorter 9-item psychometrically validated scale (Botes et al., 2021). While a majority of the longitudinal studies on FL emotions focused on change over a period of weeks and months, a smaller number of studies have focused on fluctuations over shorter time spans. Boudreau et al. (2018), for example, used the idiodynamic method to investigate second per second fluctuations in FLE and FLCA. Anglo-Canadian participants completed a one-minute speaking task in French FL and then watched the recording and reported their levels of both emotions for every second. Values were found to vary considerably during that minute and were later commented on by participants who pointed to linguistic difficulties or to fleeting moments of anxiety, enjoyment or boredom. Elahi Shirvan and Talebzadeh (2018) also used the idiodynamic approach to investigate the fluctuations in FLE of 7 Iranian EFL university learners participating in conversations on simple and more difficult topics. The results showed strong intra- and inter-individual variation linked to the conversational topics. Adopting a multi-case study design, Elahi Shirvan et al. (2020) investigated fluctuations in FLE over different time spans, ranging from seconds with the idiodynamic method, to minutes, weeks and months. The researchers used low tech “Enjoymeters” (pieces of paper with thermometer-shaped figures ranging from 0 to 10 indicating the level of FLE) to capture variation in FLE for periods of 5 minutes. Participants were two Iranian EFL students in the classroom. FLE among these two students was found to fluctuate differently over different timespans. The variation was found to be linked to unique social and personal factors, such as the ability to be creative, the appropriate challenge, the opportunity for authentic communication in English with peers, the teacher's ability to be supportive, humorous and establishing a positive classroom climate. Being laughed at by peers for making a mistake could cause a sudden drop in FLE and a spike in anxiety. Bielak and Mystkowska-Wiertelak (2024) also used the idiodynamic methodology to investigate fluctuations in FLCA and FLE in 10 Polish EFL learners working in pairs and group. Their interactions were video recorded and then viewed and rated second per second for FLE and FLCA. In later stimulated-recall interviews, they discussed the causes of the fluctuations and the emotion regulation strategies they deployed to control them. The two emotions showed brief periods of stability followed by highly idiosyncratic levels of fluctuation. Levels of FLCA were found to fluctuate more than FLE but the triggers for the fluctuation in both emotions overlapped substantially. Causes for FLCA included the awareness of having made specific errors, frustration at the lack of linguistic sophistication, and ignoring task instructions. FLE was found to be linked to the quality of the peer's performance and a productive collaboration. Taking the floor and deploying new knowledge caused peaks in both FLE and FLCA while yielding the floor cause a dip in both emotions.

Dewaele and Pavelescu (2019) used a multiple case study approach to investigate the relationship between FLE, FLCA, and WTC in two Romanian secondary school EFL learners. Qualitative data included classroom observations and semi-structured interviews on the emotional sources of fluctuation in participants' WTC in the English classroom. FLE and FLCA were found be influenced by a range of contextual factors including seating arrangements, course material and conversation topic which shaped their WTC in dynamic and unique ways.

The meta-analysis by Dewaele and MacIntyre (2022) showed that FLE is strongly positively correlated with WTC. Moderate positive relationships emerged between FLE and FL academic performance. The crucial awareness that emerged from previous research is that learner emotions do not exist in isolation. Studies on large samples reveal positive correlations between positive emotions, and between negative emotions, combined with negative correlations between positive and negative emotions. This suggests that there is a strong probability that students who are enjoying themselves are less likely to suffer from anxiety and boredom. As the idiodynamic studies show, the emotions are constantly connected with each other, with motivation and all are linked to the immediate classroom environment and the wider social context (Dewaele and MacIntyre, 2022; Li and Dewaele, 2024; Waninge et al., 2014). The teacher is central in this context and his/her perceived enthusiasm or happiness can cause a wave of positive emotional contagion (Li and Dewaele, 2021; Moskowitz and Dewaele, 2021; Talebzadeh et al., 2020). A teacher who cannot hide his/her boredom will spread this negative emotion to students (Pawlak et al., 2024). The relationship with peers and with a specific partner in pair-work will also shape individual learner emotions. Collaborating with someone who is very anxious or bored will drag down the enthusiasm of learners who actually enjoy the activity. On the other hand, working with an empathic, friendly, funny partner might boost learners' positive emotions and lower their negative emotions. The task and activity itself will also shape learners' emotions, as they may -or may not- enjoy it and grow bored with it if it lasts for too long (Li and Dewaele, 2024).

To sum up, this short overview of the existing literature shows a field in rapid transition because of the emergence of a holistic understanding of learner emotions and of their dynamic nature, combined with technological innovations. Studies using the idiodynamic method used material collected laboriously over a period of no more than a few minutes in a lab. They also focused on no more than two emotions in order not to overwhelm participants. Very few of these studies included a physiological measure. We thus argue that there is an urgent need for multimodal studies focusing on a larger number of emotions tracked for longer periods in real-world FL classroom environments using economical and efficient techniques. Adopting a non-supervised approach relying on non-intrusive, low-cost instruments with robust data processing techniques could lead to the development of a scalable and replicable system.

The current study thus proposes to analyze emotions of a small number of participants over a period of several weeks. We adopted a design similar to that of Nozawa et al. (2021) with external technical or digital measurement instruments (pulse oximeter and camera placed in the classroom) and self-perception data collected from students (enjoyment, boredom, and anxiety questionnaires).

We will focus on the following research questions:

The project took place at the French language center for foreigners at the University of Angers. Participants were preparing for the University Diploma in French Studies (DUEF) at the beginner level, A2 according to the Common European Framework of Reference for Languages (CEFR). The courses were held over four days in the second semester of the academic year, from February to April 2023. The study was conducted over 16 sessions of 2 hours and 40 minutes each (2 × 1 h and 20 min). Of the eleven students in the class, seven provided consent; however, only those consistently present across the majority of sessions—defined here as attending at least 50% of the 16 total sessions—were included in the analysis (three students: pseudonyms “Mitch,” a 21-year-old American, attended 14 sessions out of 16 sessions; “Zeynep,” a 23-year-old Turkish, attended 14 sessions out of 16 sessions; and “Oksana,” a 23-year-old Ukrainian, attended 13 sessions out of 16 sessions). This choice reflects our methodological emphasis on individualized emotional patterns rather than aggregated group data. By concentrating on a small number of consistently present participants, we were able to collect extensive multimodal data—continuous HR monitoring, facial expressions, video recordings, detailed classroom observations, and post-session questionnaires—allowing for a high-resolution, longitudinal analysis of emotional dynamics over time. A research team member took comprehensive notes for all 16 sessions. During each session, both their HR signals and EFE data were collected, and at the end of each session participants completed a questionnaire. For all attended sessions, the three modalities—HR, EFE, and self-reports (SR)—were obtained; no sessions contained partial modality data.

This study was conducted in accordance with the Declaration of Helsinki and institutional ethical standards. All participants voluntarily participated and provided signed informed consent before data collection and analysis. They explicitly agreed to the recording of their sessions, the measurement of their heart rate, and responding to a survey at the end of each session for research purposes. All procedures were designed to respect participants' rights, privacy, and well-being. No identifying information was collected or published.

Three types of instruments were used to tap into learners' physiology, EFE and self-report of FLE, FLCB, and FLCA.

We first discuss the experimental results of the proposed pipelines presented in Figure 4, applied to the dataset, in relation to the performance of the HR features in detecting anomalous moments. Figures 5–7 illustrate the HR feature space following an unsupervised clustering via GMM applied to HR features extracted from 150-second segments. This interval was selected as it offered the best balance between temporal resolution and stability of HR features, as discussed in Section 3.4.1. In these figures, we can clearly observe the formation of two clusters for students 1 (Oksana) and 3 (Zeynep). However, student 2 (Mitch) presents a different pattern with a small number of outliers appearing far from the dense area (norm).

To evaluate the cohesion and separation of the clusters resulting from the unsupervised GMM, we use the mean of silhouette score metric over all samples (Rousseeuw, 1987). Specifically, for a sample i from the data, the silhouette score s(i) is calculated as follows:

where a(i) represents the average distance between i and all other points within the same cluster, capturing the intra-cluster cohesion, and b(i) denotes the minimum average distance between i and all points in any other cluster, representing the closest inter-cluster separation. The silhouette score s(i) ranges between –1 and 1, with values close to 1 indicating well-clustered samples, values near 0 suggesting boundary points between clusters, and values below 0 indicating possible misclassification of i to its assigned cluster. In Equation (1), i denotes a single sample data point from the HR feature space illustrated in Figures 5–7. The silhouette score s(i) is calculated for each of these points, and the overall silhouette score shown in Figure 8 corresponds to the average value mean(s(i)) computed across all such data points. Figure 8 presents the silhouette scores for each GMM clustering applied to the three students. Notably, it illustrates the dominance of HR acceleration μ_a and its normalized value μaσ in the clustering process, followed by μ_v and μvσ. suggesting that these four features play a significant role in differentiating between the SM of the students and their normal behavior. By focusing on these specific physiological parameters, it might be possible to improve the accuracy of clustering or classification models that rely on HR signals to infer the SMs of the student. This observation is further supported by findings in emotion classification using physiological signals (Mera and Ichimura, 2004), which demonstrated the superior contribution of these four features over the mean and the standard deviation of HR signal.

Integrating HR signals (component 1) and EFE (component 2) could improve the detection of students' activity and emotional states. Figure 9 illustrates the percentage of SM detected through the Intersection over Union (IoU) of HR signals and EFE for the three students. This metric quantifies the overlap between the specific moments identified by HR signals and those identified by EFE. The IoU percentages were computed by aligning SMs from both modalities on a common timeline and calculating the ratio of their intersection to their union. This alignment addressed the difference in temporal resolution between modalities, as HR SMs were derived from 150-second windows, whereas EFE SMs were computed at 1 frame per second, ensuring the comparison was not biased by sampling rate differences. It is important to note that only the anomalies in EFE present during the periods of HR signal recording were considered. For Mitch and Oksana, the IoU between the SM of HR signals and EFE is 21.05% and 20.24%, respectively. These moderate overlaps indicate some consistency between the HR data and EFE, although there may be variability in capturing the students' moments of activity and their emotional states. In contrast, Zeynep shows an IoU of 27.40%, the highest among the three students, suggesting a stronger convergence between HR signals and EFE for identifying moments of activity for this student. This comparison highlights that although there is some level of agreement between the two modalities for all students, the extent of this alignment varies, with Zeynep presenting the strongest correlation.

The results presented in Table 1 illustrate the effectiveness of the HR signal-based method for detecting students' activity and emotional states during teaching sessions. The table provides a comparative analysis of the percentage of convergence between the results by our HR signal-based method (C₁) and the results obtained by our EFE based method (C₂) and finally by the students' observations made during the 16 sessions by an expert in pedagogical methods (OBS).

In addition, fusing both modalities (HR signal-based and EFE-based) through decision fusion achieves the highest convergence rates with expert observations across all students. This outcome highlights that integrating both physiological and behavioral features enhances the accuracy of detecting students' activities and emotional states compared to using unimodal approach.

The questionnaires and open-ended responses, self-report (SR)(C₃) offered valuable insights into the overall emotions experienced by students during the classes, yet they lacked the temporal precision needed to pinpoint specific moments within a session. As a result, it was not possible to statistically cross-reference C₃ with C₁ or C₂, since the latter two modalities enable the identification of specific SMs at precise times during the class. Instead, the C₃ data were used to characterize students' global emotional states across sessions, providing a broader interpretive context for the statistical findings from C₁ and C₂, as discussed later for each student. In contrast, the OBS data—derived from the researcher's detailed classroom notes described in Section 3.1—offered fine-grained temporal information, including exact time stamps and durations, allowing the progression of the lesson to be tracked with high precision.

On the other hand, the OBS are highly detailed, providing precise time and duration indicators to track course progression. These indicators use the same model as C₁ and C₂ with the SMs. Consequently, we were able to cross-reference the data between C₁ and OBS, C₂ and OBS, as well as the multimodal decision fusion of C₁ and C₂ with OBS. The results of these comparisons are presented in Table 1 below.

To answer the RQ₁, for Oksana (Student 1), the agreement between the classroom observations (OBS) and the HR signal-based method (C₁) is significant at 70%, indicating that the HR method is highly effective in objectively detecting when the student is active in class. This convergence shows that the HR method provides a reliable and objective measure of student activity compared to observational methods. In contrast, the convergence between OBS and the EFE-based method (C₂) is lower at 55%, indicating that while the EFE method can reflect classroom activity to some extent, it lacks the precision of the HR-based method. The fusion of these two modalities enhances the accordance to 80%, an average improvement of 17.5% compared to unimodal methods, highlighting the need for multimodal benefits in better detecting SMs at the local time level for this student.

Mitch (Student 2) exhibits a strong agreement of 72.22% between OBS and C₁, reinforcing that the HR method provides a more objective and reliable measure of classroom activity compared to the subjective or observation-based methods. The convergence between C₂ and classroom observations is also relatively strong at 62.5%, but it is still lower than the HR-based measure, highlighting the HR method's superiority in offering a more accurate, objective assessment. The fusion of these two modalities achieves a convergences of 90%, representing an average increase of 22.64% over unimodal methods. This underscores the importance of multimodal approaches in more effectively detecting SMs at the local time level for this student.

For Zeynep (student 3), the HR signal-based method (C₁) shows a moderate agreement with classroom observations at 54.54%, suggesting that while the HR method objectively captures Zeynep's classroom activity, the complexity of her physiological responses may not always align perfectly with direct observations. Nonetheless, the HR method remains more objective than the EFE-based method (C₂), which shows a much lower convergence of 16.67% with classroom observations. While the unimodal performance has decreased for this student, the fusion of these two modalities still enhances convergences by 63.62%, representing an average increase of 28.02% over unimodal methods. This underscores the importance of multimodal approaches in more effectively leveraging the complementary information from the EFE and HR modalities, as well as in detecting SMs at the local time level for this student.

The data in Table 1 highlight the ability of our HR signal-based method to effectively detect 3 students' activity and, at times, emotional states compared to the EFE. However, the fusion of these modalities has led to better detection of SMs at the local time level within the session. The substantial agreement with students' subjective experiences and the precise, direct classroom observations validates the robustness of our approach. This cross-verification reinforces the credibility of our HR signal-based method, demonstrating its applicability and potential to provide objective and insightful data on student activity and emotional states in educational contexts.

To answer the first part of RQ₁ and RQ₂, we will now focus on the three participants during one particular session. As we explained previously, within component 3, the data were not always convergent. As shown by the Figure 10 of the average scores of the three emotions for the three students, enjoyment is highest. However, the open-ended questions reveal that the dominant emotion for the 14 sessions where Mitch was present is boredom. This observation refers to the most frequent emotion across his individual sessions, whereas the average scores in Figure 10 reflect the mean intensity of each emotion over all sessions; a few sessions with high enjoyment scores can therefore raise the overall average above boredom despite its greater frequency. For Oksana (present at 13 sessions) and Zeynep (present at 14 sessions), enjoyment dominates in the responses to the questions, even though they sometimes experience boredom and anxiety in the French FL class.

We will now examine in more detail one session for each student where the results between the three components and the classroom observations were convergent or semi-convergent. We selected session 14 for Mitch and Oksana where the emotion of enjoyment was predominant for both, and they had moments of shared enjoyment. Session 15 was selected for Zeynep who also reported very high enjoyment.