AUTHOR=Guedat-Bittighoffer Delphine , Moufidi Abderrazzaq , Dewaele Jean-Marc , Rousseau David , Voyneau Hugo , Rasti Pejman 

TITLE=Heart rates, facial expressions and self-reports: a multimodal longitudinal approach of learners' emotions in the foreign language classroom

JOURNAL=Frontiers in Artificial Intelligence

VOLUME=Volume 8 - 2025

YEAR=2025

URL=https://www.frontiersin.org/journals/artificial-intelligence/articles/10.3389/frai.2025.1604110

DOI=10.3389/frai.2025.1604110

ISSN=2624-8212

ABSTRACT=Emotions in educational settings are often studied through self-reports or lab experiments, limiting insights into their real-world dynamics. This study examines learner emotions in authentic foreign language classrooms using a multimodal longitudinal approach. Over 16 consecutive sessions, we collected heart rate (HR) signals, emotional facial expressions (EFE), classroom observations, and self-reports on enjoyment, anxiety, and boredom to capture both physiological and self-perceived emotional responses. Rather than aggregating data across students, we focused on individualized emotional patterns to understand variations in emotional experiences. Each dataset included extensive video recordings, continuous HR monitoring, detailed observational notes, and post-session questionnaires, providing a high-resolution picture of emotional dynamics. Using unsupervised clustering techniques, we identified key emotional episodes—peaks and drops in physiological arousal (heart rate variation) and facial expression—relative to individual emotional baselines. These moments were cross-referenced with classroom observations and self-reports for validation. Findings highlight moments of positive emotional contagion during peer interactions, emphasizing the social dimension of language learning. This multimodal approach captures the interplay of physiological, behavioral, and subjective responses, offering a scalable method for studying classroom emotions. Methodologically, it demonstrates how multimodal analytics can uncover transient emotional states in real-world settings, while practically informing adaptive teaching strategies, such as leveraging peer interactions to enhance engagement or reduce anxiety. By integrating physiological, behavioral, and subjective data, this study provides a comprehensive framework for understanding the affective dimensions of learning.