Using Facial Micro-Expressions in Combination With EEG and Physiological Signals for Emotion Recognition - Saffaryazdi, Nastaran; Wasim, Syed Talal; Dileep, Kuldeep; Nia, Alireza Farrokhi; Nanayakkara, Suranga; Broadbent, Elizabeth; Billinghurst, Mark (Frontiers in Psychology, 2022)
Emotions are multimodal processes that play a crucial role in our everyday lives. A wide range of applications requires the ability to recognize emotions, including healthcare, education, human-computer interaction, virtual reality, intelligent agents, and entertainment. The most common modality for communicating emotional states is facial expression. Facial expressions, however, can be controlled and thus may not create reliable recognition. In contrast, facial micro-expressions are involuntary and subtle facial movements in response to external stimuli. Because facial micro-expressions cannot be controlled, we propose combining them with brain and physiological signals in order to detect emotions reliably. Using facial microexpressions, EEG, and physiological signals, we describe how to detect arousal and valence levels. A subject-independent strategy is used to evaluate our strategy on the DEAP dataset as well as our dataset. As a conclusion, we show our results compared with state-of-the-art and discuss the limitations and future efforts of using facial micro-expressions and physiological signals for emotion recognition.