GCF2-Net: global-aware cross-modal feature fusion network for speech emotion recognition - Li, Feng; Luo, Jiusong; Wang, Lingling; Liu, Wei; Sang, Xiaoshuang (Frontiers in Neuroscience, 2023)
Emotion recognition plays an essential role in interpersonal communication. However, existing emotion recognition systems use only features of a single modality for emotion recognition, ignoring the interaction of multimodal information. In order to improve the accuracy of speech emotion recognition, we propose a Global-aware Cross-modal feature Fusion Network (GCF$^2$-Net) for recognizing speech emotion in this paper. Firstly, we introduce a residual cross-modal fusion attention module (ResCMFA) and a Global-Aware block to fuse information from multiple modalities and capture global information. Then, we utilize transfer learning to extract wav2vec 2.0 features and text features that are fused by the ResCMFA module. The multimodal features are fed into the Global-Aware block to capture the most important emotional information on a global information. Finally, the experiment results have shown that our proposed method has significant advantages than state-of-the-art methods on the IEMOCAP and MELD datasets. Our model improves WA by 1.65$%$ and UA by 1.10$%$ on the IEMOCAP dataset, and improves accuracy by 1.90$%$ and weighted average F1 by 1.10$%$ on the MELD dataset, respectively.