TFE: A Transformer Architecture for Occlusion Aware Facial Expression Recognition - Gao, Jixun; Zhao, Yuanyuan (Frontiers in Neurorobotics, 2021)
Facial expression recognition (FER) in the wild is challenging due to various un-constrained conditions. Although existing deep learning-based FER approaches have been quite promising in recognizing frontal faces, they still struggle to accurately identify the facial expressions on the faces that are partly occluded in unconstrained scenarios. To mitigate this issue, we propose a transformer-based FER method (TFE) that is capable of adaptatively focusing on the most important and un-occluded facial regions. TFE is based on the multi-head self-attention mechanism that can flexibly attend to a sequence of image patches to encode the critical cues for FER. Compared with traditional transformer, the novelty of TFE are two-fold: i) To effectively select the discriminative facial regions, we integrate all the attention weights in various transformer layers into an attention map to guide the network to perceive the important facial regions. ii) Given an input occluded facial image, we use a decoder to reconstruct the corresponding non-occluded face. Thus TFE is capable of inferring the occluded regions to better recognize the facial expressions. The proposed TFE has been evaluated on the two largest in-the-wild facial expression datasets (RAF-DB and AffectNet) and their modifications with synthesized facial occlusions. Experimental results show that TFE improves the recognition accuracy on both the non-occluded faces and occluded faces. Compared with other state-of-the-art FE methods, TFE obtains consistent improvements. Visualization results show TFE is capable of automatically focusing on the discriminative and non-occluded facial regions for robust FER.