Multimodal Fusion of Spatial-Temporal Features for Emotion Recognition in the Wild

Making the machine understand human emotion is a challenge to realize artificial intelligence. Considering the temporal correlation widely exists in the video, we present a multimodal fusion of spatial-temporal features system to recognize emotion. For the visual modality, the spatial-temporal features are extracted to represent the dynamic emotional variance along with the facial action in the video. The audio modality is utilized to assist the visual modality. A decision-level fusion approach is presented to make full use of the complementarity between visual modality and audio modality to boost the performance of the emotion recognition system. The experiments on a challenging dataset AFEW4.0 show that the proposed system achieves better generalization performance compared with other state-of-the-art methods.