Deep Learning for Multimodal Emotion Recognition-Attentive Residual Disconnected RNN

Human communicates using verbal and non-verbal cues. One of the most essential elements that complements the understanding of communication is emotion. Emotion is expressed not only in words, but also facial expressions, body language, tone, etc. Therefore, we formulate the emotion recognition as a multimodal task. Emotions are usually described in a sequence along with the utterances. In recent years, RNN-based models have been known to be good at modeling the entire sequence and capturing long-term dependencies. However, it lacks the ability to extract local key patterns and position-invariant features. Hence, we adopt Deep Attentive Residual Disconnected RNN model which incorporates the concept from both RNN and CNN to enhance the ability to capture spatial and temporal features. We utilize CMU MOSEI dataset comprising of language, visual, and acoustic modalities for training and evaluating our model. The results show that Deep Attentive Residual Disconnected RNN model outperforms the baseline. Besides, the use of multimodal approach also solidifies the recognition better compared to those of single modalities.