Multi-modal feature fusion based on multi-layers LSTM for video emotion recognition

Emotion is a key element in video data. However, it is difficult to understand the emotions conveyed in such videos due to the sparsity of video frames expressing emotion. Meanwhile, some approaches proposed to consider utterances as independent entities and ignore the inter-dependencies and relations among the utterances in recent years. These approaches also ignore the key point of multi-modal feature fusion in the feature learning process. In order to handle this problem, in this paper, we propose an LSTM-based model that can fully consider the relations among the utterances and also handle the multi-modal feature fusion problem in the learning process. Finally, the experiments on some popular datasets demonstrate the effectiveness of our approach.