A Multitask Learning Framework for Multimodal Sentiment Analysis

Mapping continuous dimensional emotion to discrete classes is an extremely difficult task. In this paper, we predict the intensity classes of emotions based on valence and arousal in segments of audio-visual recordings about car reviews. Consequently, for unimodal features, we first employ baseline methods and principal component analysis to search for the best unimodal features in different modalities, which can simplify the relationship between feature attributes. For multimodal features, we perform multimodal fusion on the best and other unimodal features through an early fusion strategy. For sentiment analysis, we propose six hybrid temporal models for modeling complex time dependencies. To avoid overfitting the validation set and providing complementary information between different modalities, we propose a multitask learning framework, which can adaptively change the weight of loss per subtask.