Multi-layer cross-modality attention fusion network for multimodal sentiment analysis

Sentiment analysis aims to detect the sentiment polarity towards the massive opinions and reviews emerging on the internet. With the increasing of multimodal information on social media, such as text, image, audio and video, multimodal sentiment analysis has attracted more attention in recent years and our work focuses on the text and image data. The previous works usually ignore the semantic alignment between the text and image, and cannot capture the interaction between them, which will affect the correct judgement for the sentiment polarity prediction. To resolve these problems, we propose a novel multimodal sentiment analysis model LXMERT-MMSA based on cross-modality attention mechanism. The single-modality feature is encoded by multi-layer Transformer encoder to achieve the deep semantic information implied in the text and image. Moreover, the cross-modality attention mechanism enables the model to fuse the text and image features effectively and achieve the rich semantic information by the alignment. It improves the ability of the model to capture the semantic relation between text and image. The evaluation metrics of accuracy and F1 score are used, and the experimental results on MVSA-multiple dataset and Twitter dataset show that our proposed model outperforms the previous SOTA model, and the ablation experimental results further prove that the model can make well use of multimodal features. © The Author(s), under exclusive licence to Springer Science+Business Media, LLC, part of Springer Nature 2024.