Multi-Model Fusion Framework Using Deep Learning for Visual-Textual Sentiment Classification

被引:0
|
作者
Salman Al-Tameemi I.K. [1 ,2 ,3 ]
Feizi-Derakhshi M.-R. [1 ,2 ]
Pashazadeh S. [2 ]
Asadpour M. [2 ]
机构
[1] Computerized Intelligence Systems Laboratory, Department of Computer Engineering, Faculty of Electrical and Computer Engineering, University of Tabriz, Tabriz
[2] Department of Computer Engineering, Faculty of Electrical and Computer Engineering, University of Tabriz, Tabriz
[3] State Company for Engineering Rehabilitation and Testing, Iraqi Ministry of Industry and Minerals, Baghdad
来源
Computers, Materials and Continua | 2023年 / 76卷 / 02期
关键词
decision fusion; deep learning; interpretability; joint fusion; multimodal classification; Sentiment analysis;
D O I
10.32604/CMC.2023.040997
中图分类号
学科分类号
摘要
Multimodal Sentiment Analysis (SA) is gaining popularity due to its broad application potential. The existing studies have focused on the SA of single modalities, such as texts or photos, posing challenges in effectively handling social media data with multiple modalities. Moreover, most multimodal research has concentrated on merely combining the two modalities rather than exploring their complex correlations, leading to unsatisfactory sentiment classification results. Motivated by this, we propose a new visual-textual sentiment classification model named Multi-Model Fusion (MMF), which uses a mixed fusion framework for SA to effectively capture the essential information and the intrinsic relationship between the visual and textual content. The proposed model comprises three deep neural networks. Two different neural networks are proposed to extract the most emotionally relevant aspects of image and text data. Thus, more discriminative features are gathered for accurate sentiment classification. Then, a multichannel joint fusion model with a self-attention technique is proposed to exploit the intrinsic correlation between visual and textual characteristics and obtain emotionally rich information for joint sentiment classification. Finally, the results of the three classifiers are integrated using a decision fusion scheme to improve the robustness and generalizability of the proposed model. An interpretable visual-textual sentiment classification model is further developed using the Local Interpretable Model-agnostic Explanation model (LIME) to ensure the model’s explainability and resilience. The proposed MMF model has been tested on four real-world sentiment datasets, achieving (99.78%) accuracy on Binary_Getty (BG), (99.12%) on Binary_iStock (BIS), (95.70%) on Twitter, and (79.06%) on the Multi-View Sentiment Analysis (MVSA) dataset. These results demonstrate the superior performance of our MMF model compared to single-model approaches and current state-of-the-art techniques based on model evaluation criteria. © 2023 Tech Science Press. All rights reserved.
引用
收藏
页码:2145 / 2177
页数:32
相关论文
共 50 条
  • [41] An Ensemble of Deep Learning-Based Multi-Model for ECG Heartbeats Arrhythmia Classification
    Essa, Ehab
    Xie, Xianghua
    IEEE ACCESS, 2021, 9 : 103452 - 103464
  • [42] A Novel Hybrid Deep Learning Model for Sentiment Classification
    Salur, Mehmet Umut
    Aydin, Ilhan
    IEEE ACCESS, 2020, 8 (58080-58093) : 58080 - 58093
  • [43] A MULTI-MODEL FUSION FRAMEWORK FOR NIR-TO-RGB TRANSLATION
    Yan, Longbin
    Wang, Xiuheng
    Zhao, Min
    Liu, Shumin
    Chen, Jie
    2020 IEEE INTERNATIONAL CONFERENCE ON VISUAL COMMUNICATIONS AND IMAGE PROCESSING (VCIP), 2020, : 459 - 462
  • [44] Visual and Textual Sentiment Analysis of Daily News Social Media Images by Deep Learning
    Felicetti, Andrea
    Martini, Massimo
    Paolanti, Marina
    Pierdicca, Roberto
    Frontoni, Emanuele
    Zingaretti, Primo
    IMAGE ANALYSIS AND PROCESSING - ICIAP 2019, PT I, 2019, 11751 : 477 - 487
  • [45] Visual and Textual Sentiment Analysis of a Microblog Using Deep Convolutional Neural Networks
    Yu, Yuhai
    Lin, Hongfei
    Meng, Jiana
    Zhao, Zhehuan
    ALGORITHMS, 2016, 9 (02)
  • [46] Multi-model Fusion Attention Network for News Text Classification
    Li Z.
    Wu J.
    Miao J.
    Yu X.
    Li S.
    International Journal for Engineering Modelling, 2022, 35 (02) : 1 - 15
  • [47] Feature Enhancement Based Text Sentiment Classification using Deep Learning Model
    Janardhana, D. R.
    Vijay, C. P.
    Swamy, G. B. Janardhana
    Ganaraj, K.
    PROCEEDINGS OF THE 2020 5TH INTERNATIONAL CONFERENCE ON COMPUTING, COMMUNICATION AND SECURITY (ICCCS-2020), 2020,
  • [48] Deep learning-based sentiment classification of evaluative text based on Multi-feature fusion
    Abdi, Asad
    Shamsuddin, Siti Mariyam
    Hasan, Shafaatunnur
    Piran, Jalil
    INFORMATION PROCESSING & MANAGEMENT, 2019, 56 (04) : 1245 - 1259
  • [49] Multi-Model Fusion Fine-Grained Image Classification Method Based on Migration Learning
    Zhang, Wenying
    Wang, Yaping
    IEEE ACCESS, 2024, 12 : 31977 - 31987
  • [50] An Underwater Human-Robot Interaction Using a Visual-Textual Model for Autonomous Underwater Vehicles
    Zhang, Yongji
    Jiang, Yu
    Qi, Hong
    Zhao, Minghao
    Wang, Yuehang
    Wang, Kai
    Wei, Fenglin
    SENSORS, 2023, 23 (01)