Adversarial attack evaluation and defense method for multimodal sentiment analysis model

To systematically explore the robustness of the multimodal sentiment analysis model under adversarial attacks，three classical adversarial attack methods (fast gradient sign method，projected gradient descent，and momentum iterative fast gradient sign method)，two modal data input (visual and acoustic)，and four diverse data feature fusion methods (feature sum，feature concat，multimodal low-rank bilinear，and multimodal tucker fusion) were adopted to comprehensively compare the performance of the models under various combinations，and the internal factors that affect the robustness of multimodal sentiment analysis models were studied．A mutual information maximization based multimodal defense method was proposed to improve the robustness of model by reducing redundant information in input modal features．The research results demonstrate that under the combination of nonlinear feature fusion and bimodal data input，the model defends best against adversarial attacks．After applying the defense method of maximizing mutual information，the performance of the model and the ability to resist attacks can be effectively improved. © 2023 Huazhong University of Science and Technology. All rights reserved.