Fusion of deep learning features with mixture of brain emotional learning for audio-visual emotion recognition

被引:27
作者
Farhoudi, Zeinab [1 ]
Setayeshi, Saeed [2 ]
机构
[1] Islamic Azad Univ, Dept Comp Engn, Sci & Res Branch, Tehran, Iran
[2] Amirkabir Univ Technol, Dept Energy Engn & Phys, Tehran, Iran
关键词
Audio-Visual emotion recognition; Brain emotional learning; Deep learning; Convolutional neural networks; Mixture of network; Multimodal fusion; MODEL;
D O I
10.1016/j.specom.2020.12.001
中图分类号
O42 [声学];
学科分类号
070206 ; 082403 ;
摘要
Multimodal emotion recognition is a challenging task due to different modalities emotions expressed during a specific time in video clips. Considering the existed spatial-temporal correlation in the video, we propose an audio-visual fusion model of deep learning features with a Mixture of Brain Emotional Learning (MoBEL) model inspired by the brain limbic system. The proposed model is composed of two stages. First, deep learning methods, especially Convolutional Neural Network (CNN) and Recurrent Neural Network (RNN), are applied to represent highly abstract features. Second, the fusion model, namely MoBEL, is designed to learn the previously joined audio-visual features simultaneously. For the visual modality representation, the 3D-CNN model has been used to learn the spatial-temporal features of visual expression. On the other hand, for the auditory modality, the Mel-spectrograms of speech signals have been fed into CNN-RNN for the spatial-temporal feature extraction. The high-level feature fusion approach with the MoBEL network is presented to make use of a correlation between the visual and auditory modalities for improving the performance of emotion recognition. The experimental results on the eNterface'05 database have been demonstrated that the performance of the proposed method is better than the hand-crafted features and the other state-of-the-art information fusion models in video emotion recognition.
引用
收藏
页码:92 / 103
页数:12
相关论文
共 56 条
[41]  
Shen L., 2020, AFFECTIVE E LEARNING, P15
[42]   The neural basis of multisensory integration in the midbrain: Its organization and maturation [J].
Stein, Barry E. ;
Stanford, Terrence R. ;
Rowland, Benjamin A. .
HEARING RESEARCH, 2009, 258 (1-2) :4-15
[43]  
Szwoch Mariusz., 2015, Image Processing Communications Challenges, V6, P227
[44]   Smartphone Ownership and Interest in Mobile Applications to Monitor Symptoms of Mental Health Conditions [J].
Torous, John ;
Friedman, Rohn ;
Keshavan, Matcheri .
JMIR MHEALTH AND UHEALTH, 2014, 2 (01)
[45]  
Trigeorgis G, 2016, INT CONF ACOUST SPEE, P5200, DOI 10.1109/ICASSP.2016.7472669
[46]  
Vielzeuf V, 2017, PROCEEDINGS OF THE 19TH ACM INTERNATIONAL CONFERENCE ON MULTIMODAL INTERACTION, ICMI 2017, P569, DOI 10.1145/3136755.3143011
[47]   Recognizing human emotional state from audiovisual signals (vol 10, pg 659, 2008) [J].
Wang, Yongjin ;
Guan, Ling .
IEEE TRANSACTIONS ON MULTIMEDIA, 2008, 10 (05) :936-946
[48]   Kernel Cross-Modal Factor Analysis for Information Fusion With Application to Bimodal Emotion Recognition [J].
Wang, Yongjin ;
Guan, Ling ;
Venetsanopoulos, Anastasios N. .
IEEE TRANSACTIONS ON MULTIMEDIA, 2012, 14 (03) :597-607
[49]   Audio-visual affective expression recognition through multistream fused HMM [J].
Zeng, Zhihong ;
Tu, Jilin ;
Pianfetti, Brian M., Jr. ;
Huang, Thomas S. .
IEEE TRANSACTIONS ON MULTIMEDIA, 2008, 10 (04) :570-577
[50]   BAUM-1: A Spontaneous Audio-Visual Face Database of Affective and Mental States [J].
Zhalehpour, Sara ;
Onder, Onur ;
Akhtar, Zahid ;
Erdem, Cigdem Eroglu .
IEEE TRANSACTIONS ON AFFECTIVE COMPUTING, 2017, 8 (03) :300-313