Multimodal emotion recognition based on manifold learning and convolution neural network

Multimodal emotion recognition task based on physiological signals is becoming a research hotspot. Traditional methods need to design and extract a series of features from single-channel or multi-channel physiological signals on the basis of extensive domain knowledge. These methods cannot make full use of the relevant information among channels, and the emotion recognition of a single modality cannot fully express the emotional state. This paper proposes a multimodal emotion recognition model based on manifold learning and a convolutional neural network (CNN). The electroencephalograph (EEG) signals are combined with peripheral physiological signals and eye movement signals respectively, the multivariate synchrosqueezing transform (MSST) is used to simulate the joint oscillation structure of multi-channel signals, and then the related feature parameters are extracted and fused into feature vectors. The proposed method finds the corresponding low dimensional embedding features for given high-dimensional features by an improved manifold learning method, which feeds into the deep convolutional neural network (DCNN) model for emotion recognition. We perform extensive four-category experiments on the dimensions of arousal and valence. Results indicate that our proposed model achieves average accuracies of 90.05% and 88.17% on the DEAP and MAHNOB-HCI datasets respectively, which both receive better performances than most of the compared studies, verifying the effectiveness of the model.