Research on Multimodal Emotion Recognition Based on Fusion of Electroencephalogram and Electrooculography

Emotion recognition plays a vital role in building a harmonious society and emotional interaction. Recent research has demonstrated that multimodal interchannel correlations and insufficient emotion elicitation plague deep learning-based emotion identification techniques. To cope with these problems, we propose a multimodal and channel attention fusion transformer (MCAF-Transformer). First, we employ an olfactory video approach to evoke emotional expression more fully and acquire electroencephalogram (EEG) and electrooculography (EOG) signal data. Second, the model makes full use of multimodal channel information, time-domain and spatial-domain information of EEG and EOG signals, captures the correlation of different channels using channel attention, and improves the accuracy of emotion recognition by focusing on the global dependence on the temporal order using the transformer. We conducted extensive experiments on the olfactory video sentiment dataset, and the experimental results were correct at 94.63%. The results show that olfactory videos evoke emotion more adequately than pure videos and that the MCAF-Transformer model significantly outperforms other emotion recognition methods.