Soft Voting Strategy for Multi-Modal Emotion Recognition Using Deep-learning- Facial Images and EEG

Emotion recognition is an important factor in social communication and has a wide range of applications from retail to healthcare. In psychology, emotion recognition focuses on emotional states within non-verbal visual and auditory cues. It is essential to the human ability to associate meaning with events rather than treating them as mere facts. Studies of emotion recognition often utilize data gathered in response to non-verbal cues using modalities such as eye tracking, Electroencephalogram (EEG), and facial video and build classification models capable of differentiating responses to various emotions and cues. The accuracy of these emotion recognition models largely depends on feature representation and the suitability of the chosen features in magnifying the differences between patterns of various emotions. Single-modal feature extraction methods are limited in capturing between-group differences and often result in reduced classification performance. To address this problem, this paper proposes a multi-modal approach in the representation of response to emotional cues involving EEG recording and facial video data. The study utilizes the dataset containing frontal face video recordings and EEG data of 22 participants. A novel deep neural network architecture within the feature level is used to efficiently predict emotions using EEG and facial video data. The experimental result indicates 97.5% accuracy in identifying facial expressions and categorizing them into two classes, arousal (class 0) and valence (class 1), surpassing state-of-the-art for the DEAP dataset.