CoDF-Net: coordinated-representation decision fusion network for emotion recognition with EEG and eye movement signals

Physiological signals, such as EEG and eye movements, have emerged as promising research topics in emotion recognition due to their inherent advantages of objectivity, high recognition accuracy, and cost-effectiveness. However, most existing methods for fusing EEG and eye movement signals use concatenation or weighted summation, which may lead to information loss and limited ability to resist noise. To tackle this issue, in this paper, we propose a Coordinated-representation Decision Fusion Network (CoDF-Net) to efficiently fuse the representation of EEG and eye movement signals. Specifically, CoDF-Net first learns personalized information by maximizing the correlation between modalities. Next, the Decision-level Fusion Broad Learning System (DF-BLS) is developed to construct multiple sub-systems to obtain the final emotional states via the effective decision-making mechanism. To evaluate the performance of the proposed method, subject-dependent and subject-independent experiments are designed on two public datasets. Extensive experiments demonstrate that the proposed method has superior emotion recognition performance over traditional approaches and current state-of-the-art methods. The CoDF-Net achieves 94.09 and 91.62% in the subject-dependent setting and 87.04 and 83.87% in the subject-independent setting on the SEED-CHN and SEED-GER datasets, respectively. Moreover, it is found that the proposed method exhibits a more significant ability to resist noise by adding Gaussian noise with different standard deviations.