Building Cross-Subject EEG-Based Affective Models Using Heterogeneous Transfer Learning

Developing robust cross-subject EEG-based affective models is one of the key problems in affective brain-computer interfaces due to the challenges of individual differences and non-stationarity of EEG for building satisfactory affective models. In recent studies, transfer learning has been successfully applied to reducing the differences of feature distributions between source and target subjects. However, it is still necessary for users to acquire a moderately large amount of unlabeled EEG data from target subjects. Unlike the existing transfer learning frameworks solely on EEG, we propose an alternative approach to applying eye tracking data for calibration. In this paper, we propose a novel approach to leveraging heterogeneous knowledge from spatiotemporal scanpath patterns to enhance the performance of cross-subject EEG-based affective models. The main ideas behind our approach are that what and where subjects are watching would elicit their specific neural activities in the brains and such useful information would provide important clues to emotion recognition. We introduce heterogeneous transfer learning to use the modified transductive parameter transfer (TPT) framework. The TPT approach consists of three main steps. First, multiple individual classifiers are learned on each training dataset of source subjects. Second, a regression function is trained to learn the relation between the data distributions and classifiers parameter vectors. Finally, target classifier is obtained using the target feature distribution and the distribution-to-classifier mapping. To quantify the domain discrepancy across subjects, the scanpath sequences under the same film clips from different subjects are encoded and compared as a measurement of domain discrepancy for subject transfer by using dynamic time warping. The scanpath patterns are used as guidance for what knowledge should be transferred and how to transfer relevant affective information. We calculate scanpath-based kernels and EEG-based kernels and construct the cross-subject affective models with the transductive parameter transfer algorithm. The proposed approach has two advantages in comparison with the existing approaches: 1) the easily accessed eye tracking data from target subjects is utilized for subject transfer; 2) with only eye tracking data for target subjects, the proposed heterogeneous knowledge transfer approach can utilize the discriminative properties of EEG from other subjects. For conventional framework, the calibration phase requires recording moderately a large amount of labeled EEG data. In our framework, the calibration phase for new subjects can record only eye tracking data instead and transfer discriminative information of EEG from potential source subjects. It is feasible in some scenarios where collecting eye tracking data is much easier, while adaptive models are still able to recycle the representational capacity of EEG recorded previously for emotion recognition taking the advantage of high performance of EEG. The performance of our proposed approach and the state-of-the-art subject transfer methods are evaluated on an EEG and eye tracking dataset of three affective states (positive, neutral, and negative). The experiential results demonstrate that the scanpath-based transfer models achieve comparative performance in comparison with the EEG-based models and the scanpath-based subject transfer models obtain the mean accuracy of 69.72% in comparison of the conventional generic classifiers with 50.46% in average. These experimental results demonstrate the effectiveness of our proposed approach. © 2020, Science Press. All right reserved.