Mixed Entropy Down-Sampling based Ensemble Learning for Speech Emotion Recognition

The strength of emotion at different positions in a speech is strong or weak, and the weak parts with unclear emotions will bring noise to the model. We propose a boosting ensemble learning method based on mixed entropy down-sampling to effectively select emotionally salient segments to improve the classifier's performance. An independent Convolutional Neural Network (CNN) model is trained in each iteration of ensemble learning. These CNN models form an ensemble classifier, which improves the generalization ability by synthesizing all the learned results of down-sampling, making emotion recognition more accurate. We also introduce the concept of Mixed Information Entropy (MIE), which consists of Emotional Certainty Entropy (ECE) and Structural Distribution Entropy (SDE). ECE measures the emotional confusion of segments, while SDE measures the stability of segments in deep feature space. During the iteration, the deep features are obtained from the last fully connected layer of the model and down-sampled according to the weighted sum of confidence and MIE. The selected segments with stronger emotions are used for the next iteration. Our method is 3.77% higher on WA and 2.37% higher on UA than the naive CNN model on the IEMOCAP dataset.