A new multi-feature fusion based convolutional neural network for facial expression recognition

Using lightweight networks for facial expression recognition (FER) is becoming an important research topic in recent years. The key to the success of FER with lightweight networks is to explore the potentials of expression features in distinct abstract levels and regions, and design robust features to characterize the facial appearance. This paper proposes a lightweight network called Multi-feature Fusion Based Convolutional Neural Network (MFF-CNN), for image-based FER. The proposed model uses the Image Branch to extract both mid-level and high-level global features from the whole input image and utilizes the Patch Branch to extract local features from sixteen image patches of the original image. In MFF-CNN, feature selection based on L2 norm is performed to obtain more discriminative local features. Joint tuning is employed to integrate the two branches and fuse features. Experiment results on three widely used datasets, CK+, JAFFE and Oulu-CASIA show the proposed MFF-CNN outperforms the state-of-the-art methods in terms of average recognition accuracy. Compared to other competitive models with similar or larger number of parameters, our MFF-CNN improves the average recognition accuracy by 9.80% to 15.05%.