Boosting Facial Emotion Recognition by Using GANs to Augment Small Facial Expression Dataset

Facial expression recognition (FER) is an important research field for human-machine interfaces, and FER by deep learning from small training datasets is still challenging. This paper proposes a new generative adversarial network (GAN) model to transfer neutral face images to images with diverse facial expressions to deal with the problem of expressional data scarcity in deep learning for FER based on a small set of training samples. To mitigate distortions and overfitting that often happen in training GANs with small training datasets, a novel GAN architecture is proposed, which consists of a generator with two encoders and two decoders, two discriminators, and a feature extractor. Specifically, a feature map mechanism is proposed to discover regional feature differences between images in the source domain and target domain, which makes the proposed GAN architecture be able to not only generate desirable facial expression images but also maintain the original characters in input neutral face images. Experimental results show that, by using the proposed GAN to augment a training dataset of images with up to 7 facial expressions, the FER accuracy of several deep neural networks tested in the experiments can be significantly improved by over 10%.