Action unit analysis enhanced facial expression recognition by deep neural network evolution

Facial expression is one of the most powerful and natural signals for human beings conveying their emotional states and intentions. Recently, facial expression recognition from facial cues based on FACS is investigated, where facial expressions can be described by a subset of AUs, and the facial expression categories could be easily extended. The goal of our work is the proposal of an action unit analysis enhanced facial expression recognition system based on evolutional deep learning approach. The main contributions of our work include the following three aspects: (1) the temporal dynamic based 3DLeNets is exploited for video analysis based AUs detection. And a general evolutionary framework is conducted for the deep neural networks optimization. (2) The correlations among AUs and the correlation between AUs and emotions are investigated, and the relationship probability model between AUs and emotions is derived by the concept of discriminative power coefficients. (3) An adaptive subsequence matching algorithm (ASMA) is adopted to measure the similarity between AUs sequences, so that to construct the inference scheme of mapping AUs to emotions. Experimental results proved that the AUs enhanced facial expression recognition system performs well comparing to existing facial expression analysis methods, and each AU has different contribution roles for different facial expressions. It is also found to be more practical than discrete facial expression recognition as most of the facial expressions can be described using a subset of AUs. (c) 2020 Elsevier B.V. All rights reserved.