Dominant and complementary emotion recognition using hybrid recurrent neural network

Human emotion recognition has been a complex research problem due to the diversity in faces of individuals, especially in scenarios of compound emotions. Compound emotions are a combination or inclusion of two basic emotions; dominant and complementary. The recognition of compound emotions has been reported in several studies to be challenging, especially in comparison with the recognition of basic emotions. The paper proposes the use of a hybrid recurrent neural network to perform compound expression recognition. The hybrid recurrent neural network; CNN-LSTM is proposed, to train a network of extracted Facial Action Units representing the compound emotions. The iCV-MEFED is employed in the analysis. The iCV-MEFED dataset is made up of 50 classes of compound emotions which were captured in a controlled environment with the help of professional psychologists. The main contribution of the paper is the use of CNN and LSTM network fusion, with CNN acting as a feature extractor and reducer of feature dimensions, and LSTM for training using its memory blocks. This method has shown a significant improvement in the performance compared to previous studies carried out on the same dataset. The overall performances that are reported for all 50 classes of compound emotions are 24.7% of recognition accuracy, 25.3% of precision, 24.6% of recall and 24% of F1-score. The proposed method achieves comparable and encouraging results and forms a basis for future improvements.