A Deep Learning Framework with Cross Pooled Soft Attention for Facial Expression Recognition

Facial Expression Recognition (FER) is at the heart of Human–Computer Interaction (HCI) and has received a lot of attention in the field of computer vision. We present a novel attention-based deep neural network for recognizing facial expressions from images. Initially, regions such as eye-pair, mouth and face are cropped, independently passed through the pre-trained Xception network to obtain deep representations. All of these descriptors may not have same influence while recognizing the type of expression, and some of them may require special attention over others depending on the type of expression. We incorporate attention mechanism into the model to automatically learn the amount of attention to be paid to each descriptor. These attention-based features obtained from all the three regions are then fused using the proposed Cross Average Pooling (CAP) layers to produce a compact and discriminatory representation that ultimately leads to better identification of facial expressions. The proposed cross average pooled soft attention results in compact and discriminatory representations for facial images, allowing for more accurate predictions. The proposed approach is evaluated on two benchmark datasets (JAFFE and CK+), and the experimental results reveal that the proposed model outperforms existing models with an accuracy of 97.67 and 97.46% on JAFFE and CK+ datasets, respectively. © 2022, The Institution of Engineers (India).