Convolutional Neural Networks with Generalized Attentional Pooling for Action Recognition

Inspired by the recent advance in attentional pooling techniques in image classification and action recognition tasks, we propose the Generalized Attentional Pooling (GAP) based Convolutional Neural Network (CNN) algorithm for action recognition in still images. The proposed GAP-CNN can be formulated as a new approximation of the second-order/bilinear pooling techniques widely used in fine-grained image classification. Unlike the existing rank-1 approximation, a generalized factoring (with nonlinear functions) is introduced to exploit the intrinsic structural information of the sample covariance matrices of convolutional layer outputs. Without requiring preprocessing steps such as object (e.g., human body) bounding boxes detection, the proposed GAP-CNN automatically focuses on the most informative part in still images. With the additional guidance of keypoints of human pose, the proposed GAP-CNN algorithm achieves the state-of-the-art action recognition accuracy on the large-scale MPII still image dataset.