Spatial-Temporal Convolutional Attention Network for Action Recognition

In the task of video action recognition, whether in the spatial dimension or temporal dimension of video, how to fully learn and make use of the correlation between features has a great impact on the final recognition performance. Convolution obtains local features by calculating the correlation between feature points in the neighborhood, and self-attentional mechanism learns global information through the information interaction between all feature points. A single convolutional layer does not have the ability to learn feature correlation from the global perspective. Even repeated stacking of multiple layers only obtains several larger receptive fields. Although the self-attention layer has a global perspective, its focus is only the content relationship expressed by different feature points, ignoring the local location characteristics. In order to solve the above problems, a spatial-temporal convolutional attention network is proposed for action recognition. Spatial-temporal convolutional attention network is composed of spatial convolutional attention network and temporal convolutional attention network. Spatial convolutional attention network uses self-attention method to capture the apparent feature relationship of spatial dimension, and uses one-dimensional convolution to extract dynamic information. The temporal convolutional attention network obtains the correlation information between frame level features in the temporal dimension through the self-attention method, and uses 2D convolution to learn spatial features. Spatial-temporal convolutional attention network integrates the common test results of the two networks to improve the performance of model recognition. Experiments are carried out on HMDB51 data set. Taking ResNet50 as the baseline and introducing spatial-temporal convolutional attention module, the recognition accuracy of neural network is improved by 6.25 and 5.13 percentage points in spatial flow and temporal flow respectively. Compared with the current advanced methods, spatial-temporal convolutional attention network has obvious advantages in UCF101 and HMDB51 datasets. The spatial-temporal convolutional attention network proposed in this paper can effectively capture feature correlation information. This method combines the advantages of self-attention global connection and convolution local connection, and improves the spatial-temporal modeling ability of neural network. © 2023 Journal of Computer Engineering and Applications Beijing Co., Ltd.; Science Press. All rights reserved.