Recurrent Spatiotemporal Feature Learning for Action Recognition

Recurrent neural networks (RNNs) like Long Short-term Memory (LSTM) have shown excellent performance for a variety of sequence learning problems on language and speech processing. Previous works leveraging RNNs for action recognition mainly apply LSTM on the top of Convolutional Neural Networks (CNNs), feeding high level semantic feature to RNNs and neglecting to learn spatiotemporal features in video, which makes the models unable to capture complex action patterns and lead to inferior performance. In this work, instead of adopting RNNs as classifiers, we propose to learn spatiotemporal feature in a recurrent way by replacing intermediate convolutional layers in CNNs with recurrent layers to model spatiotemporal information. In order to learn discriminative spatiotemporal features, we extend the bottleneck structure of Residual Network (ResNet) to model the spatiotemporal information in action video. To fully utilize the pretrained CNN model, we also introduce approaches to transfer the weight of original convolutional layer to our proposed model. Our proposed architecture is end-to-end trainable, and of significant flexibility to be adapted in any CNN-based structure. Our model produces the state-of-the-art performance on two standard benchmark for action recognition over RNN-based approaches.