Multi-stream adaptive 3D attention graph convolution network for skeleton-based action recognition

Action recognition methods based on spatial-temporal skeleton graphs have been applied extensively. The spatial and temporal graphs are generally modeled individually in previous approaches. Recently, many researchers capture the correlation information of temporal and spatial dimensions in spatial-temporal graphs. However, the existing methods have several issues such as 1. The existing modal graphs are defined based on the human body structure which is not flexible enough; 2. The approach to extracting non-local neighborhood features is insufficiently powerful; 3. Attention modules are limited to a single scale; 4. The fusion of multiple data streams is not sufficiently effective. This work proposes a novel multi-stream adaptive 3D attention graph convolution network for skeleton-based action recognition that improves the aforementioned issues. The method utilizes an adaptive topology graph with an adaptive connection coefficient to adaptively optimize the topology of the graph during the training process according to the input data. An optimal high-order adjacency matrix is constructed in our work to balance the weight bias, which captures non-local neighborhood features precisely. Moreover, we design a multi-scale attention mechanism to aggregate information from multiple ranges, which makes the graph convolution focus on more efficient nodes, frames, and channels. To further improve the performance of the model, a novel multi-stream framework is proposed to aggregate the high-order information of the skeleton. The experiment results on the NTU-RGBD and Kinetics-Skeleton prove that our proposed method reveals better results than existing methods.