Research on Human Upper Limb Action Recognition Method Based on Multimodal Heterogeneous Spatial Temporal Graph Network

Graph convolutional neural networks have been increasingly used in human action recognition because of their powerful ability to deal with spatial topological relations. Like human action, upper limb action contains spatial and temporal features. For temporal features, graph convolutional networks cannot extract well enough to realize the coupling of spatial-temporal features. This paper proposes a multimodal heterogeneous spatial-temporal network (MHST-GCN) model based on multimodal information. Firstly, the model introduces a temporal graph based on a hybrid sparsity strategy, which captures local and global temporal features in the sequence of human upper limb actions while ensuring computational efficiency. Then, a heterogeneous graph model is proposed for fusing the two modal information to enhance the robustness of the model. Finally, extensive experiments are conducted on two standard datasets, NTU-RGB+D, and a homemade upper limb action dataset. The experimental results demonstrate the effectiveness of the proposed method.