Spatio-Temporal Weighted Posture Motion Features for Human Skeleton Action Recognition Research

In recent years, computer vision related applications (e. g. behavior surveillance, human-computer interaction, electronic games, and health care) have gained increasing popularity, and the key technology of these interactive applications is how to make the machine understand human movements, which is also known as human action recognition. Although experts have done a lot of research before, how to accurately identify human action from traditional RGB videos is still a challenging problem due to various interference factors, such as lighting changes, view changes, occlusion and background clutter. Latterly, the popularity of depth sensors and real-time skeleton estimation algorithm based on depth image have brought opportunities for human behavior recognition research. The depth map provides additional depth information, which can easily segment the desired targets from complex scenes, significantly improve the background clutter, and greatly simplify the behavior recognition model. And all of this boosts the development of skeleton-based action recognition. The skeleton estimation algorithm defines skeleton as a graphical model composed of human trunk, head and limbs position. It can quickly and accurately estimate the 3D position information of skeleton joints from depth images at the speed of 200 frames per second on Xbox 360 GPU. Existing skeleton based behavior recognition methods can be roughly divided into two categories: joint-based methods and body-based methods. The human skeleton is regarded as a set of joints and described by the location correlation feature of the joints in the joint-based methods. These features include joint position features, relative joint location features, joint orientation features in a fixed coordinate system and so on. On the other hand, part-based methods regard the human skeleton as a set of rigid segments and use joint angle features, bioinspired 3D features and geometric relationship features of different rigid body parts to represent the human skeleton. Most of these researches focus on extracting the spatial information of different body joints in single frame and temporal information of body joints between adjacent frames to represent action sequence, but these works don't take into consideration that the importance of different body joints and postures may vary in terms of deciding which type of action class the sample belongs to. Therefore, an action recognition method based on spatio-temporal weighted posture motion features is proposed in this paper. Since each 3D video sequence can be regarded as a set of ordered static gestures, and the static pose can be regarded as a set of joints. Based on this, the author first deals with the spatial relationships of all joints contained in each static pose to obtain the spatial domain characteristics of the video sequence, and then calculate location relationships of the same joint between adjacent frames to get the temporal features. And normalization scheme is introduced to obtain the final representation of the skeleton sequence. The author also adopts bilinear classifier to calculate the weights of the joints and static postures for the action class to determine the informative joints and postures. Meanwhile, dynamic time warping (DTW) algorithm and Fourier temporal pyramid (FTP) representation are introduced to process temporal modeling for better temporal analyses, and SVM is finally used for the action classification. The experimental results on three challenging datasets demonstrate that our approach achieves competitive, even the best performance compared with the state-of-the-art methods. © 2020, Science Press. All right reserved.