Adaptive Pooling of the Most Relevant Spatio-Temporal Features for Action Recognition

This paper presents a model-based action recognition system that utilizes the Kinect 3D skeleton to construct adaptive spatio-temporal motion representations. The proposed method utilizes two features, namely the joint relative distance (JRD) and joint relative angle (JRA) to encode the spatio-temporal motion patterns of different skeletal joints. To evaluate the relevance of a particular joint-pair in representing an action class, we introduce a flatness measure that quantifies the level of engagement of the corresponding joint-pair in performing the action. The flatness measures computed for all skeletal joint-pairs are accumulated to construct a joint-pair relevance (JPR) matrix, which facilitates adaptive pooling of the most relevant spatio-temporal features to construct the final motion description for individual action classes. In addition, we propose a score level fusion of JRD and JRA features with a weighted dynamic time warping (DTW)-based matching scheme to effectively boost the overall recognition performance. In our experiments, the proposed method achieves better recognition performance than well-known existing methods.