Depth-Based vs. Color-Based Pose Estimation in Human Action Recognition

Recent advances in deep learning resulted in the emergence of accurate models for human pose estimation in color videos. Distance between automatically estimated and manually annotated joint positions is commonly used for the evaluation of such methods. However, from a practical point of view, pose estimation is not a goal by itself. Therefore, in this work, we study how useful are state-of-the-art deep learning pose estimation approaches in a practical scenario of human action recognition. We compare different variants of pose estimation models with the baseline provided by the Kinect skeleton tracking, which, until recently, was the most widely used solution in such applications. We present a comprehensive framework for pose-based action recognition evaluation, which consists of both classical machine learning approaches, including feature extraction, selection, and classification steps, as well as more recent end-to-end methods. Extensive evaluation on four publicly available datasets shows, that by using state-of-the-art neural network models for pose tracking, color-based action recognition matches, or even outperforms, that of the depth-based one.