Attention-Based Video Disentangling and Matching Network for Zero-Shot Action Recognition

Zero-Shot Action Recognition (ZSAR) is achieved by learning the mapping relationship between visual space and semantic space. Existing methods usually utilize the SOTA backbone network to construct the visual space. These methods have two major limitations. First, the human motion information that crucial for action recognition is easy to be confused in the background. Second, the key information which can reflect the correlation between actions may fall into oblivion, due to the redundancy in video sequences. In this paper, we propose an Attention-based Video Disentangling Matching Network (AVDMN) to solve the above problems. Specifically, we decompose segment-wise video into background stream and human motion stream by proposing a video disentangling mechanism. Furthermore, to further highlight the correlation between actions, we design an attention module to extract the key component of the above information. Finally, a relationship learning module is introduced to learn and measure the distance or similarity between video representation and action labels. Experiments on three realistic action benchmark Olympic Sports, HMDB51, and UCF101 show that the proposed architecture achieves favorable performance among ZSAR methods.