Collaborative knowledge distillation for incomplete multi-view action prediction

Predicting future actions is a key in visual understanding, surveillance, and human behavior analysis. Current methods for video-based prediction are primarily using single-view data, while in the real world multiple cameras and produced videos are readily available, which may potentially benefit the action prediction tasks. However, it may bring up a new challenge: subjects in the videos are more likely to be occluded by objects when captured from different angles, or suffer from signal jittering in transmission. To that end, in this paper we propose a novel student network called Collaborative Knowledge Distillation (CKD) to predict human actions with missing information under a multi-view setting, i.e., incomplete multi-view action prediction. First, we create a graph attention based teacher model capable of fusing multi-view video features for prediction task. Second, we construct a corruption pattern bank (CPB) to simulate various missing segments in multi-view video, and each student model will manage one pattern through privileged information and knowledge distillation. Third, to account for arbitrary missing video segments in real-world, the ensemble of student models will be developed to make a joint prediction. The proposed framework has been extensively evaluated on popular multi-view visual action datasets, including PKU-MMD and NTU-RGB to validate the effectiveness of our approach and to the best of our knowledge action prediction has not yet been explored in the multi-view setting. (c) 2021 Elsevier B.V. All rights reserved.