Online Action Detection with Learning Future Representations by Contrastive Learning

Online Action Detection (OAD), which predicts the ongoing human action from a streaming video, is an important task in multimedia analysis. Compared with offline action detection, OAD only uses historical and current frames without any access to future information, resulting in limited performances. Current OAD methods obtain future information by training an additional future anticipation task and suffer from low efficiency. To address the low efficiency problem, we propose to integrate future information into visual representations for OAD. Specifically, we learn future representations through a contrastive-learning-based method. Further, two novel sampling strategies, multiple scale sampling and timeline overlap sampling, are introduced to enhance our method. The experimental results show that our method significantly improves inference efficiency and achieves promising effectiveness performances on two popular OAD benchmarks.