SiSe: Simultaneous and Sequential Transformers for multi-label activity recognition

Multi-label activity recognition is extremely challenging, where multiple activities may appear simultaneously or sequentially in a video. While previous works have realized the temporal co-occurrence of activities, the sequential order of activities have been largely overlooked. However, we argue that the sequential order of activities should also be preserved in correlation modeling, because shuffling the order might not form a semantically meaningful video. In this work, we present plug-and-play Simultaneous and Sequential Transformer (SiSe) modules for multi-label activity recognition. Upon frame features of all time steps, SiSe enhances spatiotemporal feature learning for multi-label activity recognition, by capturing the simultaneous and sequential activity correlations. Specifically, we employ a Simultaneous Transformer module to connect multiple activities that probably appear at each frame, and a hierarchical Sequential Transformer module to efficiently capture the sequential activity correlations in an order-preserved manner. Despite the straightforward and class- agnostic design of SiSe, it can outperform state-of-the-art approaches on three multi-label activity recognition benchmarks. In particular, we verify the significance of preserving the sequential order of activities with our Sequential Transformer in correlation modeling. We also conduct ablation studies and visual analysis for better understanding of our SiSe.