Multi-granularity hierarchical contrastive learning between foreground and background for semi-supervised video action detection

Semi-supervised video action detection has received increasing attention due to its lower data annotation cost and performance comparable to fully supervised methods. However, due to the presence of dynamic background regions in the video, existing methods may encounter biases when interpreting the foreground and background of the video. This bias causes the model to mistakenly identify dynamic background areas as action foregrounds or to overlook background information, leading to misjudgment of the foreground. In response to this issue, this paper proposes a multi-granularity hierarchical contrastive learning between foreground and background for semi-supervised video action detection method termed as Multi-FB. Specifically, this paper proposes a multi- granularity encoding network based on foreground and background. This network uses a unified encoder to represent and learn foreground and background regions in videos at different granularities, thereby improving the model's understanding of action foreground and related background. Secondly, this paper proposes an Intramodel multi-granularity hierarchical contrastive strategy, which aims to minimize the representation discrepancies of foreground-to-foreground and background-to-background at different granularities within the same video, while maximizing the representation differences between the foreground and background at various granularities within the video. Furthermore, this paper proposes a Cross-model multi-granularity hierarchical contrastive strategy, which aims to enhance the consistency of the model's representations of foregrounds and backgrounds between the original data and the augmented data. A large number of experimental results on JHMDB-21 and UCF101-24 show that the proposed method can significantly distinguish feature representations between different categories, achieving performance comparable to state-of-the-art methods under semi- supervised conditions.