Beyond coordinate attention: spatial-temporal recalibration and channel scaling for skeleton-based action recognition

Skeleton-based action recognition is an attractive issue in computer vision. Recent lightweight attention mechanisms (e.g. coordinate attention) have proven to be highly effective in skeleton-based action recognition. However, since long-range dependencies are captured along spatial and temporal directions, respectively, coordination attention cannot capture accurate long-range dependencies in the entire spatio-temporal domain and inevitably leads to inaccurate spatio-temporal location. In this work, we propose an efficient and lightweight attention mechanism, called coordinate enhanced attention, which consists of spatial-temporal recalibration and channel scaling. Spatial-temporal recalibration aims to capture precise long-range dependencies directly in the entire spatial-temporal domain. And channel scaling is introduced to efficiently utilize the multi-channel weight information. Our coordinate enhanced attention is efficient and lightweight, which can be easily integrated into classical neural networks. On two large-size datasets for skeleton-based action recognition (i.e. NTU RGB+D 60 and NTU RGB+D 120), our coordinate enhanced attention obtains consistent improvements. Experiments on two popular object detection datasets (i.e. COCO and Pascal VOC) and semantic segmentation dataset (i.e. Cityscapes) indicate that the proposed coordinate enhanced attention outperforms other lightweight attention mechanisms, which further validates its transferable ability.