GaitRA: triple-branch multimodal gait recognition with larger effective receptive fields and mixed attention

Gait Recognition, as a long-distance biometric technique for identity recognition, has attracted widespread attention in recent years. Previous academia typically employs minor convolutional networks to extract single-modal features from the silhouette or the joint skeleton. Nevertheless, silhouette-based methods are susceptible to clothing variations, while skeleton-based methods encounter the issue of missing physique information. Therefore, we propose a novel multimodal triple-branch network dubbed GaitRA to comprehensively acquire gait features from two aspects of silhouette and skeleton simultaneously. GaitRA consists of three branches: a 3D-CNN branch to extract the primary features of silhouette sequences, a 2D-CNN branch to obtain the secondary features of silhouette sequences, and a branch based on Spatio-Temporal Graph Convolution (ST-GCN) to gain joint skeleton features. More importantly, we innovatively introduce the RepLK-ACTION Module, which combines the RepLK Block based on the Swin Transformer and the ACTION Module based on a mixed attention mechanism from the Action Recognition. RepLK-ACTION Module establishes larger Effective Receptive Fields (ERFs) through the large-kernel re-parameterized convolution to attain more discriminative multimodal gait information, thereby enhancing the model performance under complex walking conditions. Experiments have demonstrated that GaitRA significantly improves performance on both CASIA-B and Gait3D datasets. Especially, the proposed method achieves excellent results of 73.1% (Rank-1), 86.4% (Rank-5), 64.3% (mAP), and 36.2% (mINP) on the Gait3D dataset. © The Author(s), under exclusive licence to Springer Science+Business Media, LLC, part of Springer Nature 2024.