MPTFormer: Toward Robust Arm Gesture Pose Tracking Using Dual-View Radar System

被引:4
作者
Chen, Lin [1 ,2 ]
Guo, Xuemei [1 ,2 ]
Wang, Guoli [1 ,2 ]
机构
[1] Sun Yat Sen Univ, Sch Comp Sci & Engn, Guangzhou 510006, Peoples R China
[2] Sun Yat Sen Univ, Key Lab Machine Intelligence & Adv Comp, Minist Educ, Guangzhou 510006, Peoples R China
基金
中国国家自然科学基金;
关键词
Arm gesture pose tracking; dual-view; millimeter-wave radar; transformer; MICRO-DOPPLER CLASSIFICATION; MMWAVE RADARS; FUSION;
D O I
10.1109/JSEN.2023.3335606
中图分类号
TM [电工技术]; TN [电子技术、通信技术];
学科分类号
0808 ; 0809 ;
摘要
Locating human joints using low-cost commercial millimeter-wave (mmWave) radars is an emerging technology. Compared to single-view radar, dual-view radar fusion is more efficient for robust arm gesture pose tracking since it can capture a wider area of human reflection and radial velocity in both directions. One challenging task for dual-view radar fusion is to effectively utilize the reflection points' spatial and velocity information and focus perceptual cues related to joint position and velocity for accurate joint tracking. In this article, we propose a millimeter-wave pose tracking transformer (MPTFormer), a method for arm gesture pose tracking using both front- and side-view mmWave radars. Specifically, MPTFormer consists of a multidomain fusion transformer (MFFormer) and a joint tracking transformer (JTFormer). In the MFFormer, joint position and velocity estimation tasks are set, and the range-azimuth map (RAM) and range-Doppler map (RDM) of the dual-view radar are set as token embeddings, which interacted with joint position and velocity tokens for learning perceptual cues related to joint position and velocity as well as joint constraint relationships. Meanwhile, the nonlocal dependencies between dual-view radar and its multidomain features are mined to obtain good global feature representation. In the JTFormer, a hybrid learning architecture of the temporal transformer and long short-term memory (LSTM) is designed, which tracks the joint position according to the global and local features of the joint in the temporal dimension. Finally, in the pose-tracking task containing 12 different arm gestures, the proposed method can achieve 25.2 mm mean per joint position error (MPJPE) and 19.1 mm Procrustes analysis-MPJPE (P-MPJPE).
引用
收藏
页码:1051 / 1064
页数:14
相关论文
共 49 条
[1]  
Ahmed S. S., 2012, Proceedings of the 2012 6th European Conference on Antennas and Propagation (EuCAP), P1755, DOI 10.1109/EuCAP.2012.6206694
[2]   Fast and Scalable Human Pose Estimation using mmWave Point Cloud [J].
An, Sizhe ;
Ogras, Umit Y. .
PROCEEDINGS OF THE 59TH ACM/IEEE DESIGN AUTOMATION CONFERENCE, DAC 2022, 2022, :889-894
[3]   MARS: mm Wave-based Assistive Rehabilitation System for Smart Healthcare [J].
An, Sizhe ;
Ogras, Umit Y. .
ACM TRANSACTIONS ON EMBEDDED COMPUTING SYSTEMS, 2021, 20 (05)
[4]   Cross Vision-RF Gait Re-identification with Low-cost RGB-D Cameras and mmWave Radars [J].
Cao, Dongjiang ;
Liu, Ruofeng ;
Li, Hao ;
Wang, Shuai ;
Jiang, Wenchao ;
Lu, Chris Xiaoxuan .
PROCEEDINGS OF THE ACM ON INTERACTIVE MOBILE WEARABLE AND UBIQUITOUS TECHNOLOGIES-IMWUT, 2022, 6 (03)
[5]   A Joint Global-Local Network for Human Pose Estimation With Millimeter Wave Radar [J].
Cao, Zhongping ;
Ding, Wen ;
Chen, Rihui ;
Zhang, Jianxiong ;
Guo, Xuemei ;
Wang, Guoli .
IEEE INTERNET OF THINGS JOURNAL, 2023, 10 (01) :434-446
[6]   End-to-End Object Detection with Transformers [J].
Carion, Nicolas ;
Massa, Francisco ;
Synnaeve, Gabriel ;
Usunier, Nicolas ;
Kirillov, Alexander ;
Zagoruyko, Sergey .
COMPUTER VISION - ECCV 2020, PT I, 2020, 12346 :213-229
[7]   Real-Time Short-Range Human Posture Estimation Using mmWave Radars and Neural Networks [J].
Cui, Han ;
Dahnoun, Naim .
IEEE SENSORS JOURNAL, 2022, 22 (01) :535-543
[8]  
Devlin J, 2019, 2019 CONFERENCE OF THE NORTH AMERICAN CHAPTER OF THE ASSOCIATION FOR COMPUTATIONAL LINGUISTICS: HUMAN LANGUAGE TECHNOLOGIES (NAACL HLT 2019), VOL. 1, P4171
[9]   Radar-Based 3D Human Skeleton Estimation by Kinematic Constrained Learning [J].
Ding, Wen ;
Cao, Zhongping ;
Zhang, Jianxiong ;
Chen, Rihui ;
Guo, Xuemei ;
Wang, Guoli .
IEEE SENSORS JOURNAL, 2021, 21 (20) :23174-23184
[10]  
Dosovitskiy A, 2021, Arxiv, DOI arXiv:2010.11929