Interactive Multi-Scale Fusion of 2D and 3D Features for Multi-Object Vehicle Tracking

被引:17
作者
Wang, Guangming [1 ,2 ]
Peng, Chensheng [1 ,2 ]
Gu, Yingying [2 ]
Zhang, Jinpeng [3 ]
Wang, Hesheng [1 ,2 ]
机构
[1] Shanghai Jiao Tong Univ, Shanghai Engn Res Ctr Intelligent Control & Manage, Key Lab Marine Intelligent Equipment & Syst, Dept Automat,Key Lab Syst Control & Informat Proc,, Shanghai 200240, Peoples R China
[2] Beijing Inst Control Engn, Space Optoelect Measurement & Percept Lab, Beijing 100190, Peoples R China
[3] China Aerosp Sci & Ind Corp, X Lab, Acad 2, Beijing 100854, Peoples R China
关键词
Multi object tracking; 3D point clouds; feature fusion; computer vision; deep learning;
D O I
10.1109/TITS.2023.3275954
中图分类号
TU [建筑科学];
学科分类号
0813 ;
摘要
Multiple Object Tracking (MOT) is a significant task in autonomous driving. Nonetheless, relying on one single sensor is not robust enough, because one modality tends to fail in some challenging situations. Texture information from RGB cameras and 3D structure information from Light Detection and Ranging (LiDAR) have respective advantages under different circumstances. Therefore, feature fusion from multiple modalities contributes to the learning of discriminative features. However, it is nontrivial to achieve effective feature fusion due to the completely distinct information modality. Previous fusion methods usually fuse the top-level features after the backbones extract the features from different modalities. The feature fusion happens solely once, which limits the information interaction between different modalities. In this paper, we propose multiscale interactive query and fusion between pixel-wise and point-wise features to obtain more discriminative features. In addition, an attention mechanism is utilized to conduct soft feature fusion between multiple pixels and points to avoid inaccurate match problems of previous single pixel-point fusion methods. We introduce PointNet++ to obtain multi-scale deep representations of point clouds and make it adaptive to our proposed interactive feature fusion between multi-scale features of images and point clouds. Through the interaction module, each modality can integrate more complementary information from the other modality. Besides, we explore the effectiveness of pre-training on each single modality and fine-tuning on the fusion-based model. Our method can achieve 90.32% MOTA and 72.44% HOTA on the KITTI benchmark and outperform other approaches without using multi-scale soft feature fusion.
引用
收藏
页码:10618 / 10627
页数:10
相关论文
共 47 条
[21]  
Lu YF, 2021, AAAI CONF ARTIF INTE, V35, P4276
[22]   HOTA: A Higher Order Metric for Evaluating Multi-object Tracking [J].
Luiten, Jonathon ;
Osep, Aljosa ;
Dendorfer, Patrick ;
Torr, Philip ;
Geiger, Andreas ;
Leal-Taixe, Laura ;
Leibe, Bastian .
INTERNATIONAL JOURNAL OF COMPUTER VISION, 2021, 129 (02) :548-578
[23]   Stabilization and Validation of 3D Object Position Using Multimodal Sensor Fusion and Semantic Segmentation [J].
Muresan, Mircea Paul ;
Giosan, Ion ;
Nedevschi, Sergiu .
SENSORS, 2020, 20 (04)
[24]   A Multimodality Fusion Deep Neural Network and Safety Test Strategy for Intelligent Vehicles [J].
Nie, Jian ;
Yan, Jun ;
Yin, Huilin ;
Ren, Lei ;
Meng, Qian .
IEEE TRANSACTIONS ON INTELLIGENT VEHICLES, 2021, 6 (02) :310-322
[25]   Quasi-Dense Similarity Learning for Multiple Object Tracking [J].
Pang, Jiangmiao ;
Qiu, Linlu ;
Li, Xia ;
Chen, Haofeng ;
Li, Qi ;
Darrell, Trevor ;
Yu, Fisher .
2021 IEEE/CVF CONFERENCE ON COMPUTER VISION AND PATTERN RECOGNITION, CVPR 2021, 2021, :164-173
[26]  
Qi CR, 2017, ADV NEUR IN, V30
[27]  
Scheidegger S, 2018, IEEE INT VEH SYM, P433
[28]   Deep Network Flow for Multi-Object Tracking [J].
Schulter, Samuel ;
Vernaza, Paul ;
Choi, Wongun ;
Chandraker, Manmohan .
30TH IEEE CONFERENCE ON COMPUTER VISION AND PATTERN RECOGNITION (CVPR 2017), 2017, :2730-2739
[29]  
Sharma S, 2018, IEEE INT CONF ROBOT, P3508, DOI 10.1109/ICRA.2018.8461018
[30]   Fast Online Tracking With Detection Refinement [J].
Shen, Jianbing ;
Yu, Dajiang ;
Deng, Leyao ;
Dong, Xingping .
IEEE TRANSACTIONS ON INTELLIGENT TRANSPORTATION SYSTEMS, 2018, 19 (01) :162-173