An Extensible Multi-Sensor Fusion Framework for 3D Imaging

被引:2
作者
Siddiqui, Talha Ahmad [1 ]
Madhok, Rishi [1 ]
O'Toole, Matthew [1 ]
机构
[1] Carnegie Mellon Univ, Pittsburgh, PA 15213 USA
来源
2020 IEEE/CVF CONFERENCE ON COMPUTER VISION AND PATTERN RECOGNITION WORKSHOPS (CVPRW 2020) | 2020年
关键词
LIDAR;
D O I
10.1109/CVPRW50498.2020.00512
中图分类号
TP18 [人工智能理论];
学科分类号
081104 ; 0812 ; 0835 ; 1405 ;
摘要
Many autonomous vehicles rely on an array of sensors for safe navigation, where each sensor captures different visual attributes from the surrounding environment. For example, a single conventional camera captures high-resolution images but no 3D information; a LiDAR provides excellent range information but poor spatial resolution; and a prototype single-photon LiDAR (SP-LiDAR) can provide a dense but noisy representation of the 3D scene. Although the outputs of these sensors vary dramatically (e.g., 2D images, point clouds, 3D volumes), they all derive from the same 3D scene. We propose an extensible sensor fusion framework that (1) lifts the sensor output to volumetric representations of the 3D scene, (2) fuses these volumes together, and (3) processes the resulting volume with a deep neural network to generate a depth (or disparity) map. Although our framework can potentially extend to many types of sensors, we focus on fusing combinations of three imaging systems: monocular/stereo cameras, regular LiDARs, and SP-LiDARs. To train our neural network, we generate a synthetic dataset through CARLA that contains the individual measurements. We also conduct various fusion ablation experiments and evaluate the results of different sensor combinations.
引用
收藏
页码:4344 / 4353
页数:10
相关论文
共 50 条
[21]   A Survey of the Multi-Sensor Fusion Object Detection Task in Autonomous Driving [J].
Wang, Hai ;
Liu, Junhao ;
Dong, Haoran ;
Shao, Zheng .
SENSORS, 2025, 25 (09)
[22]   An Engineering Solution for Multi-sensor Fusion SLAM in Indoor and Outdoor Scenes [J].
Jiang, Fengyang ;
Cheng, Yao ;
Wang, Huaizhen ;
Han, Zhe ;
Huang, Yang ;
Zhou, Fengyu ;
Jiang, Jiaju .
2024 43RD CHINESE CONTROL CONFERENCE, CCC 2024, 2024, :4522-4528
[23]   Rice Row Recognition and Navigation Control Based on Multi-sensor Fusion [J].
He J. ;
He J. ;
Luo X. ;
Li W. ;
Man Z. ;
Feng D. .
Nongye Jixie Xuebao/Transactions of the Chinese Society for Agricultural Machinery, 2022, 53 (03) :18-26and137
[24]   Research on Multi-Sensor Fusion SLAM Algorithm Based on Improved Gmapping [J].
Tian, Chengjun ;
Liu, Haobo ;
Liu, Zhe ;
Li, Hongyang ;
Wang, Yuyu .
IEEE ACCESS, 2023, 11 :13690-13703
[25]   Improved Localization Algorithm Based on Multi-Sensor Fusion for Shopping Robots [J].
Zhao, Yawen ;
Solihin, Mahmud Iwan ;
Yang, Defu ;
Cai, Bingyu ;
Liu, Chang .
2024 3RD INTERNATIONAL SYMPOSIUM ON SENSOR TECHNOLOGY AND CONTROL, ISSTC, 2024, :43-48
[26]   A Critical Review of Deep Learning-Based Multi-Sensor Fusion Techniques [J].
Marsh, Benedict ;
Sadka, Abdul Hamid ;
Bahai, Hamid .
SENSORS, 2022, 22 (23)
[27]   Multi-sensor fusion based wheeled robot research on indoor positioning method [J].
Teng, Xiao ;
Shen, Zhenjiang ;
Huang, Lin ;
Li, Hui ;
Li, Wankai .
RESULTS IN ENGINEERING, 2024, 22
[28]   A Feature Level Multi-Sensor Information Fusion Strategy for Land Region Classification [J].
Schierl, Jonathan ;
Asari, Vijayan ;
Singer, Nina ;
Aspiras, Theus ;
Stokes, Andrew ;
Keaffaber, Brett ;
Van Rynbach, Andre ;
Decker, Kevin ;
Rabb, David .
2021 IEEE APPLIED IMAGERY PATTERN RECOGNITION WORKSHOP (AIPR), 2021,
[29]   A method for apron mapping based on feature point extraction and multi-sensor fusion [J].
Niu, Guochen ;
Luan, Xiangyu .
Hangkong Xuebao/Acta Aeronautica et Astronautica Sinica, 2024, 45
[30]   Multi-Sensor Fusion for Aerial Robots in Industrial GNSS-Denied Environments [J].
Carrasco, Paloma ;
Cuesta, Francisco ;
Caballero, Rafael ;
Perez-Grau, Francisco J. ;
Viguria, Antidio .
APPLIED SCIENCES-BASEL, 2021, 11 (09)