An Extensible Multi-Sensor Fusion Framework for 3D Imaging

被引:1
作者
Siddiqui, Talha Ahmad [1 ]
Madhok, Rishi [1 ]
O'Toole, Matthew [1 ]
机构
[1] Carnegie Mellon Univ, Pittsburgh, PA 15213 USA
来源
2020 IEEE/CVF CONFERENCE ON COMPUTER VISION AND PATTERN RECOGNITION WORKSHOPS (CVPRW 2020) | 2020年
关键词
LIDAR;
D O I
10.1109/CVPRW50498.2020.00512
中图分类号
TP18 [人工智能理论];
学科分类号
081104 ; 0812 ; 0835 ; 1405 ;
摘要
Many autonomous vehicles rely on an array of sensors for safe navigation, where each sensor captures different visual attributes from the surrounding environment. For example, a single conventional camera captures high-resolution images but no 3D information; a LiDAR provides excellent range information but poor spatial resolution; and a prototype single-photon LiDAR (SP-LiDAR) can provide a dense but noisy representation of the 3D scene. Although the outputs of these sensors vary dramatically (e.g., 2D images, point clouds, 3D volumes), they all derive from the same 3D scene. We propose an extensible sensor fusion framework that (1) lifts the sensor output to volumetric representations of the 3D scene, (2) fuses these volumes together, and (3) processes the resulting volume with a deep neural network to generate a depth (or disparity) map. Although our framework can potentially extend to many types of sensors, we focus on fusing combinations of three imaging systems: monocular/stereo cameras, regular LiDARs, and SP-LiDARs. To train our neural network, we generate a synthetic dataset through CARLA that contains the individual measurements. We also conduct various fusion ablation experiments and evaluate the results of different sensor combinations.
引用
收藏
页码:4344 / 4353
页数:10
相关论文
共 50 条
[21]   A Critical Review of Deep Learning-Based Multi-Sensor Fusion Techniques [J].
Marsh, Benedict ;
Sadka, Abdul Hamid ;
Bahai, Hamid .
SENSORS, 2022, 22 (23)
[22]   Multi-sensor fusion based wheeled robot research on indoor positioning method [J].
Teng, Xiao ;
Shen, Zhenjiang ;
Huang, Lin ;
Li, Hui ;
Li, Wankai .
RESULTS IN ENGINEERING, 2024, 22
[23]   A Feature Level Multi-Sensor Information Fusion Strategy for Land Region Classification [J].
Schierl, Jonathan ;
Asari, Vijayan ;
Singer, Nina ;
Aspiras, Theus ;
Stokes, Andrew ;
Keaffaber, Brett ;
Van Rynbach, Andre ;
Decker, Kevin ;
Rabb, David .
2021 IEEE APPLIED IMAGERY PATTERN RECOGNITION WORKSHOP (AIPR), 2021,
[24]   A method for apron mapping based on feature point extraction and multi-sensor fusion [J].
Niu, Guochen ;
Luan, Xiangyu .
Hangkong Xuebao/Acta Aeronautica et Astronautica Sinica, 2024, 45
[25]   Multi-Sensor Fusion for Aerial Robots in Industrial GNSS-Denied Environments [J].
Carrasco, Paloma ;
Cuesta, Francisco ;
Caballero, Rafael ;
Perez-Grau, Francisco J. ;
Viguria, Antidio .
APPLIED SCIENCES-BASEL, 2021, 11 (09)
[26]   Multi-sensor information fusion in Internet of Vehicles based on deep learning: A review [J].
Tian, Di ;
Li, Jiabo ;
Lei, Jingyuan .
NEUROCOMPUTING, 2025, 614
[27]   Multi-Sensor Classification Framework of Urban Vegetation for Improving Ecological Services Management [J].
Tiwari, Arti ;
Kira, Oz ;
Bamah, Julius ;
Boneh, Hagar ;
Karnieli, Arnon .
REMOTE SENSING, 2024, 16 (06)
[28]   A 3D lidar sensor for volumetric imaging in highly backscattering media [J].
Elkins, WP ;
Ulich, BL ;
Lawrence, JG ;
Linton, HE .
Laser Radar Technology and Applications X, 2005, 5791 :17-28
[29]   A Framework of Wearable Sensor-System Development for Urban 3D Modeling [J].
Choi, Kanghyeok ;
Kim, Changjae .
APPLIED SCIENCES-BASEL, 2022, 12 (18)
[30]   Highly robust and accurate multi-sensor fusion localization system for complex and challenging scenarios [J].
Lan, Zhengyang ;
Wang, Jibo ;
Shen, Zhaohui ;
Fang, Zheng .
MEASUREMENT, 2024, 235