Image-based instance segmentation dense point cloud multimodal 3D object detection

3D object detection in autonomous driving systems must confront the problem of inaccurate detection of distant and small objects caused by sparse point clouds in complex environments. Especially in complex traffic environments, single-modal detection methods have difficulty meeting the high accuracy requirements. To address this challenge, this paper proposes a novel multimodal 3D object detection algorithm based on densified point clouds via image instance segmentation. The image is first segmented by instance, and then a virtual point cloud is generated based on the instance results and point cloud projection. Moreover, the class scores of the instances are encoded as additional dimensions of the point cloud to enhance the semantic information. This paper introduces dynamic voxel geometry encoding, which adjusts the size and position of the voxels based on the motion changes in the target object. This adjustment enhances the detection of distant and small objects. In addition, this paper presents a new data augmentation technique to effectively improve the training efficiency and detection performance of the model. Extensive experimental verification shows that this model achieves a 6.2% greater mean average precision (mAP) on the KITTI dataset than does the classic multimodal detection method PointPainting; it performs particularly well in detecting pedestrians and cyclists. These results demonstrate the method’s effectiveness and practicality.