Depth map reconstruction method based on multi-scale cross-modal feature fusion

Aiming at the reconstruction problem of sparse depth maps，a deep-learning based depth map reconstruction method with multi-level cross-modal feature fusion was proposed． RGB information and depth information were encoded respectively by convolution neural network and adaptive feature fusion was performed in multiple scales． The reconstruction results were further refined by residual learning in the decoding stage． The feature upsampling was guided by the skip connection from depth encoding branch，and the initial depth map was output． At the same time，the rich semantic features extracted by the RGB encoding branch were multiplexed，and the upsampling feature maps output at multiple scales were iteratively upsampled by the pyramid upsampling block． Then，the residual with the final reconstruction result was learned，which improved the quality of reconstruction results．Experimental results on both NYU-Depth-v2 and KITTI dataset show that the proposed method has better depth map reconstruction performance compared with the existing mainstream methods，and generates sharper depth boundaries in visual comparison. © 2023 Huazhong University of Science and Technology. All rights reserved.