Combining Depth-estimation-based Multi-spectral Photometric Stereo and SLAM for Real-time Dense 3D Reconstruction

Obtaining dense 3D reconstruction with low computational cost is one of the important goals in the field of Simultaneous Localization and Mapping (SLAM). In this paper we propose a dense 3D reconstruction framework from monocular multi-spectral video sequences using jointly semi-dense SLAM and depth-estimation-based Multi-spectral Photometric Stereo approaches. Starting from multi-spectral video, we use SALM to reconstruct a semi-dense 3D shape that will be densified. Then the depth maps estimated via conditional Generative Adversarial Nets (cGAN) are fed as priors into optimization-based multi-spectral photometric stereo for dense surface normal recovery. Finally, we use camera poses for view conversion in fusion procedure where we combine the relative sparse point cloud with the dense surface normal to get a dense point cloud. Experiments show that our method can effectively obtain denser 3D reconstruction.