Deep learning based multi-view dense matching with joint depth and surface normal estimation

In recent years, deep learning-based multi-view stereo matching methods have demonstrated significant potential in 3D reconstruction tasks. Ho wever, they still exhibit limitations in recovering fine geometrie details of scènes. In some traditional multi-view stereo matching methods, surface normal of ten serves as a crucial geometrie constraint to assist in finer depth inference. Nevertheless, the surface normal information, which encapsulates the geometrie information of the scène, has not been fully utilized in modern learning-based methods. This paper introduces a deep learning-based joint depth and surface normal estimation method for multi-view dense matching and 3D scène reconstruction task. The proposed method employs a multi-stage pyramid structure to simultaneously infer depth and surface normal from multi-view images and promote their joint optimization. It consists of a feature extraction module, a normal-assisted depth estimation module, a depth-assisted normal estimation module, and a depth-normal joint optimization module. Specifically, the depth estimation module constructs a geometry-aware cost volume by integrating surface normal information for fine depth estimation. The normal estimation module utilizes depth constraints to build a local cost volume for inferring fine-grained normal maps. The joint optimization module further enhances the geometrie consistency between depth and normal estimation. Experimental results on the WHU-OMVS dataset demonstrate that the proposed method performs exceptionally well in both depth and surface normal estimation, outperforming existing methods. Furthermore, the 3D reconstruction results on two different datasets indicate that the proposed method effectively recovers the geometrie structures of both local high-curvature areas and global planar regions, contributing to well-structured and high-quality 3D scène models. © 2025 SinoMaps Press. All rights reserved.