An end-to-end speckle matching network for 3D imaging

Speckle projection profilometry (SPP), which is highly suited for dynamic 3D acquisition, can build the global correspondences between stereo images by projecting a single random speckle pattern. But SPP suffers from the low matching accuracy of traditional stereo matching algorithms which limits its 3D measurement quality and precludes the recovery of the fine details of complex surfaces. For enhancing the matching precision of SPP, in this paper, we propose an end-to-end speckle matching network for 3D imaging. The proposed network first leveraged a multi-scale residual subnetwork to synchronously extract feature maps of stereo speckle images from two perspectives. Considering that the cost filtering based on 3D convolution is computationally costly, the 4D cost volume with a quarter of the original resolution is established and implemented cost filtering to achieve higher stereo matching performance with lower computational overhead. In addition, for the dataset of SPP built for supervised deep learning, the label of the sample data only has valid values in the foreground. Therefore, in our work, a simple and fast saliency detection network is integrated into our end-to-end network, which takes as input the features computed from the shared feature extraction subnetwork of the stereo matching network and produces first a low-resolution invalidation mask. The mask is then upsampled and refined with multi-scale multi-level residual layers to generate the final full-resolution mask. This allows our stereo matching network to avoid predicting the invalid pixels in the disparity maps, such as occlusions, backgrounds, thereby implicitly improving the disparity accuracy for valid pixels. The experiment results demonstrated that the proposed method can achieve fast and absolute 3D shape measurement with an accuracy of about 100um through a single speckle pattern.