3D entity-based stereo matching with ground control points and joint second-order smoothness prior

Disparity estimation for a scene with complex geometric characteristics such as slanted or highly curved surfaces is a basic and important issue in stereo matching. Traditional methods often use first-order smoothness priors that always lead to low-curvature frontal-parallel disparity maps. We propose a stereo framework that views the scene as a set of 3D entities with compact and smooth disparity distributions. The 3D entity-based representation enables some contributions to obtain a precise disparity estimation. A GCPs-plane constraint based on ground control points is used to strengthen the compact distributions of the disparities in each entity by restricting the scope of the disparity variance and reducing matching ambiguities in repetitive or low-texture areas. Furthermore, we have formulated a joint second-order smoothness prior, which combines a geometric weight with the derivative of disparity values. This prior encourages smooth disparity variations inside each entity and means that each entity is biased towards being a 3D planar surface. Segmentation is incorporated as soft constraint by effectively fusing the advantages of the image color gradient and GCPs-plane. This avoids blending of the foreground and background and retains only the disparity discontinuities from geometrically smooth regions with strong texture gradients. Our framework is formulated as a maximum a posteriori probability estimation problem that is optimized using the fusion-move approach. Evaluation results on the Middlebury benchmark show that the proposed method ranks second among the approximately listed algorithms. In addition, it performs well in real-world scenes.