Multi-Scale Guided Mask Refinement for Coarse-to-Fine RGB-D Perception

Pixel-level object segmentation is highly desired in many vision applications. Although segmentation methods purely based on visual input have achieved great success in the past decade, their further improvement is still hindered by the intrinsic drawback of color camouflage. With the rapid development and wide deployment of depth sensors, depth assisted methods are increasingly popular in visual perception systems. It is expected that RGB-D based methods can lead to significant performance improvements because color and depth are naturally complementary. However, how to merge color and depth modalities for segmentation with both high efficiency and high accuracy remains an open problem to be addressed. In this letter, we propose to divide the segmentation process into "coarse" and "refining" stages because a coarse segmentation can be easily obtained by various light-weight methods. In this way, we can tackle this problem by focusing on the refinement of coarse segmentations. In particular, we propose a multi-scale approach that selectively inherits the effective features of both edge-preserving filtering and deep neural networks. The proposed approach is evaluated on several bench-mark datasets, respectively, using the coarse segmentations from background subtraction and object detection as the input. Numerous results indicate that our approach can achieve significant accuracy improvements compared to other alternatives, demonstrating superior edge-preserving capability. Besides an effective method for merging RGB-D information, our study on the capability of coarse-to-fine refinement also brings new inspirations for designing light-weight perception systems.