Hand segmentation under different viewpoints by combination of Mask R-CNN with tracking

This paper presents a new method for hand segmentation from images and video. The method based mainly on an advanced technique for instance segmentation (Mask R-CNN) which has been shown very efficient in segmentation task on COCO dataset. However, Mask R-CNN has some limitations. It works on still images, so cannot explore temporal information of the object of interest such as dynamic hand gestures. Second Mask R-CNN usually fails to detect object suffered from motion blur at low resolution as hand. Our proposed method improves Mask R-CNN by integrating a Mean Shift tracker that tracks hands in consecutive frames and removes false alarms. We have also trained another model of Mask R-CNN on cropped regions extended from hand centers to obtain a better accuracy of segmentation. We have evaluated both methods on a self constructed multi-view dataset of hand gestures and show how robust these methods are to view point changes. Experimental results showed that our method achieved better performance than the original Mask R-CNN under different viewpoints.