Hand Gesture Recognition with Generalized Hough Transform and DC-CNN Using RealSense

Hand gesture recognition plays an important role in human-computer interaction. With the development of depth cameras, color images combined with depth images can provide richer information for hand gesture recognition. In this paper, we propose a hand gesture recognition system based on the data captured by Intel RealSense Front-Facing Camera SR300. Considering that the pixels in depth images collected by RealSense are not one-to-one to those in color images, the recognition system maps depth images to color images based on generalized Hough transform in order to segment hand from a complex background in color images using the depth information. Then, it recognizes different hand gestures by a novel double-channel convolutional neural network containing two input channels which are color images and depth images. Moreover, we built a hand gesture database of 24 different kinds of hand gestures representing 24 letters in the English alphabet. It contains a total of 168,000 images which are 84,000 RGB images and 84,000 depth images. Experimental results on our newly collected hand gesture database demonstrate the effectiveness of the proposed approach, and the recognition accuracy is 99.4%.