Recognition of Finger Spelling from Color Images Using Deep Learning

We have developed a system to recognize finger spelling in Japanese sign language using deep neural networks. As deep neural networks, we adopt Faster R-CNN. By defining an output class for each finger letter and learning the object detection network, it is possible to output where the finger letter exists in the input image. This method does not require depth cameras, magnetic sensors, or other special equipment when used. Furthermore, this does not require preprocessing that extracts the hand region using human skin colored regions and color gloves used in other methods using color images. We synthesized a training data set by processing images taken with Kinect. As the test data, we input images of performing finger letters into the trained network and check the score of the output area and class.