Demo: The Sound of Silence: End-to-End Sign Language Recognition Using SmartWatch

Sign Language is a natural and fully-fledged communication method for deaf and hearing-impaired people. In this demo, we propose the first SmartWatch-based American sign language (ASL) recognition system, which is more comfortable, portable and user-friendly and offers accessibility anytime, anywhere. This system is based on the intuitive idea that each sign has its specific motion pattern which can be transformed into unique gyroscope and accelerometer signals and then analyzed and learned by using Long-Short term memory recurrent neural network (LSTM-RNN) trained with connectionist temporal classification (CTC). In this way, signs and context information can be correctly recognized based on an off the-shelf device (e.g. SmartWatch, Smartphone). The experiments show that, in the Known user split task, our system reaches an average word error rate of 7.29% to recognize 73 sentences formed by 103 ASL signs and achieves detection ratio up to 93.7% for a single sign. The result also shows our system has a good adaptation, even including new users, it can achieve an average word error rate of 21.6% at the sentence level and reach an average detection ratio of 79.4%. Moreover, our system performs real time ASL translation, outputting the speech within 1.69 seconds for a sentence of 12 signs in average.