Continuous Chinese Sign Language Recognition with CNN-LSTM

被引:17
作者
Yang, Su [1 ]
Zhu, Qing [1 ]
机构
[1] Beijing Univ Technol, Fac Informat Technol, 100 Ping Leyuan, Beijing 100124, Peoples R China
来源
NINTH INTERNATIONAL CONFERENCE ON DIGITAL IMAGE PROCESSING (ICDIP 2017) | 2017年 / 10420卷
关键词
Sign language recognition; convolutional neural network; recurrent neural network; Long Short-Term Memory;
D O I
10.1117/12.2281671
中图分类号
TB8 [摄影技术];
学科分类号
0804 ;
摘要
The goal of sign language recognition (SLR) is to translate the sign language into text, and provide a convenient tool for the communication between the deaf-mute and the ordinary. In this paper, we formulate an appropriate model based on convolutional neural network (CNN) combined with Long Short-Term Memory (LSTM) network, in order to accomplish the continuous recognition work. With the strong ability of CNN, the information of pictures captured from Chinese sign language (CSL) videos can be learned and transformed into vector. Since the video can be regarded as an ordered sequence of frames, LSTM model is employed to connect with the fully-connected layer of CNN. As a recurrent neural network (RNN), it is suitable for sequence learning tasks with the capability of recognizing patterns defined by temporal distance. Compared with traditional RNN, LSTM has performed better on storing and accessing information. We evaluate this method on our self-built dataset including 40 daily vocabularies. The experimental results show that the recognition method with CNN-LSTM can achieve a high recognition rate with small training sets, which will meet the needs of real-time SLR system.
引用
收藏
页数:7
相关论文
共 19 条
[1]  
Akyol S., 2001, Proceedings of the IASTED International Conference Signal Processing, Pattern Recognition, and Applications, P48
[2]   Video-based signer-independent Arabic sign language recognition using hidden Markov models [J].
AL-Rousan, M. ;
Assaleh, K. ;
Tala'a, A. .
APPLIED SOFT COMPUTING, 2009, 9 (03) :990-999
[3]  
[Anonymous], MODELLING RECOGNITIO
[4]  
[Anonymous], 1995, Technical report
[5]  
[Anonymous], OPENCV 2 3 2 DOC
[6]  
[Anonymous], 2012, COMPUTER SCI
[7]  
[Anonymous], 2012, COURSERA NEURAL NETW
[8]  
[Anonymous], 2012, SUPERVISED SEQUENCE
[9]  
Chang CC, 2004, 2004 IEEE INTERNATIONAL CONFERENCE ON MULTIMEDIA AND EXP (ICME), VOLS 1-3, P1187
[10]   Sign language recognition based on HMM/ANN/DP [J].
Gao, W ;
Ma, JY ;
Wu, JQ ;
Wang, CL .
INTERNATIONAL JOURNAL OF PATTERN RECOGNITION AND ARTIFICIAL INTELLIGENCE, 2000, 14 (05) :587-602