Pakistan sign language recognition: leveraging deep learning models with limited dataset

Sign language is the predominant form of communication among a large group of society. The nature of sign languages is visual. This makes them very different from spoken languages. Unfortunately, very few able people can understand sign language making communication with the hearing-impaired extremely difficult. Research in the field of sign language recognition can help reduce the barrier between deaf and able people. A lot of work has been done on sign language recognition for numerous languages such as American sign language and Chinese sign language. Unfortunately, very little to no work has been done for Pakistan Sign Language. Any contribution in Pakistan Sign Language recognition is limited to static images instead of gestures. Furthermore, the dataset available for this language is very small in terms of the number of examples per word which makes it very difficult to train deep networks that require a considerable amount of training data. Data Augmentation techniques help the network generalize better by providing more variety in the training data. In this paper, a pipeline for the Pakistan Sign Language recognition system is proposed that incorporates an augmentation unit. To validate the effectiveness of the proposed pipeline, three deep learning models, C3D, I3D, and TSM are used. Results show that translation and rotation are the two best augmentation techniques for the Pakistan Sign Language dataset. The models trained using our data-augment-supported pipeline outperform other methods that only used the original data. The most suitable model is C3D which not only produced an accuracy of 93.33% but also has a low training time as compared to other models.