Automatic Indian sign language recognition using MediaPipe holistic and LSTM network

Sign language is a reliable medium of communication that deaf and mute individuals use to express themselves and connect with society. However, communication between a person who knows sign language and someone unfamiliar with sign language can be difficult. Indian Sign Language is slowly gaining popularity but faces similar problems. In this paper, a method for recognizing isolated words in Indian Sign Language (ISL) using MediaPipe holistic pipeline for feature extraction and Long-Short Term Memory (LSTM) network is proposed. The proposed method is evaluated on a large-scale ISL video dataset, INCLUDE, along with its smaller subset, INCLUDE-50. The proposed method achieves 94.8% and 87.4% accuracy on INCLUDE-50 and INCLUDE, respectively. Furthermore, a macro averaged F1-score of 93.5% and 86.6% is obtained on the INCLUDE-50 and INCLUDE, respectively. Both these results outperform the current state of the art performance on the respective datasets.