Developing an Offline and Real-Time Indian Sign Language Recognition System with Machine Learning and Deep Learning

Sign language is a powerful form of communication for humans, and advancements in computer vision systems are driving significant progress in sign language recognition. In the context of Indian sign language (ISL), early research focused on differentiating a limited set of distinct hand signs, often relying on specialized hardware such as sensors and gloves, also most of the works were experimented on the dataset captured under controlled environments. This research aims to enhance communication for the speech and hearing impaired community by recognizing static images of ISL digits and alphabets in both offline and real-time scenarios. To achieve this, two publicly available datasets were used, containing a total of 42,000 sign images and 36,000 static signs, respectively. The dataset1 consists of sign images that were taken under controlled environments, whereas the dataset2 consists of sign images that were taken in different environments with varying backgrounds and lighting conditions. Dataset1 was experimented with and without using preprocessing techniques, while dataset2 underwent similar testing. We employed both machine learning and deep learning with CNN to categorize the ISL alphabets and numbers. In the machine learning approach, image preprocessing techniques such as HSV conversion, skin mask generation, and skin portion extraction and Gabor filtering were used to segment the region of interest, which was then fed to five ML models for sign prediction. In contrast, the DL approach used CNN model. In addition, probability ensemble testing was performed on both datasets to compare the accuracies. Real-time recognition was also conducted using a custom dataset, employing the YOLO-NAS-S model. This study contributes to the advancement of ISL recognition by conducting a comparative analysis of ML algorithms and CNNs, examining their performance with and without preprocessing techniques. © The Author(s), under exclusive licence to Springer Nature Singapore Pte Ltd 2024.