Improving Sign Language Recognition Performance Using Multimodal Data

Sign language is a language primarily used by the hearing-impaired for communication and has more than 200 variations worldwide. Communication is nearly impossible between signers of different variations. Moreover for a person with normal hearing, learning sign language can be challenging because the syntax of sign language differs from that of natural language. Translation of signs by machine learning offers potential solutions to these challenges, facilitating communication for everyone. This study attempts to enhance the performance of the existing state-of-the-art sign language translation model, Gloss attention SLT network (GASLT), through the integration of a multimodal approach. By combining RGB video with 3D pose data extracted using Mediapipe in an innovative way, our multimodal method significantly enhances the GASLT's results. We conducted two experiments involving the fusion of video and pose data with the GASLT model. These experiments led to an 18.39% improvement in the model's BLEU score compared to the original model, showcasing the effectiveness of the multimodal approach in enhancing sign translation.