Isolated sign language recognition through integrating pose data and motion history images

被引:0
|
作者
Akdag, Ali [1 ]
Baykan, Omer Kaan [2 ]
机构
[1] Tokat Gaziosmanpasa Univ, Dept Comp Engn, Tokat, Turkiye
[2] Konya Tech Univ, Dept Comp Engn, Konya, Turkiye
关键词
Sign language recognition; Deep learning; Motion history image; Feature fusion;
D O I
10.7717/peerj-cs.2054
中图分类号
TP18 [人工智能理论];
学科分类号
081104 ; 0812 ; 0835 ; 1405 ;
摘要
This article presents an innovative approach for the task of isolated sign language recognition (SLR); this approach centers on the integration of pose data with motion history images (MHIs) derived from these data. Our research combines spatial information obtained from body, hand, and face poses with the comprehensive details provided by three-channel MHI data concerning the temporal dynamics of the sign. Particularly, our developed finger pose-based MHI (FP-MHI) feature significantly enhances the recognition success, capturing the nuances of finger movements and gestures, unlike existing approaches in SLR. This feature improves the accuracy and reliability of SLR systems by more accurately capturing the fine details and richness of sign language. Additionally, we enhance the overall model accuracy by predicting missing pose data through linear interpolation. Our study, based on the randomized leaky rectified linear unit (RReLU) enhanced ResNet-18 model, successfully handles the interaction between manual and non-manual features through the fusion of extracted features and classification with a support vector machine (SVM). This innovative integration demonstrates competitive and superior results compared to current methodologies in the field of SLR across various datasets, including BosphorusSign22k-general, BosphorusSign22k, LSA64, and GSL, in our experiments.
引用
收藏
页数:44
相关论文
共 50 条
  • [1] Isolated sign language recognition through integrating pose data and motion history images
    Akdağ A.
    Baykan Ö.K.
    PeerJ Computer Science, 2024, 10
  • [2] RECOGNITION OF SIGN LANGUAGE MOTION IMAGES
    TAMURA, S
    KAWASAKI, S
    PATTERN RECOGNITION, 1988, 21 (04) : 343 - 353
  • [3] Using Motion History Images With 3D Convolutional Networks in Isolated Sign Language Recognition
    Mercanoglu Sincan, Ozge
    Keles, Hacer Yalim
    IEEE ACCESS, 2022, 10 : 18608 - 18618
  • [4] Hand pose aware multimodal isolated sign language recognition
    Razieh Rastgoo
    Kourosh Kiani
    Sergio Escalera
    Multimedia Tools and Applications, 2021, 80 : 127 - 163
  • [5] Hand pose aware multimodal isolated sign language recognition
    Rastgoo, Razieh
    Kiani, Kourosh
    Escalera, Sergio
    MULTIMEDIA TOOLS AND APPLICATIONS, 2021, 80 (01) : 127 - 163
  • [6] Pose Recognition using Cross Correlation for Static Images of Urdu Sign Language
    Sami, Muhammad
    Ahmed, Habib
    Wahid, Ali
    Siraj, Usama
    Ahmed, Faizan
    Shahid, Shahab
    Shah, Syed Irtiza Ali
    2014 INTERNATIONAL CONFERENCE ON ROBOTICS AND EMERGING ALLIED TECHNOLOGIES IN ENGINEERING (ICREATE), 2014, : 200 - 204
  • [7] Multi-Stream Isolated Sign Language Recognition Based on Finger Features Derived from Pose Data
    Akdag, Ali
    Baykan, Omer Kaan
    ELECTRONICS, 2024, 13 (08)
  • [8] Isolated Sign Language Recognition based on Tree Structure Skeleton Images
    Laines, David
    Gonzalez-Mendoza, Miguel
    Ochoa-Ruiz, Gilberto
    Bejarano, Gissella
    2023 IEEE/CVF CONFERENCE ON COMPUTER VISION AND PATTERN RECOGNITION WORKSHOPS, CVPRW, 2023, : 276 - 284
  • [9] Turkish Sign Language Recognition Application Using Motion History Image
    Yalcinkaya, Ozge
    Atvar, Anil
    Duygulu, Pinar
    2016 24TH SIGNAL PROCESSING AND COMMUNICATION APPLICATION CONFERENCE (SIU), 2016, : 801 - 804
  • [10] Hand pose estimation for American Sign Language recognition
    Isaacs, J
    Foo, S
    PROCEEDINGS OF THE THIRTY-SIXTH SOUTHEASTERN SYMPOSIUM ON SYSTEM THEORY, 2004, : 132 - 136