Multi-Stream Isolated Sign Language Recognition Based on Finger Features Derived from Pose Data

被引:3
作者
Akdag, Ali [1 ]
Baykan, Omer Kaan [2 ]
机构
[1] Tokat Gaziosmanpasa Univ, Dept Comp Engn, Tasliciftlik Campus, TR-60250 Tokat, Turkiye
[2] Konya Tech Univ, Dept Comp Engn, TR-42250 Konya, Turkiye
关键词
sign language recognition; deep learning; feature fusion;
D O I
10.3390/electronics13081591
中图分类号
TP [自动化技术、计算机技术];
学科分类号
0812 ;
摘要
This study introduces an innovative multichannel approach that focuses on the features and configurations of fingers in isolated sign language recognition. The foundation of this approach is based on three different types of data, derived from finger pose data obtained using MediaPipe and processed in separate channels. Using these multichannel data, we trained the proposed MultiChannel-MobileNetV2 model to provide a detailed analysis of finger movements. In our study, we first subject the features extracted from all trained models to dimensionality reduction using Principal Component Analysis. Subsequently, we combine these processed features for classification using a Support Vector Machine. Furthermore, our proposed method includes processing body and facial information using MobileNetV2. Our final proposed sign language recognition method has achieved remarkable accuracy rates of 97.15%, 95.13%, 99.78%, and 95.37% on the BosphorusSign22k-general, BosphorusSign22k, LSA64, and GSL datasets, respectively. These results underscore the generalizability and adaptability of the proposed method, proving its competitive edge over existing studies in the literature.
引用
收藏
页数:28
相关论文
共 82 条
[1]  
Adaloglou N, 2021, Arxiv, DOI [arXiv:2007.12530, DOI 10.1109/TMM.2021.3070438]
[2]  
Aljuhani R, 2023, ARAB J SCI ENG, V48, P2147, DOI 10.1007/s13369-022-07144-2
[3]   Deep Learning Technology to Recognize American Sign Language Alphabet [J].
Alsharif, Bader ;
Altaher, Ali Salem ;
Altaher, Ahmed ;
Ilyas, Mohammad ;
Alalwany, Easa .
SENSORS, 2023, 23 (18)
[4]   The curse(s) of dimensionality [J].
Altman, Naomi ;
Krzywinski, Martin .
NATURE METHODS, 2018, 15 (06) :399-400
[5]  
Aly S., 2014, Communications in Computer and Information Science, VVolume 488
[6]   Isolated Arabic Sign Language Recognition Using a Transformer-based Model and Landmark Keypoints [J].
Alyami, Sarah ;
Luqman, Hamzah ;
Hammoudeh, Mohammad .
ACM TRANSACTIONS ON ASIAN AND LOW-RESOURCE LANGUAGE INFORMATION PROCESSING, 2024, 23 (01)
[7]   Improving support vector machine classifiers by modifying kernel functions [J].
Amari, S ;
Wu, S .
NEURAL NETWORKS, 1999, 12 (06) :783-789
[8]  
[Anonymous], International day of sign languages
[9]   A machine learning approach to circumventing the curse of dimensionality in discontinuous time series machine data [J].
Aremu, Oluseun Omotola ;
Hyland-Wood, David ;
McAree, Peter Ross .
RELIABILITY ENGINEERING & SYSTEM SAFETY, 2020, 195
[10]  
Barczak A.L.C., 2011, Research Letters in the Information and Mathematical Sciences, VVolume 15