Recognition of Signed Expressions Using Symbolic Aggregate Approximation

Complexity of sign language recognition system grows with growing word vocabulary. Therefore it is advisable to use units smaller than words. Such elements, called subunits, resemble phonemes in spoken language. They are concatenated to form word models. We propose a data-driven procedure for finding subunits in time series representing signed expressions. The procedure consists in: (i) transformation of video material to time series describing hand movements, (ii) using Piecewise Aggregate Approximation (PAA) coefficients to represent subunits, and (iii) applying Symbolic Aggregate Approximation (SAX), which is based on PAA, to obtain appropriate symbolic description. Signed words represented by strings of SAX symbols are classified using nearest neighbour method with Dynamic Time Warping (DTW) technique. We compare the approach with whole-word recognition by presenting ten-fold cross-validation tests on a Polish sign language (PSL) corpus of 30 words. Recognition of new words using small number of examples is also considered. The experiments show superiority of the SAX based approach.