Fusion of Spectral and Prosodic Information using Combined Error Optimization for Keyword Spotting

Incorporating prosodic information with spectral information at the feature level is challenging. In this paper, a method for feature level fusion of spectral and prosodic information is proposed. A pitch contour is first extracted from the frame blocked segments of the speech signal. These speech segments obtained herein are labeled as high pitch and low pitch segments. Both spectral and prosodic features are extracted from each segment class. An integrated feature set is obtained by concatenating spectral and prosodic features from each of these classes. In the next stage of fusion, the high and low pitch labeled features are further combined using joint error optimization approach. This optimization approach assumes that the mean of the high pitch segments can be obtained by an affine transformation on the mean of the low pitch segments. The parameters of the affine transformation are obtained using the gradient descent approach. The final integrated feature set is obtained after normalization of both sets of features thus obtained. This integrated feature set is used in a Hidden Markov Modeling (HMM) framework along with a novel sliding syllable protocol for keyword spotting. Keyword spotting experiments are conducted on the Hindi language database developed for this purpose. Experiments on keyword recognition and keyword spotting are conducted to evaluate the performance of the proposed fusion method. Experimental results obtained in terms of WER and receiver operating characteristics indicate a reasonable improvements over the use of a single feature set like the MFCC.