Dense Convolutional Recurrent Neural Network for Generalized Speech Animation

This paper presents a novel automated speech animation approach named Dense Convolutional Recurrent Neural Network (DenseCRNN). The approach learns a non-linear mapping from acoustic speech to multiple articulator movements in a unified framework to which feature extraction, context encoding and multi-parameter decoding are integrated. We propose DenseCRNN based on three insights: (1) One can use a convolutional neural network incorporated with dense connectivity to extract speaker-independent features from arbitrary spoken audio effectively. (2) A bidirectional long short-term memory neural network is able to model the context information with respect to the phoneme coarticulation. (3) Multi-domain learning can be implemented to achieve better performance on account of the implicit correlation and explicit difference among outputs, where each domain is responsible for a single visual parameter. Experiments on MNGU0 dataset demonstrate our approach achieves significant improvements over state-of-the-art methods. Moreover, the proposed approach generalizes over different gender or accent, and has the capability of deploying on various character models.