Real-time speech synthesis system driven by visual speech

Visual information from the speaker's mouth region provides speech information and benefits both human speech perception and automatic speech recognition. In this paper, we investigate using visual information only to recognize speech and propose a real-time speech synthesis system driven by visual speech. The system consists of three components: a visual front end, a lipreading recognizer and a speech synthesis module. The visual front end firstly enhances color lip-movement images by using a lip chromatic filter and then extracts relevant speech features based on a novel active contour model (Snake). An adaptive force is introduced at a point of the Snake, so control points of snake do not depend on the position of original contour and the Snake converges on the real target quickly. The recognizer based on continuous hidden Markov models (HMMs) is used to train and recognize a sequence of the combined visual features. The speech synthesis module synthesizes acoustic speech based on the recognition results. The experimental results show that our system was able to achieve 71.7% recognition accuracy for 10 isolated words of a single speaker. The system can be applied to rehabilitation of the speech handicapped.