Accurate automatic visible speech synthesis of arbitrary 3D models based on concatenation of diviseme motion capture data

被引:17
作者
Ma, JY [1 ]
Cole, R [1 ]
Pellom, B [1 ]
Ward, W [1 ]
Wise, B [1 ]
机构
[1] Univ Colorado, Ctr Spoken Language Res, Boulder, CO 80309 USA
关键词
visible speech; visual speech synthesis; animated speech; coarticulation modelling; speech animation; face animation;
D O I
10.1002/cav.11
中图分类号
TP31 [计算机软件];
学科分类号
081202 ; 0835 ;
摘要
We present a technique for accurate automatic visible speech synthesis from textual input. When provided with a speech waveform and the text of a spoken sentence, the system produces accurate visible speech synchronized with the audio signal. To develop the system, we collected motion capture data from a speaker's face during production of a set of words containing all diviseme sequences in English. The motion capture points from the speaker's face are retargeted to the vertices of the polygons of a 31) face model. When synthesizing a new utterance, the system locates the required sequence of divisemes, shrinks or expands each diviseme based on the desired phoneme segment durations in the target utterance, then moves the polygons in the regions of the lips and lower face to correspond to the spatial coordinates of the motion capture data. The motion mapping is realized by a key-shape mapping function learned by a set of viseme examples in the source and target faces. A well-posed numerical algorithm estimates the shape blending coefficients. Time warping and motion vector blending at the juncture of two divisemes and the algorithm to search the optimal concatenated visible speech are also developed to provide the final concatenative motion sequence. Copyright (C) 2004 John Wiley Sons, Ltd.
引用
收藏
页码:485 / 500
页数:16
相关论文
共 37 条
[11]  
COLE R, 1999, P ESCA SOCRRATES WOR
[12]   Sample-based synthesis of photo-realistic talking heads [J].
Cosatto, E ;
Graf, HP .
COMPUTER ANIMATION 98 - PROCEEDINGS, 1998, :103-110
[13]  
Ezzat T, 2002, ACM T GRAPHIC, V21, P388, DOI 10.1145/566570.566594
[14]   MikeTalk: A talking facial display based on morphing visemes [J].
Ezzat, T ;
Poggio, T .
COMPUTER ANIMATION 98 - PROCEEDINGS, 1998, :96-102
[15]   Data smoothing by cubic spline filters [J].
Feng, G .
IEEE TRANSACTIONS ON SIGNAL PROCESSING, 1998, 46 (10) :2790-2796
[16]   Object-oriented software for quadratic programming [J].
Gertz, EM ;
Wright, SJ .
ACM TRANSACTIONS ON MATHEMATICAL SOFTWARE, 2003, 29 (01) :58-81
[17]  
GRAF, 2000, P FAC GEST REC, P189
[18]   The use of spreadsheets in packaging thermal calculations [J].
Guenin, BM .
FOURTEENTH ANNUAL IEEE SEMICONDUCTOR THERMAL MEASUREMENT AND MANAGEMENT SYMPOSIUM, 1998, :55-56
[19]  
Hardcastle WilliamJ., 1999, Coarticulation: Theory, data and techniques
[20]   Parametric Gn blending of curves and surfaces [J].
Hartmann, E .
VISUAL COMPUTER, 2001, 17 (01) :1-13