You Said That?: Synthesising Talking Faces from Audio

被引：114

作者：

Jamaludin, Amir ^{[1
]}

Chung, Joon Son ^{[1
]}

Zisserman, Andrew ^{[1
]}

机构：

[1] Univ Oxford, Dept Engn Sci, Parks Rd, Oxford OX1 3PJ, England

来源：

INTERNATIONAL JOURNAL OF COMPUTER VISION | 2019年 / 127卷 / 11-12期

基金：

英国工程与自然科学研究理事会;

关键词：

Computer vision; Machine learning; Visual speech synthesis; Video synthesis;

D O I：

10.1007/s11263-019-01150-y

中图分类号：

TP18 [人工智能理论];

学科分类号：

081104 ; 0812 ; 0835 ; 1405 ;

摘要：

We describe a method for generating a video of a talking face. The method takes still images of the target face and an audio speech segment as inputs, and generates a video of the target face lip synched with the audio. The method runs in real time and is applicable to faces and audio not seen at training time. To achieve this we develop an encoder-decoder convolutional neural network (CNN) model that uses a joint embedding of the face and audio to generate synthesised talking face video frames. The model is trained on unlabelled videos using cross-modal self-supervision. We also propose methods to re-dub videos by visually blending the generated face into the source video frame using a multi-stream CNN model.

引用

页码：1767 / 1779

页数：13