Layered Neural Rendering for Retiming People in Video

被引：21

作者：

Lu, Erika ^{[1
,2
]}

Cole, Forrester ^{[2
]}

Dekel, Tali ^{[2
]}

Xie, Weidi ^{[1
]}

Zisserman, Andrew ^{[1
]}

Salesin, David ^{[2
]}

Freeman, William T. ^{[2
]}

Rubinstein, Michael ^{[2
]}

机构：

[1] Univ Oxford, Oxford, England

[2] Google Res, San Francisco, CA 94105 USA

来源：

ACM TRANSACTIONS ON GRAPHICS | 2020年 / 39卷 / 06期

基金：

英国工程与自然科学研究理事会;

关键词：

liming; neural rendering; video layer decomposition;

D O I：

10.1145/3414685.3417760

中图分类号：

TP31 [计算机软件];

学科分类号：

081202 ; 0835 ;

摘要：

We present a method for retiming people in an ordinary, natural video manipulating and editing the time in which different motions of individuals in the video occur. We can temporally align different motions, change the speed of certain actions (speeding up/slowing down, or entirely "freezing" people), or "erase" selected people from the video altogether. We achieve these effects computationally via a dedicated learning-based layered video representation, where each frame in the video is decomposed into separate RGBA layers, representing the appearance of different people in the video. A key property of our model is that it not only disentangles the direct motions ofeach person in the input video, but also correlates each person automatically with the scene changes they generate-e.g., shadows, reflections, and motion of loose clothing. The layers can be individually retimed and recombined into a new video, allowing us to achieve realistic, high-quality renderings of retiming effects for real-world videos depicting complex actions and involving multiple individuals, including dancing, trampoline jumping, or group running.

引用

页数：14

共 54 条

[31] Neural Rendering and Reenactment of Human Actor Videos [J].

Liu, Lingjie ;

Xu, Weipeng ;

Zollhoefer, Michael ;

Kim, Hyeongwoo ;

Bernard, Florian ;

Habermann, Marc ;

Wang, Wenping ;

Theobalt, Christian .

ACM TRANSACTIONS ON GRAPHICS, 2019, 38 (05)

[32] SMPL: A Skinned Multi-Person Linear Model [J].

Loper, Matthew ;

Mahmood, Naureen ;

Romero, Javier ;

Pons-Moll, Gerard ;

Black, Michael J. .

ACM TRANSACTIONS ON GRAPHICS, 2015, 34 (06)

[33]

Martin-Brualla Ricardo, 2018, ACM T GRAPHIC, V37, P6

[34]

McCann J., 2006, Proceedings of ACM SIGGRAPH/Eurographics Symposium on Computer Animation, P205

[35] Neural Rerendering in the Wild [J].

Meshry, Moustafa ;

Goldman, Dan B. ;

Khamis, Sameh ;

Hoppe, Hugues ;

Pandey, Rohit ;

Snavely, Noah ;

Martin-Brualla, Ricardo .

2019 IEEE/CVF CONFERENCE ON COMPUTER VISION AND PATTERN RECOGNITION (CVPR 2019), 2019, :3871-6880

[36]

Nandoriya A., 2017, ICCV

[37]

Poleg Y, 2015, PROC CVPR IEEE, P4768, DOI 10.1109/CVPR.2015.7299109

[38]

Porter T., 1984, Computers & Graphics, V18, P253

[39] Nonchronological video synopsis and indexing [J].

Pritch, Yael ;

Rav-Acha, Alex ;

Peleg, Shmuel .

IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, 2008, 30 (11) :1971-1984

[40]

Rublee E, 2011, IEEE I CONF COMP VIS, P2564, DOI 10.1109/ICCV.2011.6126544

← 1 2 3 4 5 6 →