Physics-based Motion Retargeting from Sparse Inputs

被引:8
作者
Reda, Daniele [1 ]
Won, Jungdam [2 ]
Ye, Yuting [3 ]
De Panne, Michiel Van [1 ]
Winkler, Alexander [3 ]
机构
[1] Univ British Columbia, Vancouver, BC, Canada
[2] Seoul Natl Univ, Seoul, South Korea
[3] Meta, Real Labs Res, Burlingame, CA USA
关键词
retargeting; reinforcement learning; physics-based simulation; computer animation;
D O I
10.1145/3606928
中图分类号
TP31 [计算机软件];
学科分类号
081202 ; 0835 ;
摘要
Avatars are important to create interactive and immersive experiences in virtual worlds. One challenge in animating these characters to mimic a user's motion is that commercial AR/VR products consist only of a headset and controllers, providing very limited sensor data of the user's pose. Another challenge is that an avatar might have a different skeleton structure than a human and the mapping between them is unclear. In this work we address both of these challenges. We introduce a method to retarget motions in real-time from sparse human sensor data to characters of various morphologies. Our method uses reinforcement learning to train a policy to control characters in a physics simulator. We only require human motion capture data for training, without relying on artist-generated animations for each avatar. This allows us to use large motion capture datasets to train general policies that can track unseen users from real and sparse data in real-time. We demonstrate the feasibility of our approach on three characters with different skeleton structure: a dinosaur, a mouse-like creature and a human. We show that the avatar poses often match the user surprisingly well, despite having no sensor information of the lower body available. We discuss and ablate the important components in our framework, specifically the kinematic retargeting step, the imitation, contact and action reward as well as our asymmetric actor-critic observations. We further explore the robustness of our method in a variety of settings including unbalancing, dancing and sports motions.
引用
收藏
页数:19
相关论文
共 50 条
[41]   Physics-Based Simulation of AUV Forced Diving by Self-Propulsion [J].
Wu L. ;
Feng X. ;
Ye Z. ;
Li Y. .
Feng, Xisheng (fengxsh@mail.cae.ac.cn), 1600, Shanghai Jiaotong University (55) :290-296
[42]   A survey of CAD model simplification techniques for physics-based simulation applications [J].
Thakur, Atul ;
Banerjee, Ashis Gopal ;
Gupta, Satyandra K. .
COMPUTER-AIDED DESIGN, 2009, 41 (02) :65-80
[43]   Advanced virtual prototyping of robotic cells using physics-based simulation [J].
Raffaeli, Roberto ;
Neri, Federico ;
Peruzzini, Margherita ;
Berselli, Giovanni ;
Pellicciari, Marcello .
INTERNATIONAL JOURNAL OF INTERACTIVE DESIGN AND MANUFACTURING - IJIDEM, 2023, 18 (2) :981-996
[44]   Model Predictive Control with a Visuomotor System for Physics-based Character Animation [J].
Eom, Haegwang ;
Han, Daseong ;
Shin, Joseph S. ;
Noh, Junyong .
ACM TRANSACTIONS ON GRAPHICS, 2020, 39 (01)
[45]   Automated physics-based modeling of construction equipment through data fusion [J].
Xu, Liqun ;
Veeramani, Dharmaraj ;
Zhu, Zhenhua .
AUTOMATION IN CONSTRUCTION, 2024, 168
[46]   DReCon: Data-Driven Responsive Control of Physics-Based Characters [J].
Bergamin, Kevin ;
Clavet, Simon ;
Holden, Daniel ;
Forbes, James Richard .
ACM TRANSACTIONS ON GRAPHICS, 2019, 38 (06)
[47]   MAAIP: Multi-Agent Adversarial Interaction Priors for imitation from fighting demonstrations for physics-based characters [J].
Younes, Mohamed ;
Kijak, Ewa ;
Kulpa, Richard ;
Malinowski, Simon ;
Multon, Franck .
PROCEEDINGS OF THE ACM ON COMPUTER GRAPHICS AND INTERACTIVE TECHNIQUES, 2023, 6 (03)
[48]   A GAN-Like Approach for Physics-Based Imitation Learning and Interactive Control [J].
Xu, Pei ;
Karamouzas, Ioannis .
PROCEEDINGS OF THE ACM ON COMPUTER GRAPHICS AND INTERACTIVE TECHNIQUES, 2021, 4 (03)
[49]   A Levels-of-Precision Approach for Simulating Multiple Physics-based Soft Tissues [J].
Fernandes e Silva, Daniele ;
Berndt, Iago ;
Torchelsen, Rafael P. ;
Maciel, Anderson .
2016 29TH SIBGRAPI CONFERENCE ON GRAPHICS, PATTERNS AND IMAGES (SIBGRAPI), 2016, :9-16
[50]   3D physics-based ground motion simulation and topography effects of the 05 September 2022 MW6.6 Luding earthquake, China [J].
Ba, Zhenning ;
Zhao, Jingxuan ;
Zhu, Zhihui ;
Zhou, Gaoyang .
SOIL DYNAMICS AND EARTHQUAKE ENGINEERING, 2023, 172