3D human pose and shape estimation with dense correspondence from a single depth image

We propose a novel approach to estimate the 3D pose and shape of human bodies with dense correspondence from a single depth image. In contrast to most current 3D body model recovery methods from depth images that employ motion information of depth sequences to compute point correspondences, we reconstruct 3D human body models from a single depth image by combining the correspondence learning and the parametric model fitting. Specifically, a novel multi-view coarse-to-fine correspondence network is proposed by projecting a 3D template model into multi-view depth images. The proposed correspondence network can predict 2D flows of the input depth relative to each projected depth in a coarse-to-fine manner. The predicted multi-view flows are then aggregated to establish accurate dense point correspondences between the 3D template and the input depth with the known 3D-to-2D projection. Based on the learnt correspondences, the 3D human pose and shape represented by a parametric 3D body model are recovered through a model fitting method that incorporates an adversarial prior. We conduct extensive experiments on SURREAL, Human3.6M, DFAUST, and real depth data of human bodies. The experimental results demonstrate that the proposed method outperforms the state-of-the-art methods in terms of reconstruction accuracy.