Towards unsupervised learning of joint facial landmark detection and head pose estimation

Deep learning approaches have advanced state-of-the-art performance drastically in facial landmark detection and head pose estimation. Recent work shows that meaningful landmarks could be discovered from unlabeled image collections. However, they only mine local visual patterns in images as 2D landmarks while ignoring the 3D object structure. Consequently, they can neither directly estimate the object pose from an image nor use it for improved landmark discovery. Therefore, we propose a novel framework that jointly learns both tasks. It includes a multi-task network for joint landmark and pose prediction, a set of learnable 3D canonical landmarks, and an image generation network. They are learned collaboratively on unlabeled face images through an integrated loss of conditional image generation and geometric consistency. We also investigate different strategies to handle potential face deformation. Extensive experiments show that our approach is very effective in both tasks, even comparable to some supervised methods. The code is available at https://github.com/ZhimingZo/unsup-face-analysis