Joint learning for face alignment and face transfer with depth image

Face alignment and cross-modal face transfer are two important tasks for automatic face analysis in computer vision. Over the years, they have been extensively studied. Recently, deep neural networks have attracted much research attention for both face alignment and face transfer. With the prevalence of the consumer depth sensor, depth-based face alignment and cross-modal (image and depth) are increasingly important. Different from existing RGB- image based tasks, the main challenge of depth-based tasks is the lack of annotated data. To address the challenge, we observe that these two tasks are closely related and their learning processes may benefit each other. This paper develops a joint multi-task learning algorithm for both depth-based face alignment and face transfer using the deep convolutional neural network (CNN). The proposed approach allows the CNN model to simultaneously share visual knowledge and information between two tasks. We use a dataset of 10,000 face depth images for validation. Our experiments show that the proposed approach outperforms state-of-the-art algorithms. The results also show that learning these two related tasks simultaneously improves the performance of each individual task.