3D Face Reconstruction and Multimodal Person Identification from Video Captured Using Smartphone Camera

In this paper, we propose a novel approach for reconstructing 3D face in real-life scenarios. Our main objective is to address the most challenging issue that involves reconstructing depth information from a video that is recorded from frontal camera of the smartphone. Such videos recorded using smartphones impose lot of challenges, such as motion blur, non-frontal perspectives and low resolution. This limits the applicability of state-of-the-art algorithms, which are mostly based on landmark detection. This situation is addressed with the Scale-Invariant Feature Transformation (SIFT) followed by feature matching to generate consistent tracks. These tracks are further processed to generate a 3D point cloud using Point/Cluster based Multi-view stereo (PMVS/CMVS). The usage of PMVS/CMVS will however fail to generate a dense 3D cloud points on the weak surfaces of face, such as cheeks, nose and forehead. This issue is addressed by multi-view reconstruction of these weakly supported surfaces using Visual-Hull. The effectiveness of our method is evaluated on a newly collected dataset, which simulates a realistic identification scenario using a smartphone.