A set-to-set nearest neighbor approach for robust and efficient face recognition with image sets

Set-to-set face recognition has drawn much attention thanks to its rich set information. We propose a robust and efficient Set-to-Set Nearest Neighbor Classification (S2S-NNC) approach for face recognition by using the maximum weighted correlation between sets in low-dimensional projection subspaces. A pair of face sets is represented as two sets of Mutual Typical Samples (MTS) based on their maximum weighted correlation, and the S2S distance is equivalent to that between two sets of MTS. For the variation of objects within a set, the faces are partitioned into patches and projected onto a correlation subspace to find the MTS between two sets. Furthermore, we develop a S2S-NNC approach for image set-based face recognition. Compared with existing approaches, the S2S-NNC unifies the image-to-image, image-to-set and set-to-set recognition problems into one model. Experimental results show the S2S-NNC approach significantly outperforms the state-of-art approaches on large video samples and small occluded samples.