Interpretable Local Frequency Binary Pattern (LFrBP) Based Joint Continual Learning Network for Heterogeneous Face Recognition

Heterogeneous Face Recognition (HFR) is a challenging task due to the significant intra-class variation between the query and gallery images. The reason behind this vast intra-class variation is the varying image capturing sensors and the varying image representation techniques. Visual, Infrared, thermal images are the output of different sensors and viewed sketches, and composite sketches are the output of different image representation techniques. Conventional deep learning models are trying to solve the problem. Still, progress is impeded due to small HFR data samples, task-specific models (one model trained for face sketch-photo matching can't perform well for NIR-VIS face matching), joint learning of two different HFR scenarios are not possible by one single deep network, and models are not interpretable. In this paper, to solve these major problems, we presented a novel interpretable Local Frequency Binary Pattern (LFrBP) based continual learning shallow network for HFR. The model is divided into two parts. A modality-invariant CNN model using the LFrBP feature, fine-tuned with CNN, is presented in the first part. The second part is based on continual learning to jointly learn the two HFR scenarios (face sketch-photo and NIR-VIS face matching) using a single network. Recognition results on different challenging HFR databases depict the superiority of the proposed model over other state-of-the-art deep learning-based methods.