Learning continuation: Integrating past knowledge for contrastive distillation

Current knowledge distillation methods typically transfer knowledge on data from the same batch. Nevertheless, these methodologies neglect the significance of leveraging the knowledge accumulated by the teacher model from past batches. In response to this challenge, this paper integrates the historical knowledge of the teacher network, aiming to enable the student network to acquire additional knowledge beyond the current batch. The proposed approach named PKCD includes Cross-Batch Contrastive Distillation(CBCD) and Intra-Class Distillation(ICD). Cross-Batch Contrastive Distillation(CBCD) leverages the logits of the teacher network in the previous batches to construct more positive and negative samples across batches based on contrastive learning. This enables the student network to acquire shared information among samples of the same category and distinguish differences among samples of distinct categories. At the same time, the intra-class relationship is considered during the Cross-batch Contrastive distillation(CBCD), and the Intra-Class Distillation(ICD) is carried out to prevent the samples of the same category from being too close to each other, so as to improve the effect of distillation. By integrating historical knowledge, we can leverage the historical knowledge of the teacher network and further improve the effect of knowledge distillation. Through feature transfer experiments,we validate the efficacy of this approach in facilitating knowledge transfer and enhancing the generalization capabilities of student models, and provides a new idea for further research on knowledge distillation based on logits. In contrast to existing logit-based approaches, PKCD demonstrates comparable or superior performance on CIFAR-100 and ImageNet datasets.