Augmented Knowledge Distillation via Contrastive Learning

Deploying computer animation and social agent models on devices with limited computing resources presents a significant challenge. Knowledge distillation (KD) emerges as an effective model compression technique, harnessing the extensive knowledge of a large teacher model to facilitate the training of a smaller student model. However, existing KD methodologies predominantly concentrate on transferring task-specific knowledge from supervised tasks, such as logit and feature, over-looking the valuable insights into cross-sample discrepancy inherent in teacher and student models. In response, we propose a novel KD approach, termed augmented knowledge distillation via contrastive learning (CAKD). Initially, in the supervision task, we enhance vanilla KD by integrating logit and feature outputs derived from both the original and the augmented data. Subsequently, in the self-supervision task, we identify pivotal sample pairs and delineate the inter-sample multi-discrepancy relationships using the intrinsic data structure, thus obviating the need for external labels or supervision. This enables knowledge transfer through contrastive learning. The fusion of knowledge from both tasks synergistically enhances student performance. Experimental assessments conducted on two publicly available datasets demonstrate that CAKD surpasses state-of-the-art knowledge distillation methodologies.