Collaborative Multiple-Student Single-Teacher for Online Learning

Knowledge distillation is a popular method where a large trained network (teacher) is implemented to train a smaller network (student). To decrease the need for training a much larger network (teacher) for real time application, one student self-knowledge distillation was introduced as a solid technique for compressing neural networks specially for real time applications. However, most of the existing methods consider only one type of knowledge and apply one-student one-teacher learning strategy. This paper presents a collaborative multiple-student single-teacher system (CMSST). The proposed approach is based on real time applications that contain temporal information, which play an important role in understanding. We designed a backbone old student network with target complexity for deployment, during training, once the old student provides high-quality soft labels to guide the hierarchical new student, it also offers the opportunity for the new student to make meaningful improvements based on the students' revised feedback via the shared intermediate representations. Moreover, we introduced soft target label smoothing technique to the CMSST. Experimental results showed that the accuracy can be improved on newly developed teacher knowledge distillation by 1.5% on the UCF-101. Also the accuracy was improved by 1.15% compared to normal huge teacher knowledge distillation on CIFAR100 dataset.