Joint weighted knowledge distillation and multi-scale feature distillation for long-tailed recognition

Data in the natural open world tends to follow a long-tailed class distribution, leading deep models trained on such datasets to frequently exhibit inferior performance on the tail classes. Although existing approaches improve a model's performance on tail categories through strategies such as class rebalancing, they often sacrifice the deep features that the model has already learned. In this paper, we propose a new joint distillation framework called JWAFD (Joint weighted knowledge distillation and multi-scale feature distillation) to address the long-tailed recognition problem from the perspective of knowledge distillation. The framework comprises two effective modules. Firstly, the weighted knowledge distillation module, which uses a category prior to adjust the weights of each category. By doing so, the training process becomes more balanced across all categories. Then, the multi-scale feature distillation module, which helps to further optimize the feature representation, thus solving the problem of under-learning of features encountered in previous studies. Compared with previous studies, the proposed framework significantly improves the performance of rare classes while maintaining the performance of head classes recognition. Extensive experiments on three benchmark datasets(CIFAR-100-LT, ImageNet-LT and iNaturalist2018) have demonstrated that the proposed novel distillation framework achieves comparable performance to the state-of-the-art long-tailed recognition methods. Our code is available at: https://github.com/xiaohe6/JWAFD.