An Optimal Transport View of Class-Imbalanced Visual Recognition

Deep models have achieved impressive success in class-imbalanced visual recognition. In the view of optimal transport, the current evaluation protocol for class-imbalanced visual recognition can be interpreted as follows: during training, the neural network learns an optimal transport mapping with an uneven source label distribution, and during evaluation, this mapping is needed to transfer the instances to the uniform target label distribution. The label distribution's inconsistency leads to poor cross-entropy loss performance. In this paper, we first prove that the cross-entropy loss in the classification network is a smooth approximation to the optimal transport, enhancing the interpretability of the classifier. Motivated by this conclusion, we introduce a simple and effective method named Post Optimal Transport (Post OT). Post OT can match the arbitrary target label distribution, which may also be class-imbalanced by post-processing the predictions of a model. In addition, we propose Adaptive Bias Loss (ABL) based on optimal transport theory to shift the label distribution in the class-imbalanced training, which does not depend on the category frequencies in the training set and also can avoid the overfitting of tail classes. Extensive experiments verify that our methods achieve state-of-the-art performance on benchmark datasets such as CIFAR-100-LT, ImageNet-LT, Places365-LT, and iNaturalist 2018.