KABI: Class-Incremental Learning via knowledge Amalgamation and Batch Identification

In class-incremental learning setting, classes are typically presented batch by batch over time. Incremental learning often suffers from catastrophic forgetting: the performance on previous classes abruptly degrades when adapting a model to new classes. We find that incremental models trained using knowledge distillation are skilled at discriminating classes within a batch, whereas they have confusion among classes in different batches. We propose a class-incremental learning approach with knowledge amalgamation and batch identification (KABI), which can effectively alleviate catastrophic forgetting. The idea is to first train an expert model for new classes at current state, and then train an amalgamation model by amalgamating knowledge from the expert model and the amalgamation model of previous state to discriminate different classes within a batch, and particularly train a batch identifier to discriminate different batches. We conduct extensive experiments on three datasets: MNIST, CIFAR-100, ILSVRC 2012, and show that KABI outperforms the second-best approach by 1.29%, 14.26% and 21.51% respectively. Surprisingly, classification accuracies of our approach are even sometimes higher than the oracle results which is obtained by training a model using all training samples from all classes.