A robust and anti-forgettiable model for class-incremental learning

In many real-world scenarios, neural network models are not always fixed; they are expected to adapt to a dynamic environment and incrementally learn new knowledge. However, catastrophic forgetting is a challenge for incremental learning in neural networks since updating the model parameters to incorporate new knowledge often results in performance degradation on previous tasks. In this paper, we focus on class-incremental learning (CIL) and attempt to mitigate catastrophic forgetting by improving the robustness of neural networks. Specifically, we modify two aspects of the models. First, we argue that plain batch normalization (BN) has a negative effect on CIL. Hence, we propose a variant BN, called noisy batch normalization (NBN), which introduces Gaussian noise to resist the impact of the change in feature distributions and improves feature representation robustness. Second, to address the task-level overfitting problem in CIL, we introduce a decoder-based regularization (DBR) term, which employs a decoder following the feature encoder to reconstruct the input. DBR can avoid overfitting of the current task and provide a distillation loss to retain the knowledge of previous tasks. We design two CIL scenarios and validate our approaches on the CIFAR-100, MiniImageNet, Fashion MNIST, and Omniglot datasets. The results show that the performance of CIL algorithms based on our approach is better than that of the original algorithms, indicating that our approach can enhance the model robustness and help the networks extract anti-forgettable feature representations.