Deep Learning for Class-Incremental Learning: A Survey

Recent years have witnessed the progress of deep learning in many fields, e. g., image classification and face recognition. Current deep models are deployed under the static environment, which requires collecting all the training data before the learning process. The deep model is unable to conduct further updating processes when the training process is terminated. However, data in the real world often come in stream format, which contains incoming instances from new classes. For example, in the opinion monitoring system, new topics will emerge as time goes by; in the electronic commerce platform, new types of products will arise day by day; in the robot learning scenario, the robot is required to learn new orders continually. As a result, an ideal model should learn from stream data and enhance its learning ability incrementally. Such a learning process, namely Class-Incremental Learning (CIL), is now drawing more and more attention from the machine learning community. Directly updating the incremental model with new class data will cause the forgetting of old ones and destroy the total performance, which is denoted as catastrophic forgetting in literature. As a result, the class-incremental learning model should incorporate new classes and meanwhile resist catastrophic forgetting over old ones. In this paper, we summarize and classify recent deep-leering-based class-incremental learning algorithms from three aspects, i. e., input, parameter, and algorithm. Typical class-incremental learning methods from the input aspect try to solve incremental learning tasks by regularizing and rehearsing the exemplar set, which can be divided into data replay-based and data restriction-based methods. Similar to the human learning process, data replay-based CIL methods aim to replay former instances when learning new ones, which obtain a trade-off between learning new knowledge and remembering old ones. Data restriction-based methods utilize the former examples as the regularization to restrict the direction of model updating. Class-incremental learning methods from the parameter aspect try to solve incremental learning tasks by regularizing model updating and adjusting network structure, which can be divided into parameter regularization-based and dynamic architecture-based methods. Parameter regularization-based methods weigh the importance of each parameter and restrict important parameters from being changed to overcome forgetting. On the other hand, dynamic architecture-based methods aim to dynamically adjust the network structure to meet the requirements of incoming new classes. Class-incremental learning methods from the algorithm aspect try to solve incremental learning tasks by model mapping and reducing inductive bias, which can be divided into knowledge distillation-based and post-tuning-based methods. Knowledge distillation-based CIL methods utilize the former model as the teacher to restrict the updating process of the current model. Post-tuning-based methods try to reduce the bias in the incremental model to get an unbiased prediction. In this paper, we conduct extensive experimental verification with ten typical algorithms on the benchmark datasets, i. e., CIFAR 100 and ImageNet ILSVRC2012. We analyze the behaviors of incremental models, including the accuracy trend, running time, memory budget, performance decay, and confusion matrix. We also summarize the common rules for class-incremental learning algorithms. Finally, we analyze the challenges and future trends and conclude this paper. © 2023 Science Press. All rights reserved.