Incremental Model Enhancement via Memory-based Contrastive Learning

Training data of many vision tasks may be sequentially arrived in practice, e.g., the vision tasks in autonomous driving or video surveillance applications. This raises a fundamental challenge that, how to keep improving the performance on a specific task by learning from sequentially available training splits. This paper investigates this task as Incremental Model Enhancement (IME). IME is distinct from the conventional Incremental Learning (IL), where each training split typically corresponds to a set of independent classes, domains, or tasks. In IME, each training split may only cover part of the entire data distribution for the target vision task. Consequently, the IME model should be optimized towards the joint distribution of all available training splits, instead of optimizing towards each newly arrived one like IL methods. To deal with above issues, our method stores feature vectors of previously observed training data in the memory bank, which preserves compressed knowledge of the previous training data. We hence adopt the memorized features and each newly arrived training split for training via Memory-based Contrastive Learning (MCL). A new Contrastive Relation Preserving (CRP) scheme updates the memory bank to prevent obsoleteness of the preserved features and works with MCL simultaneously to boost the model performance. Experiments on several large-scale image classification benchmarks demonstrate the effectiveness of our method. Our method also works well on semantic segmentation, showing strong generalization ability on diverse vision tasks.