Communication-efficient federated continual learning for distributed learning system with Non-IID data

Due to the privacy preserving capabilities and the low communication costs, federated learning has emerged as an efficient technique for distributed deep learning/machine learning training. However, given the typical heterogeneous data distributions in the realistic scenario, federated learning faces the challenge of performance degradation on non-independent identically distributed (Non-IID) data across clients. Therefore, we propose federated continual learning to improve the performance on Non-IID data by introducing the knowledge of the other local models. Specifically, we propose a novel federated continual learning method called FedSI, adapting the synaptic intelligence method to the federated learning scenario. Furthermore, in order to reduce the communication overheads, we propose the bidirectional compression and error compensation (BCEC) algorithm to produce the communication-efficient federated continual learning method, called CFedSI. Specifically, the proposed BCEC algorithm compresses both the uplink and the downlink transmission data and utilizes the error compensation locally to ensure training divergence. Experiments show that CFedSI improves the accuracy on Non-IID data by up to 46% with KDDCUP'99 dataset, 23% with CICIDS2017 dataset, 22% with MNIST dataset, and 8% with FashionMNIST dataset, along with the reduced communication overheads.