FedCS: Communication-Efficient Federated Learning with Compressive Sensing

In Federated Learning (FL), two-way model exchanges are required between the server and the workers every training round. Due to the large size of machine learning models, communications between them lead to high training delay and economic cost. At present, communication-efficient FL methods, for examples, top-k sparsification and quantization, taking advantages of the sparseness of model gradients and the fact that gradient-based model updating can tolerance small deviations, effectively reduce the communication cost of single training round. However, these gradient-based communication-efficient schemes cannot be applied to downlink communication. In addition, they cannot be used in conjunction with those communication-frequency-suppressed methods, e.g., FedAvg, which hinders them from further improving training efficiency. In this paper, we propose FedCS, a compressive sensing based FL method, which can effectively compress and accurately reconstruct non-sparse model (both local and global) parameters (weights), and can reduce the overall communication cost up to 10x as compared to FedAvg without decreasing test accuracy. We introduce 1) a dictionary learning scheme with a quasi-validation set, which helps to project non-sparse parameters onto a sparse domain; 2) a joint reconstruction scheme, by using which the server recovers global model parameters by executing the reconstruction algorithm only once a round, regardless of the number of compressed local models; 3) a compression ratio adjustment strategy, which balances the trade-off between total communication cost and model accuracy. We perform FedCS on three image classification tasks, and compare it with FedAvg, FedPAQ and T-FedAvg (two improvements of FedAvg). Experimental results demonstrate that FedCS outperforms comparison methods in all tasks, and always maintains a comparable test accuracy to FedAvg, even using a small quasi-validation set and on Non-IId data.