Two-stage personalized federated learning based on sparse pretraining

Aiming at solving the performance degradation of federated learning (FL) under heterogeneous data distribution, personalized FL (PFL) was proposed. It is designed to produce a dedicated model for each client. However, the existing PFL solution only focuses on the performance of personalized model, ignoring the performance of global model, which will affect the willingness of new clients to participate. In order to solve this problem, this paper proposes a new PFL solution, a two-stage PFL based on sparse pretraining, which can not only train a sparse personalized model for each client, but also obtain a sparse global model. The whole training process is divided into sparse pretraining and sparse personalized training, which focus on the performance of global model and personalized model respectively. Also, we propose a mask sparse aggregation technique to maintain the sparsity of the global model in the sparse personalized training stage. Experimental results show that compared with existing algorithms, our proposed algorithm can improve the accuracy of the global model while maintaining advanced personalized model accuracy, and has higher communication efficiency. In order to address the current problem of sparse personalized federated learning where the global model is dense and poorly performing, a new solution is proposed in the authors' work. The two-stage training approach allows for a focus on the training of global and personalized models in the early and late stages, respectively. In addition, sparse mask aggregation techniques have been proposed to guarantee the sparsity of the global model. image