Minimum Gaussian Noise Variance of Federated Learning in the Presence of Mutual Information Based Differential Privacy

Federated learning (FL), which not only protects data security and privacy, but also training models on distributed devices, receives much attention in the literature. Traditionally, stochastic gradient descent (SGD) is used in FL since its excellent empirical performance, but user's private information can still be leaked by analyzing weight updates during FL iterations. Differential privacy (DP) is an effective way to solve this privacy leakage problem, which adds noise to the user's data gradient and this artificial noise helps to prevent information leakage. However, note that the SGD based FL with DP is not yet investigated with a comprehensive theoretical analysis considering privacy and data utility jointly, especially from the information-theoretic aspect. In this paper, we investigate the FL in the presence of mutual information based DP (MI-DP). Specifically, first, Gaussian DP mechanism is applied to either clients or central server of the FL model, and privacy and utility of the FL model are characterized by conditional mutual information and distortion, respectively. For a given privacy budget, we establish lower bounds on the variance of the Gaussian noise added to clients or central server of the FL model, and show that the utility of the global model remains the same for both cases. Next, we study the privacy-utility trade-off problem by considering a more general case, where both the model parameter and the privacy requirement of the clients are flexible. A privacy-preserving scheme is proposed, which maximizes the utility of the global model while different privacy requirements of all clients are preserved. Finally, the results of this paper are further explained by experimental results.