Distributed Online Private Learning of Convex Nondecomposable Objectives

In the machine learning domain, datasets share several new features, including distributed storage, high velocity, and privacy concerns, which naturally requires the development of distributed privacy-preserving algorithms. Moreover, nodes (e.g., learners, sensors, GPUs, mobiles, etc.) in the real networks usually process tasks in real time, which inevitably requires nodes to have online learning capabilities. Therefore, we deal with a general distributed constrained online learning problem with privacy over time-varying networks, where a class of nondecomposable objectives are considered. Under this setting, each node only controls a part of the global decision, and the goal of all nodes is to collaboratively minimize the global cost over a time horizon $T$ while guarantees the security of the transmitted information. For such problems, we first design a novel generic algorithm framework, named as DPSDA, of differentially private distributed online learning using the Laplace mechanism and the stochastic variants of dual averaging method. Note that in the dual updates, all nodes of DPSDA employ the noise-corrupted gradients for more generality. Then, we propose two algorithms, named as DPSDA-C and DPSDA-PS, under this framework. In DPSDA-C, the nodes implement a circulation-based communication in the primal updates so as to alleviate the disagreements over time-varying undirected networks. In addition, for the extension to time-varying directed ones, the nodes implement the broadcast-based push-sum dynamics in DPSDA-PS, which can achieve average consensus over arbitrary directed networks. Theoretical results show that both algorithms attain an expected regret upper bound in O(root T) when the objective function is convex, which matches the best utility achievable by cutting-edge algorithms. Finally, numerous numerical experiments on both synthetic and real-world datasets verify the effectiveness of our algorithms.