PrivateDL: Privacy-preserving collaborative deep learning against leakage from gradient sharing

Large-scale data training is vital to the generalization performance of deep learning (DL) models. However, collecting data directly is associated with increased risk of privacy disclosure, particularly in special fields such as healthcare, finance, and genomics. To protect training data privacy, collaborative deep learning (CDL) has been proposed to enable joint training from multiple data owners while providing reliable privacy guarantee. However, recent studies have shown that CDL is vulnerable to several attacks that could reveal sensitive information about the original training data. One of the most powerful attacks benefits from the leakage from gradient sharing during collaborative training process. In this study, we present a new CDL framework, PrivateDL, to effectively protect private training data against leakage from gradient sharing. Unlike conventional training process that trains on private data directly, PrivateDL allows effective transfer of relational knowledge from sensitive data to public data in a privacy-preserving way, and enables participants to jointly learn local models based on the public data with noise-preserving labels. This way, PrivateDL establishes a privacy gap between the local models and the private datasets, thereby ensuring privacy against the attacks launched to the local models through gradient sharing. Moreover, we propose a new algorithm called Distributed Aggregation Stochastic Gradient Descent, which is designed to improve the efficiency and accuracy of CDL, especially in the asynchronous training mode. Experimental results demonstrate that PrivateDL preserves data privacy with reasonable performance overhead.