DP-Norm: Differential Privacy Primal-Dual Algorithm for Decentralized Federated Learning

被引：6

作者：

Fukami, Takumi ^{[1
]}

Murata, Tomoya ^{[2
]}

Niwa, Kenta ^{[3
]}

Tyou, Iifan ^{[1
]}

机构：

[1] NTT Corp, NTT Social Informat Labs, Yokosuka 2390847, Japan

[2] NTT DATA Math Syst Inc, Tokyo 1600016, Japan

[3] NTT Corp, NTT Commun Sci Labs, Yokosuka 2390847, Japan

来源：

IEEE TRANSACTIONS ON INFORMATION FORENSICS AND SECURITY | 2024年 / 19卷

关键词：

Diffusion processes; Message passing; Resource management; Peer-to-peer computing; Data models; Privacy; Noise reduction; Federated learning; differential privacy; data heterogeneity; primal-dual optimization; DISTRIBUTED OPTIMIZATION;

D O I：

10.1109/TIFS.2024.3390993

中图分类号：

TP301 [理论、方法];

学科分类号：

081202 ;

摘要：

A novel algorithm is proposed for highly privacy-preserving decentralized federated learning (FL). Several studies have reported security risks in decentralized FL by reconstructing data even from model update differences. A common approach to overcome this issue is to use the diffusion process following differential privacy (DP), i.e., message passing between nodes is hidden by noise. However, this often makes the learning process unstable, leading to degraded results compared to without using DP diffusion process. In this paper, we propose a primal-dual DP algorithm with denoising normalization (DP-Norm) for less sensitivity to noise/interference, such as DP diffusion and heterogeneous data allocation. For DP-Norm, privacy analysis to determine minimal noise level and convergence analysis are conducted. Through image classification benchmark tests, we confirmed that DP-Norm performed close to the single-node reference score, even when statistically heterogeneous data was allocated on six nodes.

引用

页码：5783 / 5797

页数：15

共 28 条

[1] Deep Learning with Differential Privacy [J].

Abadi, Martin ;

Chu, Andy ;

Goodfellow, Ian ;

McMahan, H. Brendan ;

Mironov, Ilya ;

Talwar, Kunal ;

Zhang, Li .

CCS'16: PROCEEDINGS OF THE 2016 ACM SIGSAC CONFERENCE ON COMPUTER AND COMMUNICATIONS SECURITY, 2016, :308-318

[2]

Bauschke HH, 2011, CMS BOOKS MATH, P1, DOI 10.1007/978-1-4419-9467-7

[3]

Blot M, 2016, Arxiv, DOI arXiv:1611.09726

[4] MULTINOMIAL LOGISTIC-REGRESSION ALGORITHM [J].

BOHNING, D .

ANNALS OF THE INSTITUTE OF STATISTICAL MATHEMATICS, 1992, 44 (01) :197-200

[5] Diffusion Adaptation Strategies for Distributed Optimization and Learning Over Networks [J].

Chen, Jianshu ;

Sayed, Ali H. .

IEEE TRANSACTIONS ON SIGNAL PROCESSING, 2012, 60 (08) :4289-4305

[6]

Dwork C, 2006, LECT NOTES COMPUT SC, V4052, P1

[7]

Gabay D., 1976, Computers & Mathematics with Applications, V2, P17, DOI 10.1016/0898-1221(76)90003-1

[8] TRAINING SPEECH RECOGNITION MODELS WITH FEDERATED LEARNING: A QUALITY/COST FRAMEWORK [J].

Guliani, Dhruv ;

Beaufays, Francoise ;

Motta, Giovanni .

2021 IEEE INTERNATIONAL CONFERENCE ON ACOUSTICS, SPEECH AND SIGNAL PROCESSING (ICASSP 2021), 2021, :3080-3084

[9] Deep Residual Learning for Image Recognition [J].

He, Kaiming ;

Zhang, Xiangyu ;

Ren, Shaoqing ;

Sun, Jian .

2016 IEEE CONFERENCE ON COMPUTER VISION AND PATTERN RECOGNITION (CVPR), 2016, :770-778

[10]

Huang ZH, 2020, Arxiv, DOI arXiv:2005.07890

← 1 2 3 →