Communication-Efficient Federated Deep Learning With Layerwise Asynchronous Model Update and Temporally Weighted Aggregation

被引:395
作者
Chen, Yang [1 ,2 ]
Sun, Xiaoyan [1 ]
Jin, Yaochu [3 ]
机构
[1] China Univ Min & Technol, Sch Informat & Control Engn, Xuzhou 221116, Jiangsu, Peoples R China
[2] Nanyang Technol Univ, Sch Comp Sci & Engn, Singapore 639798, Singapore
[3] Univ Surrey, Dept Comp Sci, Guildford GU2 7XH, Surrey, England
基金
中国国家自然科学基金;
关键词
Servers; Data models; Training; Data privacy; Distributed databases; Deep learning; Sun; Aggregation; asynchronous learning; deep neural network (DNN); federated learning; temporally weighted aggregation;
D O I
10.1109/TNNLS.2019.2953131
中图分类号
TP18 [人工智能理论];
学科分类号
081104 ; 0812 ; 0835 ; 1405 ;
摘要
Federated learning obtains a central model on the server by aggregating models trained locally on clients. As a result, federated learning does not require clients to upload their data to the server, thereby preserving the data privacy of the clients. One challenge in federated learning is to reduce the client-server communication since the end devices typically have very limited communication bandwidth. This article presents an enhanced federated learning technique by proposing an asynchronous learning strategy on the clients and a temporally weighted aggregation of the local models on the server. In the asynchronous learning strategy, different layers of the deep neural networks (DNNs) are categorized into shallow and deep layers, and the parameters of the deep layers are updated less frequently than those of the shallow layers. Furthermore, a temporally weighted aggregation strategy is introduced on the server to make use of the previously trained local models, thereby enhancing the accuracy and convergence of the central model. The proposed algorithm is empirically on two data sets with different DNNs. Our results demonstrate that the proposed asynchronous federated deep learning outperforms the baseline algorithm both in terms of communication cost and model accuracy.
引用
收藏
页码:4229 / 4238
页数:10
相关论文
共 24 条
[1]  
[Anonymous], 2012, Proc. the 26th International Conference on Neural Information Processing Systems
[2]  
[Anonymous], 2014, arXiv
[3]  
[Anonymous], IEEE T NEURAL NETW L
[4]   Practical Secure Aggregation for Privacy-Preserving Machine Learning [J].
Bonawitz, Keith ;
Ivanov, Vladimir ;
Kreuter, Ben ;
Marcedone, Antonio ;
McMahan, H. Brendan ;
Patel, Sarvar ;
Ramage, Daniel ;
Segal, Aaron ;
Seth, Karn .
CCS'17: PROCEEDINGS OF THE 2017 ACM SIGSAC CONFERENCE ON COMPUTER AND COMMUNICATIONS SECURITY, 2017, :1175-1191
[5]   Distributed optimization and statistical learning via the alternating direction method of multipliers [J].
Boyd S. ;
Parikh N. ;
Chu E. ;
Peleato B. ;
Eckstein J. .
Foundations and Trends in Machine Learning, 2010, 3 (01) :1-122
[6]  
Chilimbi Trishul, 2014, Proceedings of the 11th USENIX Symposium on Operating Systems Design and Implementation (OSDI '14). OSDI '14, P571
[7]  
Gers FA, 1999, IEE CONF PUBL, P850, DOI [10.1162/089976600300015015, 10.1049/cp:19991218]
[8]  
Geyer R. C., 2017, DIFFERENTIALLY PRIVA
[9]   LSTM: A Search Space Odyssey [J].
Greff, Klaus ;
Srivastava, Rupesh K. ;
Koutnik, Jan ;
Steunebrink, Bas R. ;
Schmidhuber, Juergen .
IEEE TRANSACTIONS ON NEURAL NETWORKS AND LEARNING SYSTEMS, 2017, 28 (10) :2222-2232
[10]   Parallel asynchronous particle swarm optimization [J].
Koh, Byung-Il ;
George, Alan D. ;
Haftka, Raphael T. ;
Fregly, Benjamin J. .
INTERNATIONAL JOURNAL FOR NUMERICAL METHODS IN ENGINEERING, 2006, 67 (04) :578-595