Communication-efficient federated learning

被引:208
作者
Chen, Mingzhe [1 ,2 ]
Shlezinger, Nir [3 ]
Poor, H. Vincent [1 ,2 ]
Eldar, Yonina C. [4 ]
Cui, Shuguang [1 ]
机构
[1] Chinese Univ Hong Kong, Shenzhen Res Inst Big Data, Shenzhen 518172, Peoples R China
[2] Princeton Univ, Dept Elect & Comp Engn, Princeton, NJ 08544 USA
[3] Ben Gurion Univ Negev, Sch Elect & Comp Engn, IL-84105 Beer Sheva, Israel
[4] Weizmann Inst Sci, Fac Math & Comp Sci, IL-7610001 Rehovot, Israel
基金
国家重点研发计划; 以色列科学基金会;
关键词
machine learning; federated learning; wireless communications; VECTOR QUANTIZATION;
D O I
10.1073/pnas.2024789118
中图分类号
O [数理科学和化学]; P [天文学、地球科学]; Q [生物科学]; N [自然科学总论];
学科分类号
07 ; 0710 ; 09 ;
摘要
Federated learning (FL) enables edge devices, such as Internet of Things devices (e.g., sensors), servers, and institutions (e.g., hospitals), to collaboratively train a machine learning (ML) model without sharing their private data. FL requires devices to exchange their ML parameters iteratively, and thus the time it requires to jointly learn a reliable model depends not only on the number of training steps but also on the ML parameter transmission time per step. In practice, FL parameter transmissions are often carried out by a multitude of participating devices over resource-limited communication networks, for example, wireless networks with limited bandwidth and power. Therefore, the repeated FL parameter transmission from edge devices induces a notable delay, which can be larger than the ML model training time by orders of magnitude. Hence, communication delay constitutes a major bottleneck in FL. Here, a communication-efficient FL framework is proposed to jointly improve the FL convergence time and the training loss. In this framework, a probabilistic device selection scheme is designed such that the devices that can significantly improve the convergence speed and training loss have higher probabilities of being selected for ML model transmission. To further reduce the FL convergence time, a quantization method is proposed to reduce the volume of the model parameters exchanged among devices, and an efficient wireless resource allocation scheme is developed. Simulation results show that the proposed FL framework can improve the identification accuracy and convergence time by up to 3.6% and 87% compared to standard FL.
引用
收藏
页数:8
相关论文
共 47 条
[1]  
Alistarh D., 2018, P ADV NEURAL INFORM
[2]  
Alistarh D, 2017, ADV NEUR IN, V30
[3]  
Amiri MM, 2019, IEEE INT SYMP INFO, P1432, DOI [10.1109/ISIT.2019.8849334, 10.1109/tsp.2020.2981904]
[4]  
[Anonymous], Mnist database
[5]  
Bonawitz K, 2019, P 2019 11 INT C SYST
[6]  
Boyd S., 2009, Convex Optimization, DOI DOI 10.1017/CBO9780511804441
[7]  
Chen M., IEEE T WIRELESS COMM, DOI [10.1109/TWC.2020.3042530 (2021), DOI 10.1109/TWC.2020.3042530(2021)]
[8]   Artificial Neural Networks-Based Machine Learning for Wireless Networks: A Tutorial [J].
Chen, Mingzhe ;
Challita, Ursula ;
Saad, Walid ;
Yin, Changchuan ;
Debbah, Merouane .
IEEE COMMUNICATIONS SURVEYS AND TUTORIALS, 2019, 21 (04) :3039-3071
[9]  
Chen T., 2018, P ADV NEURAL INFORM, P2440
[10]  
Hao K., 2020, MIT Technol Rev