DECENTRALIZED FEDERATED LEARNING VIA MUTUAL KNOWLEDGE DISTILLATION

被引:2
作者
Huang, Yue [1 ]
Kong, Lanju [1 ,2 ]
Li, Qingzhong [1 ,2 ]
Zhang, Baochen [1 ]
机构
[1] Shandong Univ, Sch Software, Jinan, Shandong, Peoples R China
[2] Dareway Software Co, Jinan, Peoples R China
来源
2023 IEEE INTERNATIONAL CONFERENCE ON MULTIMEDIA AND EXPO, ICME | 2023年
关键词
Federated learning; mutual knowledge distillation; decentralized;
D O I
10.1109/ICME55011.2023.00066
中图分类号
TP18 [人工智能理论];
学科分类号
081104 ; 0812 ; 0835 ; 1405 ;
摘要
Federated learning (FL), an emerging decentralized machine learning paradigm, supports the implementation of common modeling without compromising data privacy. In practical applications, FL participants heterogeneity poses a significant challenge for FL. Firstly, clients sometimes need to design custom models for various scenarios and tasks. Secondly, client drift leads to slow convergence of the global model. Recently, knowledge distillation has emerged to address this problem by using knowledge from heterogeneous clients to improve the model's performance. However, this approach requires the construction of a proxy dataset. And FL is usually performed with the assistance of a center, which can easily lead to trust issues and communication bottlenecks. To address these issues, this paper proposes a knowledge distillation-based FL scheme called FedDCM. Specifically, in this work, each participant maintains two models, a private model and a public model. The two models are mutual distillations, so there is no need to build proxy datasets to train teacher models. The approach allows for model heterogeneity, and each participant can have a private model of any architecture. The direct and efficient exchange of information between participants through the public model is more conducive to improving the participants' private models than a centralized server. Experimental results demonstrate the effectiveness of FedDCM, which offers better performance compared to s the most advanced methods.
引用
收藏
页码:342 / 347
页数:6
相关论文
共 50 条
[41]   Communication-efficient Federated Learning for UAV Networks with Knowledge Distillation and Transfer Learning [J].
Li, Yalong ;
Wu, Celimuge ;
Du, Zhaoyang ;
Zhong, Lei ;
Yoshinaga, Tsutomu .
IEEE CONFERENCE ON GLOBAL COMMUNICATIONS, GLOBECOM, 2023, :5739-5744
[42]   FedBKD: Heterogenous Federated Learning via Bidirectional Knowledge Distillation for Modulation Classification in IoT-Edge System [J].
Qi, Peihan ;
Zhou, Xiaoyu ;
Ding, Yuanlei ;
Zhang, Zhengyu ;
Zheng, Shilian ;
Li, Zan .
IEEE JOURNAL OF SELECTED TOPICS IN SIGNAL PROCESSING, 2023, 17 (01) :189-204
[43]   Provable Privacy Advantages of Decentralized Federated Learning via Distributed Optimization [J].
Yu, Wenrui ;
Li, Qiongxiu ;
Lopuhaa-Zwakenberg, Milan ;
Christensen, Mads Graesboll ;
Heusdens, Richard .
IEEE TRANSACTIONS ON INFORMATION FORENSICS AND SECURITY, 2025, 20 :822-838
[44]   Straggler Remission for Federated Learning via Decentralized Redundant Cayley Tree [J].
Tao, Yangyang ;
Zhou, Junxiu .
2020 IEEE LATIN-AMERICAN CONFERENCE ON COMMUNICATIONS (LATINCOM 2020), 2020,
[45]   Topology Control for Wireless Decentralized Federated Learning via Network Coding [J].
Chen, Jiajun ;
Sung, Chi Wan .
ICC 2023-IEEE INTERNATIONAL CONFERENCE ON COMMUNICATIONS, 2023, :3108-3113
[46]   Robust Semi-Decentralized Federated Learning via Collaborative Relaying [J].
Yemini, Michal ;
Saha, Rajarshi ;
Ozfatura, Emre ;
Gunduz, Deniz ;
Goldsmith, Andrea J. .
IEEE TRANSACTIONS ON WIRELESS COMMUNICATIONS, 2024, 23 (07) :7520-7536
[47]   Decentralized and Model-Free Federated Learning: Consensus-Based Distillation in Function Space [J].
Taya, Akihito ;
Nishio, Takayuki ;
Morikura, Masahiro ;
Yamamoto, Koji .
IEEE TRANSACTIONS ON SIGNAL AND INFORMATION PROCESSING OVER NETWORKS, 2022, 8 :799-814
[48]   Federated Continual Learning via Knowledge Fusion: A Survey [J].
Yang, Xin ;
Yu, Hao ;
Gao, Xin ;
Wang, Hao ;
Zhang, Junbo ;
Li, Tianrui .
IEEE TRANSACTIONS ON KNOWLEDGE AND DATA ENGINEERING, 2024, 36 (08) :3832-3850
[49]   Digital Twin-Assisted Knowledge Distillation Framework for Heterogeneous Federated Learning [J].
Xiucheng Wang ;
Nan Cheng ;
Longfei Ma ;
Ruijin Sun ;
Rong Chai ;
Ning Lu .
ChinaCommunications, 2023, 20 (02) :61-78
[50]   Adaptive Block-Wise Regularization and Knowledge Distillation for Enhancing Federated Learning [J].
Liu, Jianchun ;
Zeng, Qingmin ;
Xu, Hongli ;
Xu, Yang ;
Wang, Zhiyuan ;
Huang, He .
IEEE-ACM TRANSACTIONS ON NETWORKING, 2024, 32 (01) :791-805