A two-phase half-async method for heterogeneity-aware federated learning

Federated learning (FL) is a distributed machine learning paradigm that allows training models on decentralized data over large-scale edge/mobile devices without collecting raw data. However, existing methods are still far from efficient and stable under extreme statistical and environmental heterogeneity. In this work, we propose FedHA (Federated Heterogeineity Awareness), a novel half-async algorithm which simultaneously incorporates the merits of asynchronous and synchronous methods. It separates the training into two phases by estimating the consistency of optimization directions of collected local models. It applies different strategies to facilitate fast and stable training, namely model selection, adaptive local epoch, and heterogeneity weighted aggregation in these phases. We provide theoretical convergence and communication guarantees on both convex and non-convex problems without introducing extra assumptions. In the first phase (the consistent phase), the convergence rate of FedHA is O (1/e(T)), which is faster than existing methods while reducing communication. In the second phase (inconsistent phase), FedHA retains the best-known results in convergence (O(1/T)) and communication (O(1/c)). We validate our proposed algorithm on different tasks with both IID (Independently and Identically Distributed) and non-IID data, and results show that our algorithm is efficient, stable, and flexible under the twofold heterogeneity using the proposed strategies. (c) 2021 Elsevier B.V. All rights reserved.