Asynchronous Byzantine-Robust Stochastic Aggregation with Variance Reduction for Distributed Learning

We consider Byzantine-robust distributed learning with asynchronous participation of clients at a certain probability, where Byzantine clients can send malicious messages to the server. Instead of relying on traditional robust aggregation rules, such as Krum and Median, that can only tolerate a limited proportion of Byzantine clients, we propose an asynchronous Byzantine-robust stochastic aggregation method that employs regularization-based techniques to mitigate Byzantine attacks, and adopts variance-reduced techniques to eliminate the effect of stochastic noise of gradient sampling. Leveraging a properly designed Lyapunov function, we show that the proposed algorithm converges linearly to an error ball that is independent of stochastic gradient variance. Extensive experiments are conducted to show its efficacy in dealing with Byzantine attacks compared to the existing counterparts.