An Overview of Autonomous Connection Establishment Methods in Peer-to-Peer Deep Learning

The exchange of model parameters between peers is critical in peer-to-peer deep learning. Historically, connections between agents were assigned randomly based on network topology. However, recent methodologies enable agents to autonomously establish their connections, which is especially beneficial for non-IID data settings. Recent studies suggest favorable learning outcomes for autonomous connections compared to fixed topology when evaluated in synthetic non-IID environments. To that end, this study will explore various methodologies to enhance learning outcomes. Through several large-scale experiments with synthetic and realistic non-IID data sets, it evaluates communication efficiency, message exchange frequency, and centralization tendencies. The findings underscore the potential of these methods to enhance local model accuracy, uphold communication efficiency, and resist centralization, rendering them highly suitable for decentralized learning systems. The evaluation results establish PANMGrad and PANMLoss as the most effective solutions in such environments.