Access Point Clustering in Cell-Free Massive MIMO Using Conventional and Federated Multi-Agent Reinforcement Learning

Cell-free massive multiple-input multiple-output (MIMO) systems consist of geographically-distributed multi-antenna access points (APs) that form a virtual massive MIMO array. To make the network arbitrarily scalable in size, each user should be served by the best possible personalized user-centric cluster of nearby APs. Unfortunately, determining that cluster is a combinatorially-complex problem made even harder when the users are in motion. Therefore, in this work, we develop a multi-agent reinforcement learning (MARL) algorithm for AP selection and clustering. Each AP is an agent in the MARL algorithm and it is trained to near-optimally select for itself which users to serve. Conventional MARL algorithms require a centralized reward system to train the agents, and the agents' neural network weights tend to strongly depend on their locations during training. To counteract these problems, we also consider a federated MARL framework. Simulation results demonstrate both our conventional and federated MARL algorithms outperform existing published AP selection algorithms, and also provide performance comparable to the case of all APs serving all users. The results also show the conventional algorithm has somewhat superior performance in the environment it was trained in, but the federated algorithm transfers its learning to changed environments much better, with very little performance loss.