Policy-Gradient-Based Reinforcement Learning for Maximizing Operator's Profit in Open-RAN

Open radio access network (O-RAN) is a novel architecture that enables the disaggregation and the virtualization of network components. This would provide new ways to mix and match network components by opening the interfaces linking them. O-RAN drives down network deployment costs and allows new players to enter the RAN market. It permits network operators to maximize resource utilization and deliver new network edge services at a lower price, resulting in higher profits for operators. In this context, we consider a computing resource allocation problem for maximizing the operator's profit. Given that an operator receives payments from subscribers and incurs costs from the infrastructure provider, the objective is to maximize the difference between these payments and costs. We model the problem using mixed integer linear programming (MILP), and we prove it to be NP-Hard. The aim is to allocate computing resources to users considering different processing priority levels. Then, we propose to solve the problem using policy-gradient-based reinforcement learning (RL). We consider different variants for the RL model. This includes using different neural network architectures such as Feed Forward and Bi-directional Long-Short Term Memory. Additionally, we consider REINFORCE and Actor-Critic algorithms, and we evaluate two strategies, Single Action per State and multiple actions per state (MAS). Our simulation results demonstrate that MAS combined with the Feed-Forward neural network and Actor-Critic algorithm outperforms all other RL variants regarding the operator's profit optimization. MAS combined with Actor-Critic reaches up to 93.06% of the optimal MILP-based solution yielded by solving the MILP formulation. Additionally, it has significantly lower complexity compared to solving the MILP problem to optimality, as our numerical experimentation shows that the execution time is reduced by up to 99.53%.