Multi-Layer Energy Management and Strategy Learning for Microgrids: A Proximal Policy Optimization Approach

An efficient energy management system (EMS) enhances microgrid performance in terms of stability, safety, and economy. Traditional centralized or decentralized energy management systems are unable to meet the increasing demands for autonomous decision-making, privacy protection, global optimization, and rapid collaboration simultaneously. This paper proposes a hierarchical multi-layer EMS for microgrid, comprising supply layer, demand layer, and neutral scheduling layer. Additionally, common mathematical optimization methods struggle with microgrid scheduling decision problem due to challenges in mechanism modeling, supply-demand uncertainty, and high real-time and autonomy requirements. Therefore, an improved proximal policy optimization (PPO) approach is proposed for the multi-layer EMS. Specifically, in the centrally managed supply layer, a centralized PPO algorithm is utilized to determine the optimal power generation strategy. In the decentralized demand layer, an auction market is established, and multi-agent proximal policy optimization (MAPPO) algorithm with an action-guidance-based mechanism is employed for each consumer, to implement individual auction strategy. The neutral scheduling layer interacts with other layers, manages information, and protects participant privacy. Numerical results validate the effectiveness of the proposed multi-layer EMS framework and the PPO-based optimization methods.