Reinforcement learning for the optimization of electric vehicle virtual power plants

Integrating weather-dependent renewable energy sources into the electricity system impose challenges on the power grid. Balancing services are needed, which can be provided by virtual power plants (VPP) that aggregate distributed energy resources (DER) to consume or produce electricity on demand. Electric vehicle (EV) fleets can use idle cars' batteries as combined storage to offer balancing services on smart electricity markets. However, there are risks associated with this business model extension. The fleet faces severe imbalance penalties if it cannot charge the offered amount of balancing energy due to the vehicles' unpredicted mobility demand. Ensuring the fleet can fulfill all market commitments risks denying profitable customer rentals. We study the design of a decision support system that estimates these risks, dynamically adjusts the composition of a VPP portfolio, and profitably places bids on multiple electricity markets simultaneously. Here we show that a reinforcement learning agent can optimize the VPP portfolio by learning from favorable market conditions and fleet demand uncertainties. In comparison to previous research, in which the bidding risks were unknown and fleets could only offer conservative amounts of balancing power to a single market, our proposed approach increases the amount of offered balancing power by 48% to 82% and achieves a charging cost reduction of the fleet by 25%. In experiments with real-world carsharing data of 500 EVs, we found that mobility demand forecasting algorithms' accuracy is crucial for a successful bidding strategy. Moreover, we show that recent advancements in deep reinforcement learning decrease the convergence time and improve the results' robustness. Our results demonstrate how modern RL algorithms can be successfully used for fleet management, VPP optimization, and demand response in the smart grid. We anticipate that DER, such as EVs, will play an essential role in providing reliable backup power for the grid and formulate market design recommendations to allow easier access to these resources.