Reinforcement Learning for Energy-Storage Systems in Grid-Connected Microgrids: An Investigation of Online vs. Offline Implementation

被引:8
作者
Ali, Khawaja Haider [1 ,2 ]
Sigalo, Marvin [1 ]
Das, Saptarshi [1 ]
Anderlini, Enrico [3 ]
Tahir, Asif Ali [1 ]
Abusara, Mohammad [1 ]
机构
[1] Univ Exeter, Coll Engn Math & Phys Sci, Penryn Campus, Penryn TR10 9FE, Cornwall, England
[2] Sukkur IBA Univ, Dept Elect Engn, Sukkur 65200, Pakistan
[3] UCL, Dept Mech Engn, Roberts Bldg, London WC1E 7JE, England
基金
英国工程与自然科学研究理事会;
关键词
reinforcement learning (RL); microgrid; battery management; offline and online RL; optimisation; SOLAR-RADIATION; MANAGEMENT; LOAD; MODELS;
D O I
10.3390/en14185688
中图分类号
TE [石油、天然气工业]; TK [能源与动力工程];
学科分类号
0807 ; 0820 ;
摘要
Grid-connected microgrids consisting of renewable energy sources, battery storage, and load require an appropriate energy management system that controls the battery operation. Traditionally, the operation of the battery is optimised using 24 h of forecasted data of load demand and renewable energy sources (RES) generation using offline optimisation techniques, where the battery actions (charge/discharge/idle) are determined before the start of the day. Reinforcement Learning (RL) has recently been suggested as an alternative to these traditional techniques due to its ability to learn optimal policy online using real data. Two approaches of RL have been suggested in the literature viz. offline and online. In offline RL, the agent learns the optimum policy using predicted generation and load data. Once convergence is achieved, battery commands are dispatched in real time. This method is similar to traditional methods because it relies on forecasted data. In online RL, on the other hand, the agent learns the optimum policy by interacting with the system in real time using real data. This paper investigates the effectiveness of both the approaches. White Gaussian noise with different standard deviations was added to real data to create synthetic predicted data to validate the method. In the first approach, the predicted data were used by an offline RL algorithm. In the second approach, the online RL algorithm interacted with real streaming data in real time, and the agent was trained using real data. When the energy costs of the two approaches were compared, it was found that the online RL provides better results than the offline approach if the difference between real and predicted data is greater than 1.6%.
引用
收藏
页数:18
相关论文
共 47 条
[1]  
Ahmad R. F., 2021, PAK J ENG TECHNOL, V4, P100, DOI DOI 10.51846/VOL4ISS2PP100-107
[2]   Supervisory Control for Power Management of an Islanded AC Microgrid Using a Frequency Signalling-Based Fuzzy Logic Controller [J].
Al Badwawi, Rashid ;
Issa, Walid R. ;
Mallick, Tapas K. ;
Abusara, Mohammad .
IEEE TRANSACTIONS ON SUSTAINABLE ENERGY, 2019, 10 (01) :94-104
[3]   Optimal sizing of thermal energy storage systems for CHP plants considering specific investment costs: A case study [J].
Benalcazar, Pablo .
ENERGY, 2021, 234
[4]   Estimation of demand diversity and daily demand profile for off-grid electrification in developing countries [J].
Boait, P. ;
Advani, V. ;
Gammon, R. .
ENERGY FOR SUSTAINABLE DEVELOPMENT, 2015, 29 :135-141
[5]   Double Deep Q-Learning-Based Distributed Operation of Battery Energy Storage System Considering Uncertainties [J].
Bui, Van-Hai ;
Hussain, Akhtar ;
Kim, Hak-Man .
IEEE TRANSACTIONS ON SMART GRID, 2020, 11 (01) :457-469
[6]   Q-Learning-Based Operation Strategy for Community Battery Energy Storage System (CBESS) in Microgrid System [J].
Bui, Van-Hai ;
Hussain, Akhtar ;
Kim, Hak-Man .
ENERGIES, 2019, 12 (09)
[7]   Approximate Bayes Optimal Policy Search using Neural Networks [J].
Castronovo, Michael ;
Francois-Lavet, Vincent ;
Fonteneau, Raphael ;
Ernst, Damien ;
Couetoux, Adrien .
ICAART: PROCEEDINGS OF THE 9TH INTERNATIONAL CONFERENCE ON AGENTS AND ARTIFICIAL INTELLIGENCE, VOL 2, 2017, :142-153
[8]   Economic analysis and optimal energy management models for microgrid systems: A case study in Taiwan [J].
Chen, Yen-Haw ;
Lu, Su-Ying ;
Chang, Yung-Ruei ;
Lee, Ta-Tung ;
Hu, Ming-Che .
APPLIED ENERGY, 2013, 103 :145-154
[9]   Mixed-integer linear programming based optimization strategies for renewable energy communities [J].
Cosic, Armin ;
Stadler, Michael ;
Mansoor, Muhammad ;
Zellinger, Michael .
ENERGY, 2021, 237
[10]   A Computationally Efficient Optimization Approach for Battery Systems in Islanded Microgrid [J].
Das, Avijit ;
Ni, Zhen .
IEEE TRANSACTIONS ON SMART GRID, 2018, 9 (06) :6489-6499