Systematic Performance Evaluation of Reinforcement Learning Algorithms Applied to Wastewater Treatment Control Optimization

Treatment of wastewater using activated sludge relieson severalcomplex, nonlinear processes. While activated sludge systems can providehigh levels of treatment, including nutrient removal, operating thesesystems is often challenging and energy intensive. Significant researchinvestment has been made in recent years into improving control optimizationof such systems, through both domain knowledge and, more recently, machine learning. This study leverages a novel interface betweena common process modeling software and a Python reinforcement learningenvironment to evaluate four common reinforcement learning algorithmsfor their ability to minimize treatment energy use while maintainingeffluent compliance within the Benchmark Simulation Model No. 1 (BSM1)simulation. Three of the algorithms tested, deep Q-learning, proximalpolicy optimization, and synchronous advantage actor critic, generallyperformed poorly over the scenarios tested in this study. In contrast,the twin delayed deep deterministic policy gradient (TD3) algorithmconsistently produced a high level of control optimization while maintainingthe treatment requirements. Under the best selection of state observationfeatures, TD3 control optimization reduced aeration and pumping energyrequirements by 14.3% compared to the BSM1 benchmark control, outperformingthe advanced domain-based control strategy of ammonia-based aerationcontrol, although future work is necessary to improve robustness ofRL implementation. Reinforcementlearning can provide better than human controloptimization. This study evaluates reinforcement learning controloptimization strategies to reduce wastewater treatment energy costswhile maintaining effluent compliance.