Factored Particle Swarm Optimization for Policy Co-training in Reinforcement Learning

Uncertainty of the environment limits the circumstances with which any optimization problem can provide meaningful information. Multiple optimizers can combat this problem by communicating di.erent information through cooperative coevolution. In reinforcement learning (RL), uncertainty can be reduced by applying learned policies collaboratively with another agent. Here, we propose policy Co-training with Factored Evolutionary Algorithms (CoFEA) to evolve an optimal policy for such scenarios. We hypothesize that self-paced co-training can allow factored particle swarms with imperfect knowledge to consolidate knowledge from each of their imperfect policies in order to approximate a single optimal policy. Additionally, we show how the performance of co-training swarms of RL agents can be maximized through the speci.c use of Expected SARSA as the policy learner. We evaluate CoFEA against comparable RL algorithms and attempt to establish limits for which our procedure does and does not provide bene.t. Our results indicate that Particle Swarm Optimization (PSO) is e.ective in training multiple agents under uncertainty and that FEA reduces swarm and policy updates. This paper contributes to the.eld of cooperative co-evolutionary algorithms by proposing a method by which factored evolutionary techniques can signi.cantly improve how multiple RL agents collaborate under extreme uncertainty to solve complex tasks faster than a single agent can under identical conditions.