Rethinking Stochasticity in Neural Networks for Reinforcement Learning with Continuous Actions

In this paper, we reconsider the use of stochasticity in neural networks for reinforcement learning in continuous action spaces. We consider stochasticity from both a reinforcement learning perspective and a neural networks perspective, leading us to reconsider whether noise sampling for exploration should take place at the level of the synapse (S), unit (U), or network (N). To investigate this question, we introduce a superset of the venerable multiparameter REINFORCE algorithm that we call REINFORCE SUN because it allows for stochasticity at each of these levels, and compare these variants on multidimensional problem sets with either continuous or discrete states. Our results show that moving stochasticity from the unit level to the synapse level substantially improves performance across all instances considered. As placing stochasticity at the unit level is nearly ubiquitous within the discipline, our results suggest that this standard practice should be reexamined more broadly.