Online reinforcement learning control for maintaining an optimum beam collision on an electron-positron collider

For double-ring colliders, it is a vital task to make the two beams meeting each other properly at the interaction point. BEPCII is a double-ring collider that operates in the decay mode, as the beam currents decrease over time, the beam orbits need to be continuously adjusted to maintain the optimum collision conditions. Originally, this task was carried out manually, with operators adjusting three offset control knobs (x, y, y') according to the luminosity. There is a crucial necessity to implement advanced automated control approaches. However, the feedback methods are ineffective due to the constraints imposed by machine characteristics. Nevertheless, the optimization methods are also limited by the slow response speed of BEPCII and have intrinsic limitations. Reinforcement learning (RL), which learns from past experiences, provides a new approach for handling such problems. In this paper, we implemented two automated control methods: the dither method and the deep Q-network (DQN) RL method. The dither method is a numerical optimization method used to provide more online data for DQN training. With the help of the historical data from the dither method, we successfully trained a DQN agent to control the offset knobs to optimize the luminosity. The DQN method exhibited significantly better performance than the dither method, achieving faster optimization speeds and yielding higher integral luminosity. Furthermore, the DQN method has been integrated into the daily operations of BEPCII, achieving long-term stable deployment and effectively substituting manual labor.