An improved Q-learning algorithm (XQL) for mobile robot path planning in unknown environments

Traditional Q-Learning's slow convergence, non-smooth paths, and high energy consumption hinder mobile robots' path planning in environments with random obstacles. We introduce an improved algorithm, XQL (eXtended Q-Learning), designed to address these limitations. First, we enhance the Q-value update mechanism by introducing a neighboring state smoothing Q-function. This enhancement significantly accelerates the convergence process. Next, we apply the artificial potential field method to design a reward function. This divides the reward into a global gravitational field and a target area reward, providing more accurate guidance for the robot's movement. We also introduce a cosine function to adaptively adjust the greedy factor. This allows the epsilon value to change dynamically based on the algorithm's state, balancing exploration and exploitation. Finally, to reduce energy consumption associated with sharp turns, we introduce an optimal action pool. This pool filters and optimizes the action set at each step. Simulation results show that, compared to SARSA, Q-Learning, DQN, and EPRQL, the XQL algorithm outperforms these methods in random environments. It improves convergence speed and path planning efficiency. It also reduces the number of path inflection points, resulting in smoother paths. This enhances both the robot's safety and energy efficiency. These findings highlight the strong adaptability of the XQL algorithm in dealing with random environments.