Q Learning-Based Routing Protocol With Accelerating Convergence for Underwater Wireless Sensor Networks

Underwater wireless sensor networks have emerged as a promising technology for various underwater applications. Considering the characteristics such as limited energy and high end-to-end delay in underwater wireless sensor networks, it is important to design an underwater routing protocol with high energy efficiency, low end-to-end delay and high reliability. Therefore, a Q learning (QL)-based routing protocol is proposed in this paper. First, a Q learning-based framework is constructed by considering link connectivity and the information of residual energy, depth and neighboring nodes. The framework enables protocols to adapt to the dynamic environment and facilitate efficient transmission. Furthermore, to address the slow convergence of Q learning in underwater wireless sensor networks, a Q value initialization strategy using layer information is designed to accelerate the convergence speed. In addition, an adaptive discount mechanism and a dynamic learning mechanism are proposed to update Q values for adapting to the changing network topology and improve the reliability of Q values for nodes rarely selected, respectively. Finally, the superior performance of the proposed protocol is evaluated through simulations. Simulation results show that the proposed protocol can still accelerate the convergence speed in reducing the energy tax by 37.16% and 23.08%, and the average end-to-end delay by 29.94% and 16.91% as compared to other Q learning-based routing protocols QELAR and QDAR under dynamic environment, while maintaining a higher packet delivery ratio (PDR).