Maximizing Age-Energy Efficiency in Wireless Powered Industrial IoE Networks: A Dual-Layer DQN-Based Approach

This paper investigates the age of information (AoI) and energy efficiency of wireless powered industrial Internet of Everything (IIoE) network, where multiple low-power IIoE devices (IIoEDs) are wirelessly charged by a hybrid access point (HAP) to transmit their sensing information to the control nodes. To enhance the system's information timeliness with high energy efficiency, we define a novel performance metric, i.e., age-energy efficiency (AEE), which depicts the achievable AoI gain per unit energy consumption. Then, an optimization problem is formulated to maximize the system long-term AEE by jointly optimizing the IIoEDs scheduling and the HAP's transmit power. Due to the non-convexity of the formulated problem and the intractable challenges with discrete binary variables, we first model the problem as a two-stage discrete-time Markov decision process (MDP) with carefully designed state spaces, action spaces, and reward functions. We then propose a deep reinforcement learning (DRL)-based approach to find the effective scheduling strategy and transmit power. To improve the accuracy of the learned policy, we design a dual-layer deep Q-network (DLDQN) algorithm with fast convergence. Simulation results show that our proposed DLDQN algorithm can improve the AEE by at least 25% when the number of IIoEDs exceeds 50 compared with benchmarks. Moreover, with the proposed DLDQN algorithm, the system long-term AEE can be improved with the increase of the number of IIoEDs.