Population-Invariant MADRL for AoI-Aware UAV Trajectory Design and Communication Scheduling in Wireless Sensor Networks

Unmanned aerial vehicles (UAVs) are recognized as effective data collectors for wireless sensor networks. The Age of Information (AoI), a metric indicating data freshness, is crucial for decision making in time-sensitive applications. It can be significantly reduced by jointly optimizing UAV trajectories and communication scheduling of sensor nodes (SNs). However, rapid changes in the environment make it challenging to predesign UAV trajectories and communication scheduling decisions using traditional methods, especially when central controllers are absent and the numbers of UAVs and SNs vary. In this article, we propose hypernetwork-based QMIX (HyperQMIX), a population-invariant multiagent deep reinforcement learning (MADRL) algorithm capable of transferring policies across tasks with varying population sizes. First, we design neural network modules adaptable to varying input and output dimensions, facilitated by parameter generation through a hypernetwork. Then, HyperQMIX leverages these modules to process fluctuations in state and action dimensions. This approach ensures that the network structure remains consistent regardless of population sizes, thereby enhancing the algorithm's scalability. Extensive simulations demonstrate that HyperQMIX significantly outperforms state-of-the-art algorithms in terms of learning efficiency and converged performance. Moreover, agents pretrained with HyperQMIX perform well in tasks of different population sizes without additional training. Fine-tuning these models achieves performance comparable to training from scratch.