Computation Offloading via Multi-Agent Deep Reinforcement Learning in Aerial Hierarchical Edge Computing Systems

The exponential growth of Internet of Things (IoT) devices and emerging applications have significantly increased the requirements for ubiquitous connectivity and efficient computing paradigms. Traditional terrestrial edge computing architectures cannot provide massive IoT connectivity worldwide. In this article, we propose an aerial hierarchical mobile edge computing system composed of high-altitude platforms (HAPs) and unmanned aerial vehicles (UAVs). In particular, we consider non-divisible tasks and formulate a task offloading problem to minimize the long-term processing cost of tasks while satisfying the queueing mechanism in the offloading procedure and processing procedure of tasks. We propose a multi-agent deep reinforcement learning (DRL) based computation offloading algorithm in which each device can make its offloading decision according to local observations. Due to the limited computing resources of UAVs, high task loads of UAVs will increase the ratio of abandoning offloaded tasks. To increase the success ratio of completing tasks, the convolutional LSTM (ConvLSTM) network is utilized to estimate the future task loads of UAVs. In addition, a prioritized experience replay (PER) method is proposed to increase the convergence speed and improve the training stability. The experimental results demonstrate that the proposed computation offloading algorithm outperforms other benchmark methods.