IRS Assisted NOMA Aided Mobile Edge Computing With Queue Stability: Heterogeneous Multi-Agent Reinforcement Learning

被引:43
作者
Yu, Jiadong [1 ]
Li, Yang [2 ]
Liu, Xiaolan [3 ]
Sun, Bo [4 ]
Wu, Yuan [2 ]
Tsang, Danny Hin-Kwok [1 ,5 ]
机构
[1] Hong Kong Univ Sci & Technol Guangzhou, Internet Things Thrust, Guangzhou 511400, Guangdong, Peoples R China
[2] Univ Macao, Dept Comp & Informat Sci, State Key Lab Internet Things Smart City, Macau, Peoples R China
[3] Loughborough Univ, Inst Digital Technol, London E20 3BS, England
[4] Chinese Univ Hong Kong, Dept Comp Sci & Engn, Shatin, Hong Kong, Peoples R China
[5] Hong Kong Univ Sci & Technol, Dept Elect & Comp Engn, Clear Water Bay, Hong Kong, Peoples R China
关键词
IRS; mobile edge computing; reinforcement learning; deep deterministic policy gradient; INTELLIGENT REFLECTING SURFACE; HYBRID NOMA; ENERGY; OPTIMIZATION; NETWORKS; MEC; MAXIMIZATION; POWER; 5G;
D O I
10.1109/TWC.2022.3224291
中图分类号
TM [电工技术]; TN [电子技术、通信技术];
学科分类号
0808 ; 0809 ;
摘要
By employing powerful edge servers for data processing, mobile edge computing (MEC) has been recognized as a promising technology to support emerging computation-intensive applications. Besides, non-orthogonal multiple access (NOMA)-aided MEC system can further enhance the spectral efficiency with massive tasks offloading. However, with more dynamic devices brought online and the uncontrollable stochastic channel environment, it is even desirable to deploy appealing technique, i.e., intelligent reflecting surfaces (IRS), in the MEC system to flexibly tune the communication environment and improve the system energy efficiency. In this paper, we investigate the joint offloading, communication and computation resource allocation for the IRS-assisted NOMA MEC system. We first formulate a mixed integer energy efficiency maximization problem with system queue stability constraint. We then propose the Lyapunov-function-based Mixed Integer Deep Deterministic Policy Gradient (LMIDDPG) algorithm which is based on the centralized reinforcement learning (RL) framework. To be specific, we design the mixed integer action space mapping which contains both continuous mapping and integer mapping. Moreover, the award function is defined as the upper-bound of the Lyapunov drift-plus-penalty function. To enable end devices (EDs) to choose actions independently at the execution stage, we further propose the Heterogeneous Multi-agent LMIDDPG (HMA-LMIDDPG) algorithm based on distributed RL framework with homogeneous EDs and heterogeneous base station (BS) as heterogeneous multi-agent. Numerical results show that our proposed algorithms can achieve superior energy efficiency performance to the benchmark algorithms while maintaining the queue stability. Specially, the distributed structure HMA-LMIDDPG can acquire more energy efficiency gain than the centralized structure LMIDDPG.
引用
收藏
页码:4296 / 4312
页数:17
相关论文
共 46 条
[1]   Internet of Things: A Survey on Enabling Technologies, Protocols, and Applications [J].
Al-Fuqaha, Ala ;
Guizani, Mohsen ;
Mohammadi, Mehdi ;
Aledhari, Mohammed ;
Ayyash, Moussa .
IEEE COMMUNICATIONS SURVEYS AND TUTORIALS, 2015, 17 (04) :2347-2376
[2]   Resource Allocation for Intelligent Reflecting Surface Aided Wireless Powered Mobile Edge Computing in OFDM Systems [J].
Bai, Tong ;
Pan, Cunhua ;
Ren, Hong ;
Deng, Yansha ;
Elkashlan, Maged ;
Nallanathan, Arumugam .
IEEE TRANSACTIONS ON WIRELESS COMMUNICATIONS, 2021, 20 (08) :5389-5407
[3]   Mobility Models, Traces and Impact of Mobility on Opportunistic Routing Algorithms: A Survey [J].
Batabyal, Suvadip ;
Bhaumik, Parama .
IEEE COMMUNICATIONS SURVEYS AND TUTORIALS, 2015, 17 (03) :1679-1707
[4]   Lyapunov-Guided Deep Reinforcement Learning for Stable Online Computation Offloading in Mobile-Edge Computing Networks [J].
Bi, Suzhi ;
Huang, Liang ;
Wang, Hui ;
Zhang, Ying-Jun Angela .
IEEE TRANSACTIONS ON WIRELESS COMMUNICATIONS, 2021, 20 (11) :7519-7537
[5]   Lyapunov Drift-Plus-Penalty Optimization for Queues With Finite Capacity [J].
Bracciale, Lorenzo ;
Loreti, Pierpaolo .
IEEE COMMUNICATIONS LETTERS, 2020, 24 (11) :2555-2558
[6]   Optimized Computation Offloading Performance in Virtual Edge Computing Systems via Deep Reinforcement Learning [J].
Chen, Xianfu ;
Zhang, Honggang ;
Wu, Celimuge ;
Mao, Shiwen ;
Ji, Yusheng ;
Bennis, Mehdi .
IEEE INTERNET OF THINGS JOURNAL, 2019, 6 (03) :4005-4018
[7]   NOMA-Based Multi-User Mobile Edge Computation Offloading via Cooperative Multi-Agent Deep Reinforcement Learning [J].
Chen, Zhao ;
Zhang, Lei ;
Pei, Yukui ;
Jiang, Chunxiao ;
Yin, Liuguo .
IEEE TRANSACTIONS ON COGNITIVE COMMUNICATIONS AND NETWORKING, 2022, 8 (01) :350-364
[8]   6G service-oriented space-air-ground integrated network: A survey [J].
Cheng, Nan ;
He, Jingchao ;
Yin, Zhisheng ;
Zhou, Conghao ;
Wu, Huaqing ;
Lyu, Feng ;
Zhou, Haibo ;
Shen, Xuemin .
CHINESE JOURNAL OF AERONAUTICS, 2022, 35 (09) :1-18
[9]   Intelligent Reflecting Surface Assisted Mobile Edge Computing for Internet of Things [J].
Chu, Zheng ;
Xiao, Pei ;
Shojafar, Mohammad ;
Mi, De ;
Mao, Juquan ;
Hao, Wanming .
IEEE WIRELESS COMMUNICATIONS LETTERS, 2021, 10 (03) :619-623
[10]   WHAT ROLE DO INTELLIGENT REFLECTING SURFACES PLAY IN MULTI-ANTENNA NON-ORTHOGONAL MULTIPLE ACCESS? [J].
de Sena, Arthur S. ;
Carrillo, Dick ;
Fang, Fang ;
Nardelli, Pedro H. J. ;
da Costa, Daniel B. ;
Dias, Ugo S. ;
Ding, Zhiguo ;
Papadias, Constantinos B. ;
Saad, Walid .
IEEE WIRELESS COMMUNICATIONS, 2020, 27 (05) :24-31