Two-Timescale Learning-Based Task Offloading for Remote IoT in Integrated Satellite-Terrestrial Networks

被引:33
作者
Han, Dairu [1 ]
Ye, Qiang [2 ]
Peng, Haixia [3 ]
Wu, Wen [4 ]
Wu, Huaqing [5 ]
Liao, Wenhe [1 ]
Shen, Xuemin [6 ]
机构
[1] Nanjing Univ Sci & Technol, Dept Aeronaut & Astronaut Engn, Nanjing 210094, Peoples R China
[2] Mem Univ Newfoundland, Dept Comp Sci, St John, NF A1B 3X5, Canada
[3] Xi An Jiao Tong Univ, Sch Informat & Commun Engn, Xian 710049, Peoples R China
[4] Peng Cheng Lab, Frontier Res Ctr, Shenzhen 518066, Peoples R China
[5] Univ Calgary, Dept Elect & Software Engn, Calgary, AB T2N 1N4, Canada
[6] Univ Waterloo, Dept Elect & Comp Engn, Waterloo, ON N2L 3G1, Canada
基金
中国国家自然科学基金; 加拿大自然科学与工程研究理事会;
关键词
Task analysis; Satellites; Backhaul networks; Internet of Things; Satellite broadcasting; Computer architecture; Delays; Bandwidth allocation; integrated satellite-terrestrial networks (ISTNs); offloading link selection; reinforcement learning (RL); remote Internet of Things (IoT); task offloading; TECHNOLOGIES; OPTIMIZATION; CHALLENGES; INTERNET; 5G;
D O I
10.1109/JIOT.2023.3237209
中图分类号
TP [自动化技术、计算机技术];
学科分类号
0812 ;
摘要
In this article, we propose an integrated satellite-terrestrial network (ISTN) architecture to support delay-sensitive task offloading for remote Internet of Things (IoT), in which satellite networks serve as a complement to terrestrial networks by providing additional communication resources, backhaul capacities, and seamless coverage. Under this architecture, we investigate how to jointly make offloading link selection and bandwidth allocation decisions for BSs and IoT users. Considering the differentiated decision-making time granularities, we formulate a two-timescale stochastic optimization problem to minimize the overall task offloading delay. To accommodate the two-timescale network dynamics and characterize state-action relations, we establish a hierarchical Markov decision process (H-MDP) framework with two separate agents tackling two-timescale network management decisions, and two evolved MDP-based subproblems are formulated accordingly. To efficiently solve the subproblems, we further develop a hybrid proximal policy optimization (H-PPO)-based algorithm. Specifically, a hybrid actor-critic architecture is designed to deal with the mixed discrete and continuous actions. In addition, an action mask layer and an action shaping function are designed to sample feasible task offloading decisions from the time-variant action set. Extensive simulation results have validated the superiority of the proposed ISTN architecture and the H-PPO-based algorithm, especially, in scenarios with scarce spectrum resources and heavy traffic loads.
引用
收藏
页码:10131 / 10145
页数:15
相关论文
共 40 条
[1]   Latency-Aware Offloading in Integrated Satellite Terrestrial Networks [J].
Abderrahim, Wiem ;
Amin, Osama ;
Alouini, Mohamed-Slim ;
Shihada, Basem .
IEEE OPEN JOURNAL OF THE COMMUNICATIONS SOCIETY, 2020, 1 :490-500
[2]   Deep Reinforcement Learning For Multi-User Access Control in Non-Terrestrial Networks [J].
Cao, Yang ;
Lien, Shao-Yu ;
Liang, Ying-Chang .
IEEE TRANSACTIONS ON COMMUNICATIONS, 2021, 69 (03) :1605-1619
[3]   A Survey on Technologies, Standards and Open Challenges in Satellite IoT [J].
Centenaro, Marco ;
Costa, Cristina E. ;
Granelli, Fabrizio ;
Sacchi, Claudio ;
Vangelista, Lorenzo .
IEEE COMMUNICATIONS SURVEYS AND TUTORIALS, 2021, 23 (03) :1693-1720
[4]   Learning-Based Computation Offloading for IoRT Through Ka/Q-Band Satellite-Terrestrial Integrated Networks [J].
Chen, Tianjiao ;
Liu, Jiang ;
Ye, Qiang ;
Zhuang, Weihua ;
Zhang, Weiting ;
Huang, Tao ;
Liu, Yunjie .
IEEE INTERNET OF THINGS JOURNAL, 2021, 9 (14) :12056-12070
[5]   QoS Optimisation of eMBB Services in Converged 5G-Satellite Networks [J].
de Cola, Tomaso ;
Bisio, Igor .
IEEE TRANSACTIONS ON VEHICULAR TECHNOLOGY, 2020, 69 (10) :12098-12110
[6]   Ultra-Dense LEO: Integrating Terrestrial-Satellite Networks Into 5G and Beyond for Data Offloading [J].
Di, Boya ;
Zhang, Hongliang ;
Song, Lingyang ;
Li, Yonghui ;
Li, Geoffrey Ye .
IEEE TRANSACTIONS ON WIRELESS COMMUNICATIONS, 2019, 18 (01) :47-62
[7]   A prediction model that combines rain attenuation and other propagation impairments along earth-satellite paths [J].
Dissanayake, A ;
Allnutt, J ;
Haidara, F .
IEEE TRANSACTIONS ON ANTENNAS AND PROPAGATION, 1997, 45 (10) :1546-1558
[8]   Application Aware Workload Allocation for Edge Computing-Based IoT [J].
Fan, Qiang ;
Ansari, Nirwan .
IEEE INTERNET OF THINGS JOURNAL, 2018, 5 (03) :2146-2153
[9]  
Fan Z, 2019, Arxiv, DOI arXiv:1903.01344
[10]   5G Embraces Satellites for 6G Ubiquitous IoT: Basic Models for Integrated Satellite Terrestrial Networks [J].
Fang, Xinran ;
Feng, Wei ;
Wei, Te ;
Chen, Yunfei ;
Ge, Ning ;
Wang, Cheng-Xiang .
IEEE INTERNET OF THINGS JOURNAL, 2021, 8 (18) :14399-14417