Bringing Fairness to Actor-Critic Reinforcement Learning for Network Utility Optimization

被引:10
作者
Chen, Jingdi [1 ]
Wang, Yimeng [1 ]
Lan, Tian [1 ]
机构
[1] George Washington Univ, Dept ECE, Washington, DC 20052 USA
来源
IEEE CONFERENCE ON COMPUTER COMMUNICATIONS (IEEE INFOCOM 2021) | 2021年
关键词
MAX-MIN;
D O I
10.1109/INFOCOM42981.2021.9488823
中图分类号
TP3 [计算技术、计算机技术];
学科分类号
0812 ;
摘要
Fairness is a crucial design objective in virtually all network optimization problems, where limited system resources are shared by multiple agents. Recently, reinforcement learning has been successfully applied to autonomous online decision making in many network design and optimization problems. However, most of them try to maximize the long-term (discounted) reward of all agents, without taking fairness into account. In this paper, we propose a family of algorithms that bring fairness to actorcritic reinforcement learning for optimizing general fairness utility functions. In particular, we present a novel method for adjusting the rewards in standard reinforcement learning by a multiplicative weight depending on both the shape of fairness utility and some statistics of past rewards. It is shown that for proper choice of the adjusted rewards, a policy gradient update converges to at least a stationary point of general alpha-fairness utility optimization. It inspires the design of fairness optimization algorithms in actor-critic reinforcement learning. Evaluations show that the proposed algorithm can be easily deployed in real-world network optimization problems, such as wireless scheduling and video QoE optimization, and can significantly improve the fairness utility value over previous heuristics and learning algorithms.
引用
收藏
页数:10
相关论文
共 45 条
  • [1] Agarwal M., 2019, Reinforcement learning with non-markovian rewards
  • [2] Al-Abbasi A.O., 2018, CoRR
  • [3] Dynamic Heterogeneous Learning Games for Opportunistic Access in LTE-Based Macro/Femtocell Deployments
    Alnwaimi, Ghassan
    Vahid, Seiamak
    Moessner, Klaus
    [J]. IEEE TRANSACTIONS ON WIRELESS COMMUNICATIONS, 2015, 14 (04) : 2294 - 2308
  • [4] [Anonymous], 1998, COMPUTER
  • [5] Beynier A., 2018, 17 INT C AUT AG MULT
  • [6] Understanding Fairness and its Impact on Quality of Service in IEEE 802.11
    Bredel, Michael
    Fidler, Markus
    [J]. IEEE INFOCOM 2009 - IEEE CONFERENCE ON COMPUTER COMMUNICATIONS, VOLS 1-5, 2009, : 1098 - 1106
  • [7] Bu T., 2006, P IEEE INFOCOM, P1
  • [8] Castro D.D., 2009, CoRR
  • [9] Multi-Agent Reinforcement Learning-Based Resource Allocation for UAV Networks
    Cui, Jingjing
    Liu, Yuanwei
    Nallanathan, Arumugam
    [J]. IEEE TRANSACTIONS ON WIRELESS COMMUNICATIONS, 2020, 19 (02) : 729 - 743
  • [10] de Jong S., 2007, Considerations for fairness in multi-agent systems