Fighting Uncertainty with Gradients: Offline Reinforcement Learning via Diffusion Score Matching

被引:0
作者
Suh, H. J. Terry [1 ]
Chou, Glen [1 ]
Dai, Hongkai [2 ]
Yang, Lujie [1 ]
Gupta, Abhishek [3 ]
Tedrake, Russ [1 ]
机构
[1] MIT, CSAIL, Cambridge, MA 02139 USA
[2] Toyota Res Inst, Los Altos, CA 94022 USA
[3] Univ Washington, Seattle, WA 98195 USA
来源
CONFERENCE ON ROBOT LEARNING, VOL 229 | 2023年 / 229卷
关键词
Diffusion; Score-Matching; Offline; Model-Based Reinforcement Learning; Imitation Learning; Planning under Uncertainty;
D O I
暂无
中图分类号
TP18 [人工智能理论];
学科分类号
081104 ; 0812 ; 0835 ; 1405 ;
摘要
Gradient-based methods enable efficient search capabilities in high dimensions. However, in order to apply them effectively in offline optimization paradigms such as offline Reinforcement Learning (RL) or Imitation Learning (IL), we require a more careful consideration of how uncertainty estimation interplays with first-order methods that attempt to minimize them. We study smoothed distance to data as an uncertainty metric, and claim that it has two beneficial properties: (i) it allows gradient-based methods that attempt to minimize uncertainty to drive iterates to data as smoothing is annealed, and (ii) it facilitates analysis of model bias with Lipschitz constants. As distance to data can be expensive to compute online, we consider settings where we need amortize this computation. Instead of learning the distance however, we propose to learn its gradients directly as an oracle for first-order optimizers. We show these gradients can be efficiently learned with score-matching techniques by leveraging the equivalence between distance to data and data likelihood. Using this insight, we propose Score-Guided Planning (SGP), a planning algorithm for offline RL that utilizes score-matching to enable first-order planning in high-dimensional problems, where zeroth-order methods were unable to scale, and ensembles were unable to overcome local minima. Website: https://sites.google.com/view/score-guided-planning/home
引用
收藏
页数:27
相关论文
共 78 条
  • [1] Abbeel Pieter, 2006, P 23 INT C MACH LEAR, V148, P1, DOI DOI 10.1145/1143844.1143845
  • [2] [Anonymous], 2002, OXF LEC S MATH APPL
  • [3] [Anonymous], 2021, ADV NEURAL INFORM PR
  • [4] Argenson A., 2021, Model-based offline planning
  • [5] Atkeson CG, 1997, IEEE INT CONF ROBOT, P3557, DOI 10.1109/ROBOT.1997.606886
  • [6] Chi Cheng, 2023, Diffusion policy: Visuomotor policy learning via action diffusion
  • [7] Chou G., 2023, ARXIV
  • [8] Safe Output Feedback Motion Planning from Images via Learned Perception Modules and Contraction Theory
    Chou, Glen
    Ozay, Necmiye
    Berenson, Dmitry
    [J]. ALGORITHMIC FOUNDATIONS OF ROBOTICS XV, 2023, 25 : 349 - 367
  • [9] Model Error Propagation via Learned Contraction Metrics for Safe Feedback Motion Planning of Unknown Systems
    Chou, Glen
    Ozay, Necmiye
    Berenson, Dmitry
    [J]. 2021 60TH IEEE CONFERENCE ON DECISION AND CONTROL (CDC), 2021, : 3576 - 3583
  • [10] Chua K., 2018, DEEP REINFORCEMENT L