Universal Off-Policy Evaluation

被引：0

作者：

Chandak, Yash ^{[1
]}

Niekum, Scott ^{[2
]}

da Silva, Bruno Castro ^{[1
]}

Learned-Miller, Erik ^{[1
]}

Brunskill, Emma ^{[3
]}

Thomas, Philip S. ^{[1
]}

机构：

[1] Univ Massachusetts, Amherst, MA 01003 USA

[2] Univ Texas Austin, Austin, TX 78712 USA

[3] Stanford Univ, Stanford, CA 94305 USA

来源：

ADVANCES IN NEURAL INFORMATION PROCESSING SYSTEMS 34 (NEURIPS 2021) | 2021年

基金：

美国国家科学基金会;

关键词：

MARKOV DECISION-PROCESSES; WILD BOOTSTRAP; VARIANCE; INFERENCE; BOUNDS;

D O I：

暂无

中图分类号：

TP18 [人工智能理论];

学科分类号：

081104 ; 0812 ; 0835 ; 1405 ;

摘要：

When faced with sequential decision-making problems, it is often useful to be able to predict what would happen if decisions were made using a new policy. Those predictions must often be based on data collected under some previously used decision-making rule. Many previous methods enable such off-policy (or counterfactual) estimation of the expected value of a performance measure called the return. In this paper, we take the first steps towards a universal off-policy estimator (UnO)-one that provides off-policy estimates and high-confidence bounds for any parameter of the return distribution. We use UnO for estimating and simultaneously bounding the mean, variance, quantiles/median, inter-quantile range, CVaR, and the entire cumulative distribution of returns. Finally, we also discuss UnO's applicability in various settings, including fully observable, partially observable (i.e., with unobserved confounders), Markovian, non-Markovian, stationary, smoothly non-stationary, and discrete distribution shifts.

引用

页数：16

共 105 条

[1] On the coherence of expected shortfall [J].