Gradient compensation traces based temporal difference learning

For online updates and data efficiency, forward-view algorithms are transformed into backward-views, such as temporal difference learning (TD) and its control versions, by eligibility traces. Existing researches on eligibility traces, such as TD(A) and true-online TD(A), mainly focus on the equivalence between forward-views and backward-views. However, the choice of A refers to the time scope of the credit assignment, and a small A accelerates the decay of credit over the time. This paper takes a different implementation of the backward-view named gradient compensation traces (GCT). GCT compensates the difference between a bootstrapping estimated gradient and the true gradient online to remove the extra decay of the credit. Based on GCT, the corresponding temporal difference learning (gradient compensation TD, GCTD) is proved to converge conditionally. The sensitivity of GCTD's hyper-parameter is analyzed in the nonlinear long-corridor and linear random-walk task. The proposed algorithm is comparable with true-online TD(A) in the basic Mountain Car task, and outperforms the baselines in the reward sparse setting. (c) 2021 Elsevier B.V. All rights reserved.