A New Approach for Value Function Approximation Based on Automatic State Partition

Value function is usually used to deal with the reinforcement learning problems. In large or even continuous states, function approximation must be used to represent value function. Much of the current work carried out, however, has to design the structure of function approximation in advanced which cannot be adjusted during learning. In this paper, we propose a novel function approximation called Puzzy CMAC (FCMAC) with automatic state partition (ASP-FCMAC) to automate the structure design for FCMAC. Based on CMAC (also known as tile coding), ASP-FCMAC employs fuzzy membership function to avoid the setting of parameter in CMAC, and makes use of Bellman error to partition the state automatically so as to generate the structure of FCMAC. Empirical results in both mountain car and RoboCup Keepaway domains demonstrate that ASP-FCMAC can automatically generate the structure of FCMAC and agent using it can learn efficiently.