Feature Extraction in Q-Learning using Neural Networks

Integrating deep neural networks with reinforcement learning has exhibited excellent performance in the literature, highlighting the ability of neural networks to extract features. This paper begins with a simple Markov decision process inspired from a cognitive task. We show that Q-learning, and approximate Q-learning using a linear function approximation fail in this task. Instead, we show that Q-learning combined with a neural network-based function approximator can learn the optimal policy. Motivated by this finding, we outline procedures that allow the use of a neural network to extract appropriate features, which can then be used in a Q-learning framework with a linear function approximation, obtaining performance similar to that observed using Q-learning with neural networks. Our work suggests that neural networks can be used as feature extractors in the context of Q-learning.