Double Q-learning Agent for Othello Board Game

This paper presents the first application of the Double Q-learning algorithm to the game of Othello. Reinforcement learning has previously been successfully applied to Othello using the canonical reinforcement learning algorithms, Q-learning and TD-learning. However, the algorithms suffer from considerable drawbacks. Q-learning frequently tends to be overoptimistic during evaluation, while TD-learning can get stuck in local minima. To overcome the disadvantages of the existing work, we propose using a Double Q-learning agent to play Othello and prove that it performs better than the existing learning agents. In addition to developing and implementing the Double Q-learning agent, we implement the Q-learning and TD-learning agents. The agents are trained and tested against two fixed opponents: a random player and a heuristic player. The performance of the Double Q-learning agent is compared with performance of the existing learning agents. The Double Q-learning agent outperforms them, although it takes longer, on average, to make each move. Further, we show that the Double Q-learning agent performs at its best with two hidden layers using the tanh function.