Surfing Information: The Challenge of Intelligent Decision-Making

Reinforcement learning (RL) is indispensable for building intelligent decision-making agents. However, current RL algorithms suffer from statistical and computational inefficiencies that render them useless in most real-world applications. We argue that high-value information in the real world is essential for intelligent decision-making; however, it is not addressed by most RL formalisms. Through a closer investigation of high-value information, it becomes evident that, to exploit high-value information, there is a need to formalize intelligent decision-making as bounded-optimal lifelong RL. Thus, the challenge of achieving intelligent decision-making is summarized as effectively surfing information, specifically regarding handling the non-IID (independent and identically distributed) information stream while operating with limited resources. This study discusses the design of an intelligent decision-making agent and examines its primary challenges, which are (a) online learning for non-IID data streams, (b) efficient reasoning with limited resources, and (c) the exploration–exploitation dilemma. We review relevant problems and research in the field of RL literature and conclude that current RL methods are insufficient to address these challenges. We propose that an agent capable of overcoming these challenges could effectively surf the information overload in the real world and achieve sample-and compute-efficient intelligent decision-making. © 2023 Chenyang Wu and Zongzhang Zhang.