Detection of Man in the Middle Attacks in Model-Free Reinforcement Learning for the Linear Quadratic Regulator

We consider the problem of a learning-based, man-in-the-middle (MITM) attack in a cyber-physical system. We use a simple abstraction where an agent performs linear quadratic regulation (LQR) of a discrete-time, linear, time-invariant (LTI) system with stochastic disturbances, using model-free reinforcement learning. The system may be subject to an adversarial attack that overrides the feedback signal and the controller actions. We propose a "Bellman Deviation" algorithm that can be used by the agent to detect the attack. This algorithm only requires an estimate of the Q-function, and optimal average stage cost, and no explicit information of the system parameters. We show that the proposed algorithm asymptotically guarantees attack detection (AD) with high probability while avoiding false alarms, when an "informational advantage" condition is met. This condition compares the amount of information the agent has aquired about the system with the one aquired by the adversary.