A reinforcement learning-based approach for online optimal control of self-adaptive real-time systems

This paper deals with self-adaptive real-time embedded systems (RTES). A self-adaptive system can operate in different modes. Each mode encodes a set of real-time tasks. To be executed, each task is allocated to a processor (placement) and assigned a priority (scheduling), while respecting timing constraints. An adaptation scenario allows switching between modes by adding, removing, and updating task parameters that must meet related deadlines after adaptation. For such systems, anticipating all operational modes at design time is usually impossible. Online reinforcement learning is increasingly used in the presence of design-time uncertainty. To tackle this problem, we formalize the placement and scheduling problems in self-adaptive RTES as a Markov decision process and propose related algorithms based on Q-learning. Then, we introduce an approach that integrates the proposed algorithms to assist designers in the development of self-adaptive RTES. At the design level, the RL Placement and the RL Scheduler are proposed to process predictable adaptation scenarios. These modules are designed to generate placement and scheduling models for an application while maximizing system extensibility and ensuring real-time feasibility. At the execution level, the RL Adapter is defined to process online adaptations. Indeed, the goal of the RL Adapter agent is to reject the adaptation scenario when feasibility concerns are raised; otherwise, it generates a new feasible placement and scheduling. We apply and simulate the proposed approach to a healthcare robot case study to show its applicability. Performance evaluations are conducted to prove the effectiveness of the proposed approach compared to related works.