Energy-Aware VNF-FG Placement with Transformer-based Deep Reinforcement Learning

Although Network Function Virtualization (NFV) has introduced better flexibility and agility to the way network operators design, manage, and deploy their network services, it is still challenging to find the optimal real-time placement of network services which have evolved into complex dynamic graphs (or VNF-FGs) to satisfy the Quality of Service (QoS) requirements of their end-users and accommodate their dynamically changing service demands. Another crucial challenge that compounds the complexity of the online network service provisioning is to efficiently improve the utilization of the limited resources and reduce energy consumption and costs for service and infrastructure providers in large-scale networking environments such as 5G networks, edge computing, and Internet of Things (IoT). To meet both user and service provider needs, this paper proposes a novel Transformer-based Deep Reinforcement Learning (DRL) architecture, called TDRL (Transformer based-DRL), to address the dynamic energy-aware VNF-FG placement problem. Our intelligent encoder-decoder architecture leverages the power of both Graph Attention Networks (GAT) which extract the important features of the physical network, and sequence-to-sequence (seq2seq) models with Transformers which encode the ordered requirements of the complex VNF-FG service graphs. The main aim of these techniques is to improve the combined representation of the current state environment, and help our actor-critic DRL agent learn the optimal policy that achieves a "one-shot" placement decision of all VNFs in the service graph, thereby improving placement efficiency and resource utilization, especially in large-scale systems. Our extensive simulation results show that our TDRL approach significantly outperforms other state-of-the-art baseline learning algorithms in terms of achieving the optimal balance between acceptance ratio and energy efficiency.