A Hierarchical Multi-Action Deep Reinforcement Learning Method for Dynamic Distributed Job-Shop Scheduling Problem With Job Arrivals

The Distributed Job-shop Scheduling Problem (DJSP) is a significant issue in both academic and industrial fields. In real-world production, uncertain disturbances such as job arrivals are inevitable. In the paper, the DJSP with job arrivals is addressed with a Multi-action Deep Reinforcement Learning (MDRL) method. Firstly, a multi-action Markov Decision Process (MDP) is formulated, where a hierarchical multi-action space combining operation set and factory set is proposed. The reward function is related to the machine idle time. Additionally, the state transition is also elaborately designed, which includes four typical cases based on job arrival times. Then, a scheduling policy with two decision networks is proposed, where the Graph Neural Network (GNN) is applied to extract the intrinsic information of the scheduling scheme. A Proximal Policy Optimization (PPO) with two actor-critic frameworks is designed to train the model to achieve intelligent decision-making with hierarchical action selections. Extensive experiments are conducted based on 1350 instances. The comparison among 17 composite rules, 3 closely-rated DRL methods, and 2 metaheuristics has proven the outperformance of the proposed MDRL. The application of the MDRL in an automotive engine manufacturing company has demonstrated its engineering value in the industrial field. Note to Practitioners-The DJSP with job arrivals is a common challenge faced by equipment manufacturers, specifically in the electronic device manufacturing industry. These manufacturers are located in different areas and have varying facility configurations and operation trajectories. To address this challenge, a machine learning-based method can be applied for scheduling daily production tasks. This method divides the DJSP into two subproblems, namely job assigning and job sequencing, and uses two decision networks based on DRL to solve them. To address the uncertainty caused by job arrivals, the rescheduling process and the state update mechanism are carefully designed. A GNN is used for feature extraction at each decision point, and it feeds the decision networks with the extracted features to make the optimal selection. The proposed method has the ability of self-learning and self-adapting, and its effectiveness has been proven through experiments on 1350 test instances. Its practical application has been demonstrated in the production scenarios of an automotive engine manufacturing company. In the future, the method can be adopted to solve more complex distributed manufacturing problems that have constraints such as transportation costs and machine breakdowns.