The third party logistics provider freight management problem: a framework and deep reinforcement learning approach

In many large manufacturing companies, freight management is handled by a third-party logistics (3PL) provider, thus allowing manufacturers and their suppliers to focus on the production of goods rather than managing their delivery. Provided their pivotal supply chain role, in this work we propose a general framework for what we term as "the 3PL freight management problem" (3PLFMP). Our framework identifies three primary activities involved in 3PL freight management: the assignment of orders to a fleet of vehicles, efficient routing of the fleet, and packing the assigned orders in vehicles. Furthermore, we provide a specific instantiation of the 3PLFMP that considers direct vs. consolidated shipping strategies, one dimensional packing constraints, and a fixed vehicle routing schedule. We solve this instantiated problem using several Reinforcement Learning (RL) methods, including Q-learning, Double Q-learning, SARSA, Deep Q-learning, and Double Deep Q-learning, comparing against two benchmark methods, a simulated annealing heuristic and a variable neighborhood descent algorithm. We evaluate the performance of these methods on two datasets. One is fully simulated and based on past work, while another is semi-simulated using real-world automobile manufacturers and part supplier locations, and is of our own design. We find that RL methods vastly outperform the benchmark heuristic methods on both datasets, thus establishing the superiority of RL methods in solving this highly complicated and stochastic problem.