NewTL: Engineering an Extract, Transform, Load (ETL) Software System for Business on a Very Large Scale

Large-scale ETL (Extract, Transform, Load) Software Systems are very complex to build, and even harder to maintain, due to the many moving pieces, and varying ownership and responsibilities, across multiple parties. While some research has been done in the area of ETL, it primarily focuses on techniques for optimization of expensive operations, and identifying common challenges; but sheds little light on how to build a reliable and robust system in practice. In this paper, we discuss NewTL, an ETL software system that was built completely in-house at Varidesk Inc., that allows for ETL on a very large scale. and has facilitated seamless integration between our 3rd Party Logistics Providers (3PLs) and our ERP (Enterprise Resource Planning) system. In doing so, we provide-technical insights on real-world concerns such as speed and scalability; transparency on our decision to build versus buy; and touch upon other aspects that would be relevant to any ETL system in practice. We also provide metrics/real data (collected both directly by us, as well as through telemetry from external monitoring), that demonstrate the quality of NewTL. We find this to be the first study of its kind on such a scale, that serves to help other practitioners, as well as stimulate further research into this highly important field.