Though Widespread Interest in software containers is a relatively recent phenomenon, at Google we have been managing Linux containers at scale for more than 10 years and built three different containermanagement systems in that time. Each system is heavily influenced by its predecessors, even though they were developed for different reasons. This article describes the lessons we've learned from developing and operating them. The first unified container-management system developed at Google was the system we internally call Borg.7 It was built to manage both long-running services and batch jobs, which had previously been handled by two separate systems: Babysitter and the Global Work Queue. The latter's architecture strongly influenced Borg, but was focused on batch jobs; both predated Linux control groups. Borg shares machines between these two types of applications as a way of increasing resource utilization and thereby reducing costs. Such sharing was possible because container support in the Linux kernel was becoming available (indeed, Google contributed much of the container code to the Linux kernel), which enabled better isolation between latency-sensitive user-facing services and CPU-hungry batch processes. © 2016 ACM.
