Container-based data-intensive application scheduling in hybrid cloud-edge collaborative environment

Container virtualization technology represented by Docker has been widely used in the industry due to its advantages of lightweight, fast deployment, and easy portability. It can bring convenience to system deployment, operation, and maintenance. This article considers the implementation of AI-based IoT applications based on Docker and container technology in a cloud-edge collaborative environment. Three typical data-intensive applications, preprocessing, training, and inference in machine learning, are deployed using Docker containers. For these three types of tasks, this article proposes a container-based data-intensive application scheduling framework. Using an attribute-based data-driven method, a new scheduling approach considering multiple weighting factors is proposed, which is called DICS-OPT. The cloud nodes and edge nodes are scored and sorted uniformly. The container is scheduled to the node with the highest score, and then the task data is transmitted to the node to execute the task. NSGA-III-based genetic algorithm has been proposed to search for the optimal weighting factors. This article introduces the implementation of the prototype system and builds a cloud-edge collaborative testbed consisting of Raspberry Pis and PCs. The performance evaluation results indicates that the proposed scheduling approach outperforms existing approaches. Compared with existing approaches, DICS-OPT improves the average edge resource utilization by 10.05% to 69.04%, and saves the cloud resource cost by 16.02% to 36.68%.