Predicting the Stability of Large-scale Distributed Stream Processing Systems on the Cloud

Large-scale topology-based stream processing systems are non-trivial to build and deploy. They require understanding of the performance, cost of deployment and considerations of potential downtime. Our work considers stability as a primary characteristic of these systems. By stability, we mean that unstable systems exhibit large-spikes in latency and can drop throughput frequently or unpredictably. Such instabilities can be due to variations of workloads or underlying hardware platforms that are often difficult to predict. To understand and tackle this for large-scale stream processing systems, we apply queueing theory and simulate the results through a series of experiments on the Cloud.