Measuring Streaming System Robustness Using Non-parametric Goodness-of-Fit Tests

Due to unpredictable disturbances in the operating environment, stream processing systems may experience performance degradation and even catastrophic failure. Streaming systems must be robust in the face of such uncertainty in order to be deemed fit for purpose. Measuring and quantifying a system's level of robustness is a non-trivial task. We present, compare and contrast a range of non-parametric goodness-of-fit tests which can act as quantifiers of a system's level of robustness. We show that different tests produce differing relative measures of system robustness, affected by not only the test statistics inherent characteristics, but also by the particular latency percentile under scrutiny.