Atomicity of batches in stream processing

Stream processing is about processing continuous streams of data by programs in a workflow. Continuous execution is discretized by grouping input stream tuples into batches and using one batch at a time for the execution of programs. As source input batches arrive continuously, several batches may be processed in the workflow simultaneously. Ensuring correctness of these concurrent executions is important. We apply (database) transaction concept for the correctness. A general requirement is that each batch be processed completely in the workflow. That is, all the programs triggered by the batch, directly and transitively, in the workflow must be executed successfully. We say that a batch is executed atomically if it is processed completely, independently of the processing of other batches, and, if needed, the processing can be compensated without affecting the processing of other batches. The batches which can be executed atomically are called atomic batches. If batches are processed in isolation in the workflow, ensuring atomicity is fairly straightforward. However, when they are split, merged or overlapped along the workflow computation, ensuring atomicity becomes complicated. In some cases, several source input batches can be combined to form an atomic batch. In some other cases, execution can be prompted to yield atomic batches. In this paper, we study these issues.