StreamStorage: High-throughput and scalable storage technology for streaming data

Real-time analytical processing (RTAP) of vast amounts of time-series data from sensors, server logs, and other sources has come to be widely used in recent years. To make thorough use of such streaming data, it is essential that the same data be analyzed iteratively from diverse viewpoints, which has increased the need for loss-free storage and data reconstruction functions for large volumes of data. Since individual items of data targeted for storage are generally small in size and repeatedly received from multiple sources, it is difficult for existing storage systems to simultaneously satisfy throughput and capacity demands. Additionally, high-speed access optimized for the chronologically ordered data characteristic of streaming data has not yet been achieved for such vast amounts of data. StreamStorage developed by Fujitsu Laboratories is a new storage technology that manages time-series data in units of streams, enabling high-speed storage and data reconstruction. It achieves high throughput and scalability by using distributed storage technology in which streaming data is partitioned into blocks and data is input and output in a parallel and asynchronous process. This paper presents an overview of the StreamStorage architecture and an application example.