In-memory big data analytics under space constraints using dynamic programming

The emergence of persistent memories has powered the data processing with the in-memory environment and in-memory data analytics have become an advance of high-performance data processing. Recent explorations of using in-memory technologies address the improvement of the memory performance from re-designing file systems. Most current approaches mitigate data exchanges between buffers and disks by migrating workload to memories. However, this type of solutions will be encountering the restriction of the memory size with the rapid growth of the application volume. This paper focuses on the issue caused by the large amount of data processing within in-memory systems and proposes a novel approach that is designed to dynamically determine whether the data processing should be accomplished in the memory. The proposed approach is called Smart In-Memory Data Analytics Manager (SIM-DAM) model, which utilizes a dynamic working manner of the file system, as well as fully uses hardware mappings. The experimental results obtained from our laboratory evaluations represent that the throughputs of SIM-DAM can achieve a high-level performance with different input data sizes without the constraints of the memories' spaces. (C) 2018 Elsevier B.V. All rights reserved.