Optimizing Cloud MapReduce for Processing Stream Data using Pipelining

被引：5

作者：

Karve, Rutvik ^{[1
]}

Dahiphale, Devendra ^{[1
]}

Chhajer, Amit ^{[1
]}

机构：

[1] Pune Inst Comp Technol, Dept Comp Engn, Pune, Maharashtra, India

来源：

UKSIM FIFTH EUROPEAN MODELLING SYMPOSIUM ON COMPUTER MODELLING AND SIMULATION (EMS 2011) | 2011年

关键词：

MapReduce; Cloud Computing; Pipelining; Stream Processing; Distributed Computing;

D O I：

10.1109/EMS.2011.76

中图分类号：

TP39 [计算机的应用];

学科分类号：

081203 ; 0835 ;

摘要：

Cloud MapReduce (CMR) is a framework for processing large data sets of batch data in cloud. The Map and Reduce phases run sequentially, one after another. This leads to: 1. Compulsory batch processing 2. No parallelization of the map and reduce phases 3. Increased delays. The current implementation is not suited for processing streaming data. We propose a novel architecture to support streaming data as input using pipelining between the Map and Reduce phases in CMR, ensuring that the output of the Map phase is made available to the Reduce phase as soon as it is produced. This 'Pipelined MapReduce' approach leads to increased parallelism between the Map and Reduce phases; thereby 1. Supporting streaming data as input 2. Reducing delays 3. Enabling the user to take ' snapshots' of the approximate output generated in a stipulated time frame. 4. Supporting cascaded MapReduce jobs. This cloud implementation is light-weight and inherently scalable.

引用

页码：344 / 349

页数：6