Static and Dynamic Big Data Partitioning on Apache Spark

被引：10

作者：

Bertolucci, Massimiliano ^{[2
]}

Carlini, Emanuele ^{[1
]}

Dazzi, Patrizio ^{[1
]}

Lulli, Alessandro ^{[1
,2
]}

Ricci, Laura ^{[1
,2
]}

机构：

[1] CNR, Ist Sci & Tecnol Informaz, Pisa, Italy

[2] Univ Pisa, Dept Comp Sci, Pisa, Italy

来源：

PARALLEL COMPUTING: ON THE ROAD TO EXASCALE | 2016年 / 27卷

关键词：

BigData; Graph algorithms; Data partitioning; Apache Spark;

D O I：

10.3233/978-1-61499-621-7-489

中图分类号：

TP3 [计算技术、计算机技术];

学科分类号：

0812 ;

摘要：

Many of today's large datasets are organized as a graph. Due to their size it is often infeasible to process these graphs using a single machine. Therefore, many software frameworks and tools have been proposed to process graph on top of distributed infrastructures. This software is often bundled with generic data decomposition strategies that are not optimised for specific algorithms. In this paper we study how a specific data partitioning strategy affects the performances of graph algorithms executing on Apache Spark. To this end, we implemented different graph algorithms and we compared their performances using a naive partitioning solution against more elaborate strategies, both static and dynamic.

引用

页码：489 / 498

页数：10

共 28 条

[11]

Danelutto Marco, 2005, JAVA JINI FRAMEWORK

[12]

Jeffrey Dean, 2008, COMMUNUNICATION ACM, V1

[13] The Impact of a Wildland Fire on Air Pollution Concentrations UsingWRF/Chem/Fire: An Application over Murcia (Spain) [J].

Jose, Roberto San ;

Perez, Juan Luis ;

Gonzalez, R. M. ;

Pecci, J. ;

Palacios, M. .

AIR POLLUTION MODELING AND ITS APPLICATION XXIII, 2014, :599-603

[14]

Karypis G., 1995, Proceedings of the 1995 International Conference on Parallel Processing, P113

[15]

Kuchen H, 2002, SKELETON LIB

[16]

Leskovec J., 2014, SNAP GEN PURPOSE NET

[17] Skandium: Multi-core Programming with Algorithmic Skeletons [J].

Leyton, Mario ;

Piquer, Jose M. .

PROCEEDINGS OF THE 18TH EUROMICRO CONFERENCE ON PARALLEL, DISTRIBUTED AND NETWORK-BASED PROCESSING, 2010, :289-296

[18]

Lulli A., 2015, 20 IEEE S COMP COMM

[19]

McAfee A, 2012, HARVARD BUS REV, V90, P60

[20]

McCune Robert Ryan, THINKING VERTEX SURV

← 1 2 3 →