Performance Characterization of Spark Workloads on Shared NUMA Systems

被引：2

作者：

Baig, Shuja-ur-Rehman ^{[1
,2
,3
]}

Amaral, Marcelo ^{[1
,2
]}

Polo, Jorda ^{[2
]}

Carrera, David ^{[1
,2
]}

机构：

[1] Univ Politecn Catalunya UPC, Barcelona, Spain

[2] Barcelona Supercomp Ctr BSC, Barcelona, Spain

[3] Univ Punjab PU, Lahore, Pakistan

来源：

2018 IEEE FOURTH INTERNATIONAL CONFERENCE ON BIG DATA COMPUTING SERVICE AND APPLICATIONS (IEEE BIGDATASERVICE 2018) | 2018年

基金：

欧洲研究理事会;

关键词：

Performance; Modeling; Characterization; Memory; NUMA; Spark; Benchmark; AWARE DATA;

D O I：

10.1109/BigDataService.2018.00015

中图分类号：

TP [自动化技术、计算机技术];

学科分类号：

0812 ;

摘要：

As the adoption of Big Data technologies becomes the norm in an increasing number of scenarios, there is also a growing need to optimize them for modern processors. Spark has gained momentum over the last few years among companies looking for high performance solutions that can scale out across different cluster sizes. At the same time, modern processors can be connected to large amounts of physical memory, in the range of up to few terabytes. This opens an enormous range of opportunities for runtimes and applications that aim to improve their performance by leveraging low latencies and high bandwidth provided by RAM. The result is that there are several examples today of applications that have started pushing the in-memory computing paradigm to accelerate tasks. To deliver such a large physical memory capacity, hardware vendors have leveraged Non-Uniform Memory Architectures (NUMA). This paper explores how Spark-based workloads are impacted by the effects of NUMA-placement decisions, how different Spark configurations result in changes in delivered performance, how the characteristics of the applications can be used to predict workload collocation conflicts, and how to improve performance by collocating workloads in scale-up nodes. We explore several workloads run on top of the IBM Power8 processor, and provide manual strategies that can leverage performance improvements up to 40% on Spark workloads when using smart processor-pinning and workload collocation strategies.

引用

页码：41 / 48

页数：8

共 18 条

[11] Marketized state ownership and foreign expansion of emerging market multinationals: Leveraging institutional competitive advantages [J].

Li, Ming Hua ;

Cui, Lin ;

Lu, Jiangyong .

ASIA PACIFIC JOURNAL OF MANAGEMENT, 2017, 34 (01) :19-46

[12] Memory Management in NUMA Multicore Systems: Trapped between Cache Contention and Interconnect Overhead [J].

Majo, Zoltan ;

Gross, Thomas R. .

ACM SIGPLAN NOTICES, 2011, 46 (11) :11-32

[13]

Psaroudakis I, 2016, PROC VLDB ENDOW, V10, P37

[14]

Psaroudakis I, 2015, PROC VLDB ENDOW, V8, P1442

[15] Advanced features in IBM POWER8 systems [J].

Sinharoy, B. ;

Swanberg, R. ;

Nayar, N. ;

Mealey, B. ;

Stuecheli, J. ;

Schiefer, B. ;

Leenstra, J. ;

Jann, J. ;

Oehler, P. ;

Levitan, D. ;

Eisen, S. ;

Sanner, D. ;

Pflueger, T. ;

Lichtenau, C. ;

Hall, W. E. ;

Block, T. .

IBM JOURNAL OF RESEARCH AND DEVELOPMENT, 2015, 59 (01)

[16]

Zaharia M., 2010, SPARK CLUSTER COMPUT

[17]

Zaharia Matei., 2012, NSDI 12

[18]

Zhang KY, 2015, ACM SIGPLAN NOTICES, V50, P183, DOI [10.1145/2688500.2688507, 10.1145/2858788.2688507]

← 1 2 →