GeePS: Scalable deep learning on distributed GPUs with a GPU-specialized parameter server

被引：202

作者：

Cui, Henggang ^{[1
]}

Zhang, Hao ^{[1
]}

Ganger, Gregory R. ^{[1
]}

Gibbons, Phillip B. ^{[1
]}

Xing, Eric P. ^{[1
]}

机构：

[1] Carnegie Mellon Univ, Pittsburgh, PA 15213 USA

来源：

PROCEEDINGS OF THE ELEVENTH EUROPEAN CONFERENCE ON COMPUTER SYSTEMS, (EUROSYS 2016) | 2016年

基金：

美国国家科学基金会;

关键词：

D O I：

10.1145/2901318.2901323

中图分类号：

TP301 [理论、方法];

学科分类号：

081202 ;

摘要：

Large-scale deep learning requires huge computational resources to train a multi-layer neural network. Recent systems propose using 100s to 1000s of machines to train networks with tens of layers and billions of connections. While the computation involved can be done more efficiently on GPUs than on more traditional CPU cores, training such networks on a single GPU is too slow and training on distributed GPUs can be inefficient, due to data movement overheads, GPU stalls, and limited GPU memory. This paper describes a new parameter server, called GeePS, that supports scalable deep learning across GPUs distributed among multiple machines, overcoming these obstacles. We show that GeePS enables a state-of-the-art single-node GPU implementation to scale well, such as to 13 times the number of training images processed per second on 16 machines (relative to the original optimized single-node code). Moreover, GeePS achieves a higher training throughput with just four GPU machines than that a state-of-the-art CPU-only system achieves with 108 machines.

引用

页数：16