The Impact of Synchronization in Parallel Stochastic Gradient Descent

被引：0
作者：
Backstrom, Karl [1 ]
Papatriantafilou, Marina [1 ]
Tsigas, Philippas [1 ]
机构：
[1] Chalmers Univ Technol, Dept Comp Sci & Engn, Gothenburg, Sweden
来源：
DISTRIBUTED COMPUTING AND INTELLIGENT TECHNOLOGY, ICDCIT 2022 | 2022年 / 13145卷
基金：
瑞典研究理事会;
关键词：
Stochastic gradient descent; Lock-free; Machine Learning;
D O I：
10.1007/978-3-030-94876-4_4
中图分类号：
TP31 [计算机软件];
学科分类号：
081202 ; 0835 ;
摘要：
In this paper, we discuss our and related work in the domain of efficient parallel optimization, using Stochastic Gradient Descent, for fast and stable convergence in prominent machine learning applications. We outline the results in the context of aspects and challenges regarding synchronization, consistency, staleness and parallel-aware adaptiveness, focusing on the impact on the overall convergence.
引用
收藏
页码：60 / 75
页数：16
相关论文
共 37 条
  • [1] Agarwal A, 2012, IEEE DECIS CONTR P, P5451, DOI 10.1109/CDC.2012.6426626
  • [2] Alistarh D., 2020, ARXIV PREPRINT ARXIV
  • [3] The Convergence of Stochastic Gradient Descent in Asynchronous Shared Memory
    Alistarh, Dan
    De Sa, Christopher
    Konstantinov, Nikola
    [J]. PODC'18: PROCEEDINGS OF THE 2018 ACM SYMPOSIUM ON PRINCIPLES OF DISTRIBUTED COMPUTING, 2018, : 169 - 177
  • [4] Alistarh D, 2017, ADV NEUR IN, V30
  • [5] Amdahl G.M., 1967, P APRIL 18 20 1967 S, P483, DOI DOI 10.1145/1465482.1465560
  • [6] [Anonymous], 1989, PARALLEL DISTRIBUTED
  • [7] [Anonymous], 2014, 11 USENIX S OPERATIN
  • [8] Consistent Lock-free Parallel Stochastic Gradient Descent for Fast and Stable Convergence
    Backstrom, Karl
    Walulya, Ivan
    Papatriantafilou, Marina
    Tsigas, Philippas
    [J]. 2021 IEEE 35TH INTERNATIONAL PARALLEL AND DISTRIBUTED PROCESSING SYMPOSIUM (IPDPS), 2021, : 423 - 432
  • [9] Bäckström K, 2019, IEEE INT CONF BIG DA, P16, DOI [10.1109/bigdata47090.2019.9006054, 10.1109/BigData47090.2019.9006054]
  • [10] Demystifying Parallel and Distributed Deep Learning: An In-depth Concurrency Analysis
    Ben-Nun, Tal
    Hoefler, Torsten
    [J]. ACM COMPUTING SURVEYS, 2019, 52 (04)
1234