The Simple Essence of Automatic Differentiation

被引:52
作者
Elliott, Conal [1 ]
机构
[1] Target, San Andreas, CA USA
来源
PROCEEDINGS OF THE ACM ON PROGRAMMING LANGUAGES | 2018年
关键词
automatic differentiation; program calculation; category theory;
D O I
10.1145/3236765
中图分类号
TP31 [计算机软件];
学科分类号
081202 ; 0835 ;
摘要
Automatic differentiation (AD) in reverse mode (RAD) is a central component of deep learning and other uses of large-scale optimization. Commonly used RAD algorithms such as backpropagation, however, are complex and stateful, hindering deep understanding, improvement, and parallel execution. This paper develops a simple, generalized AD algorithm calculated from a simple, natural specification. The general algorithm is then specialized by varying the representation of derivatives. In particular, applying well-known constructions to a naive representation yields two RAD algorithms that are far simpler than previously known. In contrast to commonly used RAD implementations, the algorithms defined here involve no graphs, tapes, variables, partial derivatives, or mutation. They are inherently parallel-friendly, correct by construction, and usable directly from an existing programming language with no need for new data types or programming style, thanks to use of an AD-agnostic compiler plugin.
引用
收藏
页数:29
相关论文
共 56 条
  • [1] Abadi M, 2016, PROCEEDINGS OF OSDI'16: 12TH USENIX SYMPOSIUM ON OPERATING SYSTEMS DESIGN AND IMPLEMENTATION, P265
  • [2] [Anonymous], 1989, Mathematical Programming: recent developments and applications
  • [3] [Anonymous], 1980, THESIS
  • [4] [Anonymous], 2008, EVALUATING DERIVATIV
  • [5] Appel A., 2007, COMPILING CONTINUATI
  • [6] Awodey S., 2006, Oxford Logic Guides, V49
  • [7] Bird Richard S., 1996, Algebra of Programming
  • [8] Bolingbroke M., 2011, CONSTRAINT KINDS GHC
  • [9] Chollet F., 2016, GITHUB REPOSITORY
  • [10] Cormen T. H., 2001, Introduction to Algorithms, V2nd