The Global Landscape of Neural Networks: An Overview

被引：59

作者：

Sun, Ruoyu ^{[1
,2
,3
,4
]}

Li, Dawei ^{[1
]}

Liang, Shiyu ^{[5
]}

Ding, Tian ^{[6
]}

Srikant, Rayadurgam ^{[2
,3
]}

机构：

[1] Univ Illinois Urbana Champaign UIUC, Dept Ind & Enterprise Syst Engn, Champaign, IL 61820 USA

[2] Univ Illinois Urbana Champaign UIUC, Coordinated Sci Lab, Champaign, IL 61820 USA

[3] Univ Illinois Urbana Champaign UIUC, Dept Elect & Comp Engn, Champaign, IL 61820 USA

[4] Stanford Univ, Stanford, CA 94305 USA

[5] Univ Illinois Urbana Champaign UIUC, Champaign, IL USA

[6] Chinese Univ Hong Kong, Hong Kong, Peoples R China

来源：

IEEE SIGNAL PROCESSING MAGAZINE | 2020年 / 37卷 / 05期

关键词：

Optimization; Signal processing algorithms; Training; Biological neural networks; Convergence; Machine learning;

D O I：

10.1109/MSP.2020.3004124

中图分类号：

TM [电工技术]; TN [电子技术、通信技术];

学科分类号：

0808 ; 0809 ;

摘要：

One of the major concerns for neural network training is that the nonconvexity of the associated loss functions may cause a bad landscape. The recent success of neural networks suggests that their loss landscape is not too bad, but what specific results do we know about the landscape? In this article, we review recent findings and results on the global landscape of neural networks.

引用

页码：95 / 108

页数：14

共 52 条

[1] Allen-Zhu Z, 2019, Advances in neural information processing systems, P6673
[2] Allen-Zhu Z, 2019, PR MACH LEARN RES, V97
[3] Anandkumar Animashree, 2016, C LEARNING THEORY, P81
[4] [Anonymous], 2017, ICLR 2017 C TRACK OP
[5] Arora S, 2019, 33 C NEURAL INFORM P, V32
[6] Arora S, 2019, PR MACH LEARN RES, V97
[7] Shaping the learning landscape in neural networks around wide flat minima
Baldassi, Carlo
Pittorino, Fabrizio
Zecchina, Riccardo
[J]. PROCEEDINGS OF THE NATIONAL ACADEMY OF SCIENCES OF THE UNITED STATES OF AMERICA, 2020, 117 (01) : 161 - 170
[8] NEURAL NETWORKS AND PRINCIPAL COMPONENT ANALYSIS - LEARNING FROM EXAMPLES WITHOUT LOCAL MINIMA
BALDI, P
HORNIK, K
[J]. NEURAL NETWORKS, 1989, 2 (01) : 53 - 58
[9] Optimal learning in artificial neural networks: A review of theoretical results
Bianchini, M
Gori, M
[J]. NEUROCOMPUTING, 1996, 13 (2-4) : 313 - 346
[10] A nonlinear programming algorithm for solving semidefinite programs via low-rank factorization
Burer, S
Monteiro, RDC
[J]. MATHEMATICAL PROGRAMMING, 2003, 95 (02) : 329 - 357

← 1 2 3 4 5 6 →