Complexity issues in natural gradient descent method for training multilayer perceptrons

被引：53

作者：

Yang, HH ^{[1
]}

Amari, S

机构：

[1] Oregon Grad Inst, Dept Comp Sci, Portland, OR 97291 USA

[2] RIKEN, Brain Sci Inst, Lab Informat Synth, Wako, Saitama 35101, Japan

来源：

NEURAL COMPUTATION | 1998年 / 10卷 / 08期

关键词：

D O I：

10.1162/089976698300017007

中图分类号：

TP18 [人工智能理论];

学科分类号：

081104 ; 0812 ; 0835 ; 1405 ;

摘要：

The natural gradient descent method is applied to train an n-m-1 multilayer perceptron. Based on an efficient scheme to represent the Fisher information matrix for an n-m-1 stochastic multilayer perceptron, a new algorithm is proposed to calculate the natural gradient without inverting the Fisher information matrix explicitly. When the input dimension n is much larger than the number of hidden neurons m, the time complexity of computing the natural gradient is O(n).

引用

页码：2137 / 2157

页数：21

共 9 条

[1] Natural gradient works efficiently in learning
Amari, S
[J]. NEURAL COMPUTATION, 1998, 10 (02) : 251 - 276
[2] Amari S, 1997, ADV NEUR IN, V9, P127
[3] ORR GB, 1997, ADV NEURAL INFORMATI, V9
[4] ONLINE LEARNING IN SOFT COMMITTEE MACHINES
SAAD, D
SOLLA, SA
[J]. PHYSICAL REVIEW E, 1995, 52 (04): : 4225 - 4243
[5] Stewart GW., 1973, INTRO MATRIX COMPUTA
[6] Statistical inference: learning in artificial neural networks
Yang, HH
Murata, N
Amari, S
[J]. TRENDS IN COGNITIVE SCIENCES, 1998, 2 (01) : 4 - 10
[7] YANG HH, 1997, ICONIP 97 P NZ
[8] YANG HH, 1997, UNPUB NATURAL GRADIE
[9] YANG HH, 1998, ADV NEURAL INFORMATI, V10

← 1 →