Boundedness and Convergence of Online Gradient Method With Penalty for Feedforward Neural Networks

被引：61

作者：

Zhang, Huisheng ^{[1
,2
]}

Wu, Wei ^{[1
]}

Liu, Fei ^{[3
]}

Yao, Mingchen ^{[1
]}

机构：

[1] Dalian Univ Technol, Dept Appl Math, Dalian 116023, Peoples R China

[2] Dalian Maritime Univ, Dept Math, Dalian 116026, Peoples R China

[3] Univ Missouri, Dept Stat, Columbia, MO 65211 USA

来源：

IEEE TRANSACTIONS ON NEURAL NETWORKS | 2009年 / 20卷 / 06期

基金：

中国国家自然科学基金; 美国国家科学基金会;

关键词：

Boundedness; convergence; feedforward neural networks; online gradient method; penalty; ALGORITHMS;

D O I：

10.1109/TNN.2009.2020848

中图分类号：

TP18 [人工智能理论];

学科分类号：

081104 ; 0812 ; 0835 ; 1405 ;

摘要：

In this brief, we consider an online gradient method with penalty for training feedforward neural networks. Specifically, the penalty is a term proportional to the norm of the weights. Its roles in the method are to control the magnitude of the weights and to improve the generalization performance of the network. By proving that the weights are automatically bounded in the network training with penalty, we simplify the conditions that are required for convergence of online gradient method in literature. A numerical example is given to support the theoretical analysis.

引用

页码：1050 / 1054

页数：5

共 21 条

[1]

[Anonymous], WSEAS T MATH

[2]

[Anonymous], 1979, Wiley Series in Probability and Mathematical Statistics

[3]

[Anonymous], 1986, CMUCS86126

[4]

Bertsekas Dimitri, 1996, Neuro dynamic programming

[5] Gradient convergence in gradient methods with errors [J].

Bertsekas, DP ;

Tsitsiklis, JN .

SIAM JOURNAL ON OPTIMIZATION, 2000, 10 (03) :627-642

[6] Regularization networks and support vector machines [J].

Evgeniou, T ;

Pontil, M ;

Poggio, T .

ADVANCES IN COMPUTATIONAL MATHEMATICS, 2000, 13 (01) :1-50

[7] Parameter convergence and learning curves for neural networks [J].

Fine, TL ;

Mukherjee, S .

NEURAL COMPUTATION, 1999, 11 (03) :747-769

[8]

Gaivoronski A.A., 1994, Optim. Methods Softw., V4, P117, DOI 10.1080/10556789408805582

[9] Convergent on-line algorithms for supervised learning in neural networks [J].

Grippo, L .

IEEE TRANSACTIONS ON NEURAL NETWORKS, 2000, 11 (06) :1284-1299

[10]

HANSON S., 1989, Advances in Neural Information Processing I, P177

← 1 2 3 →