Deterministic convergence of an online gradient method for neural networks

The online gradient method has been widely used as a learning algorithm for neural networks. We establish a deterministic convergence of online gradient methods for the training of a class of nonlinear feedforward neural networks when the training examples are linearly independent. We choose the learning rate eta to be a constant during the training procedure. The monotonicity of the error function in the iteration is proved. A criterion for choosing the learning rate eta is also provided to guarantee the convergence. Under certain conditions similar to those for the classical gradient methods, an optimal convergence rate for our online gradient methods is proved. (C) 2001 Elsevier Science B.V. All rights reserved.