Learning long-term dependencies by the selective addition of time-delayed connections to recurrent neural networks

Recurrent neural networks possess interesting universal approximation capabilities, making them good candidates for time-series modeling. Unfortunately, long-term dependencies are difficult to learn if gradient descent algorithms are employed. We support the view that it is easier for these algorithms to find good solutions if time-delayed connections are included in the recurrent networks. The algorithm we present here allows one to choose the right locations and delays for such connections. As we show on several univariate benchmarks and one multivariate real-world problem, this algorithm produces very good results while keeping the total number of connections in the recurrent network to a minimum. (C) 2002 Elsevier Science B.V. All rights reserved.