Time-delay neural networks, Volterra series, and rates of approximation

We consider a large family of approximately-finite-memory causal time-invariant maps G from an input set S to a set of R-valued functions, with the members of both sets of functions defined on the nonnegative integers, and we give an upper bound on the error in approximating a G using a two-stage structure consisting of a tapped delay line followed by a static neural network. As an application, information is given concerning the long-standing problem of determining the order of a Volterra-series approximation so that a given quality of approximation can be achieved. We also give corresponding results for the approximation of not necessarily causal input-output maps with inputs and outputs that may depend on more than one variable. These results are of interest, for example, in connection with image processing.