Though autoregressive-moving average (ARMA) models are widely used in time series analysis, it is not suitable for modelling nonlinear systems. As neural networks has the attractive property of approximating linear or nonlinear functions with arbitrary accuracy, it is a good alternative to the ARMA models. In earlier works, neural networks with AR structure are proposed. In this paper, neural networks with ARMA structure is proposed, and an iterative procedure is devised to train the networks.