Ensemble echo network with deep architecture for time-series modeling

Echo state network belongs to a kind of recurrent neural networks that have been extensively employed to model time-series datasets. The function of reservoir in echo state network is expected to extract the feature context from time-series datasets. However, generalization of echo state networks is limited in real-world application because the architectures of the network are fixed and the hyper-parameters are hard to be automatically determined. In the present study, the ensemble Bayesian deep echo network (EBDEN) model with deep and flexible architecture is proposed. Such networks with deep architecture progressively extract more dynamic echo states through multiple reservoirs than those with the shallow reservoir. To enhance the flexibility of the configuration for the network, this study investigates the Bayesian optimization procedure of hyper-parameters and ensures the suitable hyper-parameters to activate the network. In addition, when dealing with more complex time-series datasets, ensemble mechanism of EBDEN can measure the redundancy for the channels of the time series without sacrificing the algorithm's performance. In this paper, the deep, optimization and ensemble architectures of EBDEN are verified by experiments benchmarked on multivariate time-series repositories and realistic tasks such as chaotic series representation and Dansgaard-Oeschger estimation tasks. According to the results, EBDEN achieves high level of the goodness-of-fit and classification performance in comparison with state-of-the-art models.