BERT-LSTM network prediction model based on Transformer

A network model based on Transformer encoding by using BERT and decoding by using long short-term memory (LSTM) architecture is designed to predict multivariable systems. The BERT-LSTM network model with feature compensation block uses a sliding window to process data. The BERT network is used as the encoder block to achieve the capture of long-distance position features of time series data. The feature compensation block is composed of the CNN-LSTM model in the binary tree framework, which rectify the BERT network's inability to capture the global long-term dependence between time series data. The decoding characteristics of LSTM are used to predict the global long-term dependence features with distance and position information, in order to realize the prediction function of the network. Finally, by compared with the existing network models, the superiority of the proposed network model is verified.