A joint learning Im-BiLSTM model for incomplete time-series Sentinel-2A data imputation and crop classification

Multi-temporal deep learning approaches can make full use of crop growth patterns and phenological characteristics, resulting in excellent crop classification performance in large areas. However, obtaining complete time series remote sensing images during the growing season is challenging due to cloud contamination. Hence, given time-series multispectral data, it is important to impute missing data and accurately classify crops. A novel Imputation-BiLSTM model (Im-BiLSTM) was developed based on Bidirectional Long Short-term Memory network (BiLSTM) to jointly perform missing data imputation and crop classification. The Im-BiLSTM model regards missing data as network variables, which are efficiently updated during backpropagation. Im-BiLSTM treats the interaction between imputation and classification tasks, reducing the error and uncertainty caused by the separation operation of imputation to classification. Furthermore, we improved the interpretability of the ImBiLSTM model by evaluating the importance of input features and visualizing hidden state units. In Shawan County, Xinjiang, China, we acquired a total of 10 Sentinel-2A images from April to October 2016, of which 3 images lost partial data due to cloud cover. The Im-BiLSTM model was applied to incomplete time-series data containing 10 time-steps for pixel-level crop classification, and the BiLSTM model was constructed based on cloud-free images for comparison. The performance of the proposed model was tested in four different cases of images missing and missing rates. The results showed that the classification of the Im-BiLSTM model outperformed the BiLSTM model, the overall accuracy was improved by a maximum of 4.2%, and the F1-scores of spring corn and tomato was improved by a maximum of 16.1% and 21.4%, respectively. Therefore, the ImBiLSTM model can effectively improve classification performance by jointly imputing missing data. The imputation results (the coefficient of determination values range 0.4 ~ 0.9) indicated that the bands with the larger contribution to the classification had higher imputation accuracy. Feature importance evaluation showed that the Im-BiLSTM model captured the key phenological periods and features from time-series input. The visualization of hidden units demonstrated that the Im-BiLSTM model accumulated useful information over time, and the learned high-level features made the crops more separable than the original inputs.