MUSTFN: A spatiotemporal fusion method for multi-scale and multi-sensor remote sensing images based on a convolutional neural network

被引:17
作者
Qin, Peng [1 ,2 ]
Huang, Huabing [1 ,2 ,3 ,4 ]
Tang, Hailong [1 ,2 ]
Wang, Jie [3 ]
Liu, Chong [1 ,2 ]
机构
[1] Sun Yat Sen Univ, Sch Geospatial Engn & Sci, Zhuhai 519082, Peoples R China
[2] Southern Marine Sci & Engn Guangdong Lab Zhuhai, Zhuhai 519082, Peoples R China
[3] Peng Cheng Lab, Shenzhen 518066, Peoples R China
[4] Int Res Ctr Big Data Sustainable Dev Goals, Beijing, Peoples R China
基金
国家重点研发计划;
关键词
Spatiotemporal fusion; CNN; Multi-sensor satellite data; Large-area image fusion; Multi-scale fusion scenarios; SURFACE REFLECTANCE; CLOUD REMOVAL; SATELLITE IMAGES; LANDSAT; MODIS; SERIES; INDEX;
D O I
10.1016/j.jag.2022.103113
中图分类号
TP7 [遥感技术];
学科分类号
081102 ; 0816 ; 081602 ; 083002 ; 1404 ;
摘要
Spatiotemporal data fusion is a commonly-used and well-proven technique to enhance the application potential of multi-source remote sensing images. However, most existing methods have trouble in generating quality fusion results when areas covered by the images undergoes rapid land cover changes or images have substantial registration errors. While deep learning algorithms have demonstrated their capabilities for imagery fusion, it is challenging to apply deep-learning-based fusion methods in regions that experiences persistent cloud covers and have limited cloud-free imagery observations. To address these challenges, we developed a Multi-scene Spatiotemporal Fusion Network (MUSTFN) algorithm based on a Convolutional Neural Network (CNN). Our approach uses multi-level features to fuse images at different resolutions acquired by multiple sensors. Furthermore, MUSTFN uses the multi-scale features to overcome the effects of geometric registration errors between different images. Additionally, a multi-constrained loss function is proposed to improve the accuracy of imagery fusion over large areas and solve fusion and gap-filling problems simultaneously by utilizing cloud-contaminated images with the fine-tuning method. Compared with several commonly-used methods, our pro-posed MUSTFN performs better in fusing the 30-m Landsat-7 images and 500-m MODIS images over a small area that has undergone large changes (the average relative Mean Absolute Errors (rMAE) of the first four bands are 6.8% by MUSTFN as compared to 14.1% by the Enhanced Spatial and Temporal Adaptive Reflectance Fusion Model (ESTARFM), 12.8% by the Flexible Spatiotemporal Data Fusion (FSDAF), 8.4% by the Extended Super-Resolution Convolutional Neural Network (ESRCNN), 8.1% by the Spatiotemporal Fusion Using a Generative Adversarial Network (STFGAN)). In particularly for images at different resolutions with different registration accuracies (e.g., 16-m Chinese GaoFen-1 and 500-m MODIS), MUSTFN achieved fusion results of good quality with an average rMAE of 9.3% in spectral reflectance at the first four bands. Finally, we demonstrated the applicability of MUSTFN (average rMAE of 9.18%) when fusing long-term Landsat-8 composite images and MODIS images over a large region (830 km x 600 km). Overall, our results suggest the effectiveness of MUSTFN to address the challenges in imagery fusion, including rapid land cover changes between image acquisition dates, geometric misregistration between images and limited availabilities of cloud-free images. The program of MUSTFN is freely available at: https://github.com/qpyeah/MUSTFN.
引用
收藏
页数:16
相关论文
共 50 条
[31]   Estimation of multi-scale urban vegetation coverage based on multi-source remote sensing images [J].
Gao Yong-Gang ;
Xu Han-Qiu .
JOURNAL OF INFRARED AND MILLIMETER WAVES, 2017, 36 (02) :225-234
[32]   Graph convolutional neural network for multi-scale feature learning [J].
Edwards, Michael ;
Xie, Xianghua ;
Palmer, Robert, I ;
Tam, Gary K. L. ;
Alcock, Rob ;
Roobottom, Carl .
COMPUTER VISION AND IMAGE UNDERSTANDING, 2020, 194
[33]   A Spatiotemporal Fusion Based Cloud Removal Method for Remote Sensing Images With Land Cover Changes [J].
Shen, Huanfeng ;
Wu, Jingan ;
Cheng, Qing ;
Aihemaiti, Mahemujiang ;
Zhang, Chengyue ;
Li, Zhiwei .
IEEE JOURNAL OF SELECTED TOPICS IN APPLIED EARTH OBSERVATIONS AND REMOTE SENSING, 2019, 12 (03) :862-874
[34]   A Multi-Scale Fusion Convolutional Neural Network Based on Attention Mechanism for the Visualization Analysis of EEG Signals Decoding [J].
Li, Donglin ;
Xu, Jiacan ;
Wang, Jianhui ;
Fang, Xiaoke ;
Ji, Ying .
IEEE TRANSACTIONS ON NEURAL SYSTEMS AND REHABILITATION ENGINEERING, 2020, 28 (12) :2615-2626
[35]   Multi-Scale Bilateral Spatial Direction-Aware Network for Cropland Extraction Based on Remote Sensing Images [J].
Hou, Weimin ;
Wang, Yanxia ;
Su, Jia ;
Hou, Yanli ;
Zhang, Ming ;
Shang, Yan .
IEEE ACCESS, 2023, 11 :109997-110009
[36]   Building Change Detection in Remote Sensing Images Based on Dual Multi-Scale Attention [J].
Zhang, Jian ;
Pan, Bin ;
Zhang, Yu ;
Liu, Zhangle ;
Zheng, Xin .
REMOTE SENSING, 2022, 14 (21)
[37]   Spatiotemporal Fusion Model of Remote Sensing Images Combining Single-Band and Multi-Band Prediction [J].
Wang, Zhiyuan ;
Fang, Shuai ;
Zhang, Jing .
REMOTE SENSING, 2023, 15 (20)
[38]   Retrieval of grassland plant coverage on the Tibetan Plateau based on a multi-scale, multi-sensor and multi-method approach [J].
Lehnert, Lukas W. ;
Meyer, Hanna ;
Wang, Yun ;
Miehe, Georg ;
Thies, Boris ;
Reudenbach, Christoph ;
Bendix, Joerg .
REMOTE SENSING OF ENVIRONMENT, 2015, 164 :197-207
[39]   Explicit and stepwise models for spatiotemporal fusion of remote sensing images with deep neural networks [J].
Ma, Yaobin ;
Wei, Jingbo ;
Tang, Wenchao ;
Tang, Rongxin .
INTERNATIONAL JOURNAL OF APPLIED EARTH OBSERVATION AND GEOINFORMATION, 2021, 105
[40]   Multi-Scale Convolutional Neural Network-Based Intra Prediction for Video Coding [J].
Wang, Yang ;
Fan, Xiaopeng ;
Liu, Shaohui ;
Zhao, Debin ;
Gao, Wen .
IEEE TRANSACTIONS ON CIRCUITS AND SYSTEMS FOR VIDEO TECHNOLOGY, 2020, 30 (07) :1803-1815