Two-stream spatiotemporal image fusion network based on difference transformation

被引:0
|
作者
Fang, Shuai [1 ,2 ]
Meng, Siyuan [1 ]
Zhang, Jing [1 ]
Cao, Yang [3 ,4 ]
机构
[1] Hefei Univ Technol, Key Lab Knowledge Engn Big Data, Hefei, Peoples R China
[2] Anhui Prov Key Lab Ind Safety & Emergency Technol, Hefei, Peoples R China
[3] Univ Sci & Technol China, Dept Automat, Hefei, Peoples R China
[4] Univ Sci & Technol China, Inst Adv Technol, Hefei, Peoples R China
基金
中国国家自然科学基金; 国家重点研发计划;
关键词
spatiotemporal image fusion; convolutional neural network; deep learning; remote sensing; REFLECTANCE FUSION; LANDSAT; MODIS; DYNAMICS; MODEL; NDVI;
D O I
10.1117/1.JRS.16.038506
中图分类号
X [环境科学、安全科学];
学科分类号
08 ; 0830 ;
摘要
For satellite imaging instruments, the tradeoff between spatial and temporal resolution leads to the spatial-temporal contradiction of image sequences. Spatiotemporal image fusion (STIF) provides a solution to generate images with both high-spatial and high-temporal resolutions, thus expanding the applications of existing satellite images. Most deep learning-based STIF methods throw the task to network as a whole and construct an end-to-end model without caring about the intermediate physical process. This leads to high complexity, less interpretability, and low accuracy of the fusion model. To address this problem, we propose a two-stream difference transformation spatiotemporal fusion (TSDTSF), which includes transformation and fusion streams. In the transformation stream, an image difference transformation module reduces the pixel distribution difference of images from different sensors with the same spatial resolution, and a feature difference transformation module improves the feature quality of low-resolution images. The fusion stream focuses on feature fusion and image reconstruction. The TSDTSF shows superior performance in accuracy, vision quality, and robustness. The experimental results show that TSDTSF achieves the effect of the average coefficient of determination (R-2 = 0.7847) and the root mean square error (RMSE = 0.0266), which is better than the suboptimal method average (R-2 = 0.7519) and (RMSE = 0.0289). The quantitative and qualitative experimental results on various datasets demonstrate our superiority over the state-of-the-art methods. (C) 2022 Society of Photo-Optical Instrumentation Engineers (SPIE)
引用
收藏
页数:15
相关论文
共 50 条
  • [1] StfNet: A Two-Stream Convolutional Neural Network for Spatiotemporal Image Fusion
    Liu, Xun
    Deng, Chenwei
    Chanussot, Jocelyn
    Hong, Danfeng
    Zhao, Baojun
    IEEE TRANSACTIONS ON GEOSCIENCE AND REMOTE SENSING, 2019, 57 (09): : 6552 - 6564
  • [2] Remote Sensing Image Fusion Based on Two-Stream Fusion Network
    Liu, Xiangyu
    Wang, Yunhong
    Liu, Qingjie
    MULTIMEDIA MODELING, MMM 2018, PT I, 2018, 10704 : 428 - 439
  • [3] Remote sensing image fusion based on two-stream fusion network
    Liu, Xiangyu
    Liu, Qingjie
    Wang, Yunhong
    INFORMATION FUSION, 2020, 55 : 1 - 15
  • [4] A two-stream network with complementary feature fusion for pest image classification
    Wang, Chao
    Zhang, Jinrui
    He, Jin
    Luo, Wei
    Yuan, Xiaohui
    Gu, Lichuan
    ENGINEERING APPLICATIONS OF ARTIFICIAL INTELLIGENCE, 2023, 124
  • [5] A fully convolutional two-stream fusion network for interactive image segmentation
    Hu, Yang
    Soltoggio, Andrea
    Lock, Russell
    Carter, Steve
    NEURAL NETWORKS, 2019, 109 : 31 - 42
  • [6] Automated Classification of General Movements in Infants Using Two-Stream Spatiotemporal Fusion Network
    Hashimoto, Yuki
    Furui, Akira
    Shimatani, Koji
    Casadio, Maura
    Moretti, Paolo
    Morasso, Pietro
    Tsuji, Toshio
    MEDICAL IMAGE COMPUTING AND COMPUTER ASSISTED INTERVENTION, MICCAI 2022, PT II, 2022, 13432 : 753 - 762
  • [7] A Spatiotemporal Heterogeneous Two-Stream Network for Action Recognition
    Chen, Enqing
    Bai, Xue
    Gao, Lei
    Tinega, Haron Chweya
    Ding, Yingqiang
    IEEE ACCESS, 2019, 7 : 57267 - 57275
  • [8] Remote Sensing Image Fusion Algorithm Based on Two-Stream Fusion Network and Residual Channel Attention Mechanism
    Huang, Mengxing
    Liu, Shi
    Li, Zhenfeng
    Feng, Siling
    Wu, Di
    Wu, Yuanyuan
    Shu, Feng
    WIRELESS COMMUNICATIONS & MOBILE COMPUTING, 2022, 2022
  • [9] Two-stream spatiotemporal feature fusion for human action recognition
    Abdelbaky, Amany
    Aly, Saleh
    VISUAL COMPUTER, 2021, 37 (07): : 1821 - 1835
  • [10] Two-stream spatiotemporal feature fusion for human action recognition
    Amany Abdelbaky
    Saleh Aly
    The Visual Computer, 2021, 37 : 1821 - 1835