Integrating model- and data-driven methods for synchronous adaptive multi-band image fusion

A novel synchronous adaptive framework for multi-band image fusion is proposed, based on integrated model-and data-driven (MDDR) techniques. This approach includes a deep stack convolutional neural network (DSCNN) for multi-band images, established by redefining convolutional kernels in the first layer using Gaussian and Gaussian-Laplace filters. The structure of the convolutional neural network (CNN) was improved by removing a sample CNN layer to reduce information loss, prior to decomposing and reconstructing input images in an adaptive framework. A deep gate convolution neural network (DGCNN) was then established using a gate structure principle common in long short-term memory (LSTM) techniques. As a result, the network can adaptively fuse high- and low-frequency components, similar to conventional image fusion rules in model-driven algorithms. Finally, a synchronous adaptive multi-band image fusion neural network (SAMIFNN) was constructed by embedding the DGCNN into decompose- and reconstruct-subnets in the DSCNN. Data from ImageNet IL SVRC2013 and TNO image fusion datasets were used for training (80%) and testing (20%). SAMIFNN was then compared with seven state-of-the-art methods applied to eight groups of representative images, the TRICLOBS dynamic multiband image dataset, and a series of medical CT, MR, and PET scans. The proposed network required significantly lower runtimes than conventional algorithms, producing satisfactory results across 21 different evaluation metrics (compared with a maximum of 15 achieved by conventional techniques). These experimental results demonstrate that the proposed algorithm can successfully implement synchronous adaptive multi-band image fusion with higher contrast, better visual perception, and less distortion, without requiring a priori knowledge or manual intervention.