Convolutional neural network-based multimodal image fusion via similarity learning in the shearlet domain

Recently, deep learning has been shown effectiveness in multimodal image fusion. In this paper, we propose a fusion method for CT and MR medical images based on convolutional neural network (CNN) in the shearlet domain. We initialize the Siamese fully convolutional neural network with a pre-trained architecture learned from natural data; then, we train it with medical images in a transfer learning fashion. Training dataset is made of positive and negative patch pair of shearlet coefficients. Examples are fed in two-stream deep CNN to extract features maps; then, a similarity metric learning based on cross-correlation is performed aiming to learn mapping between features. The minimization of the logistic loss objective function is applied with stochastic gradient descent. Consequently, the fusion process flow starts by decomposing source CT and MR images by the non-subsampled shearlet transform into several subimages. High-frequency subbands are fused based on weighted normalized cross-correlation between feature maps given by the extraction part of the CNN, while low-frequency coefficients are combined using local energy. Training and test datasets include pairs of pre-registered CT and MRI taken from the Harvard Medical School database. Visual analysis and objective assessment proved that the proposed deep architecture provides state-of-the-art performance in terms of subjective and objective assessment. The potential of the proposed CNN for multi-focus image fusion is exhibited in the experiments.