Infrared and visible image fusion based on VPDE model and VGG network

Infrared (IR) and visible (VIS) image fusion techniques are widely applied to many high-level vision tasks, such as object detection, recognition, and tracking. However, most existing image fusion algorithms exhibit varying degrees of edge-step effect and texture information degradation in their fused images. To improve the fusion quality, an IR and VIS image fusion method based on a variational partial differential equation (VPDE) model and a VGG network is proposed. A productive smoothing segmentation is integrated into the energy function of the VPDE model, which is based on a novel regularization function. To decompose source images into low-frequency and high-frequency components, the new VPDE model is employed. To fuse low-frequency components, a probabilistic parameter model based on space-alternating generalized expectation-maximization (SAGE) is utilized rather than the traditional average fusion rule. Then, multi-layer features of the high-frequency components are extracted using a VGG network. To generate several candidates of the fused detail content, the l1\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$l_1$$\end{document}-norm and weighted average rule are adopted, and the final details are obtained by using the maximum selection strategy. Finally, fused images are obtained by reconstructing the fused low-frequency and high-frequency components. Extensive experiments on the TNO and RoadScene datasets demonstrate that the proposed technique effectively eliminates artifacts as well as the step effect. In the subjective comparison, the proposed method can highlight the salient objects of the fused images while strengthening the texture information. In terms of the evaluation metrics, the proposed method outperforms 13 state-of-the-art methods in objective comparison in addition to the subjective evaluation.