Image Harmonization Guided by Semantic Information

Image harmonization occupies an important position in image processing. It aims to adjust the foreground appearance, e.g., illumination, color, texture, etc., to be visually consistent with the background. However, existing deep learning-based methods usually use the feature distribution of the overall image background as a cue to adjust the foreground, without focusing on the critical role of semantic information for foreground alignment, resulting in local areas in the foreground to appear visually different from the background. To this end, based on the multi-resolution selective fusion module (MRSFM) and the lightweight convolutional block attention module (CBAM), this paper designs a multi-resolution selective fusion module based on dual attention mechanism (MRSF-DAM), which makes the final output feature map rich in semantic information, thus guiding the network to better understand the correlation between the foreground of an image and its surrounding scene, more enabling the network to fully obtain the various information needed to coordinate the foreground from the background, and eventually reducing the visual discrepancy between the foreground and background regions of an image. In addition, this article designs a new network architecture to selectively fuse the shallow and deep feature information. By multi-scale fusion and enhancement of the output feature maps of the first six network layers of the decoder and MRSF-DAM, the generated enhanced feature maps are fed into the final layer of the decoder, which can alleviate the problem introduced by skip connections of the unrelated features to the foreground,and besides, it reduces the loss of spatial feature information caused by multiple downsampling of the decoder, further improving the authenticity of the generated harmonized images. A large number of experiments were conducted on the widely used iHarmony4 benchmark dataset to verify the effectiveness of our method. Compared to the latest method SCS Co (Self Consistent Style Comparative learning for image harmonization), this proposed method improves the mean squared error (MSE), foreground mean squared error (fMSE) and peak signal to noise ratio (PSNR) of the entire dataset by 4.28, 61.97, and 1 dB, respectively. © 2023 Chinese Institute of Electronics. All rights reserved.