Atrous Pyramid Transformer with Spectral Convolution for Image Inpainting

Owing to the ability of extracting features of images on long-range dependencies naturally, transformer is possible to reconstruct the damaged areas of images with the information from the uncorrupted regions globally. In this paper, we propose a two-stage framework based on a novel atrous pyramid transformer (APT) for image inpainting that recovers the structure and texture of an image progressively. Specifically, the patches of APT blocks are embedded in an atrous pyramid manner to explicitly enhance the correlation for both inter-and intra-windows to restore the high-level semantic structures of images more precisely, which could be served as a guide map for the second phase. Subsequently, a dual spectral transform convolution (DSTC) module is further designed to work together with APT to infer the low-level features of the generated areas. The DSTC module decouples the image signal into high frequency and low frequency for capturing texture information with a global view. Experiments on the CelebA-HQ, Paris StreetView, and Places2 demonstrate the superiority of the proposed approach. Code is available at: https://github.com/MuqiH/APT-with-DSTC.git.