Image inpainting based on CNN-Transformer framework via structure and texture restoration

Both coherent structure and fine texture are of great importance in image inpainting. However, existing methods fail to consider both structure and texture simultaneously, resulting in discontinuous edges, blurred details and artifacts of inpainting results. In addition, Convolutional Neural Network (CNN) suffers from its limited modeling capability for long-range dependencies. To address these issues, we propose a novel method called Image inpainting based on CNN-Transformer framework via structure and texture restoration (CTSTNet). In CTSTNet, the Structure Reconstruction Network (SRN) restores the overall structure of the damaged image by leveraging its ability to model long dependencies. Meanwhile, the Texture Reconstruction Network (TRN) restores the grayscale map and edge map to generate the texture of the damaged image. Finally, Image Completion Network (ICN) fuses the structure and texture using a Context Aggregation Module (CAM) to generate the inpainting result. Experimental results on two public datasets CelebA, Paris StreetView, and our constructed mural dataset MuralCN, demonstrate that CTSTNet achieves significant improvements in evaluation metrics and visual quality compared to other related methods. This indicates that our CTSTNet can effectively restore images and can also be applied in the field of cultural relic image restoration.