Automatic segmentation, inpainting, and classification of defective patterns on ancient architecture using multiple deep learning algorithms

Ancient architectures have a lot of distinctive patterns on their component surfaces. But due to the wind and the Sun, the patterns will present extensive deficiencies, which are not conducive to the routine inspections and the subsequent repair work. To overcome these limits, an automatic deep learning-based method of segmentation, inpainting, and classification for defective patterns on ancient architecture is proposed. First, You Only Look At CoefficienTs, which is a real-time instance segmentation network, is applied to obtain the mask of the defective parts. Then Generative Image Inpainting with Contextual Attention, which is an image inpainting algorithm, is used to reconstruct the defective parts. Finally, Residual Neural Network-50 is used to classify the reconstructed images. In this paper, three types of dragon motifs in the Forbidden City were studied. The results show that the classification accuracy of the reconstructed images is increased by an average of 13.1%, with the maximum increasing by 23.5%. The proposed methods can prepare for future routine inspections in advance.