Creating Historical Building Models by Deep Fusion of Multi-Source Heterogeneous Data Using Residual 3D Convolutional Neural Network

There are a large number of historical buildings in the world, but many of them are seriously damaged. A fundamental task to revitalize them is to record their details and redraw them, but it is a highly inefficient task. This paper aimed to create three-dimensional (3D) models automatically for those damaged historical buildings while respecting the 1964 Venice Charter, the ICOMOS recommendations, and local ordinances. After reviewing the advantages and disadvantages of various measurement tools, we implemented a machine learning algorithm, the residual 3D convolutional neural network (CNN) method, to combine heterogeneous data from multiple sources including a 3D laser scanner, an unmanned aerial vehicle, a total station, and a camera, in a historical building information model (HBIM) to increase its accuracy. Those data, preprocessed by four steps including data registration, reducing noises, filtering data, and data integration, were trained and fused in a 3D CNN model, which consisted of one input layer, five 3D convolutional and pooling layers, and two connected layers. A Shanghai historical building conservation project was taken as a pilot case to verify its productivity and accuracy in developing the HBIM, on which the architectural features were thoroughly recorded, and the damaged details were recovered.