Research on Multi-View Stereo 3D Reconstruction in Virtual Reality System of Silk Road Cultural Inheritance

Silk Road culture is an important link in the Belt and Road strategy. Its heritage is of great significance. However, due to historical and geographical reasons, the representative historical heritage in the Silk Road culture is scattered or damaged, and it is difficult to present the historical heritage effectively. Therefore, in this work we propose and implement a virtual reality platform for the Silk Road Cultural Heritage. Through historical restoration and image-based 3D reconstruction, we effectively restored the historical sites, cultural relics and events of Guyuan of Ningxia Province in China, one of the important nodes in Silk Road Culture. For outdoor historical sites, we use a DJI Mavic Pro to capture 4K video clips of the giant Buddha of Xumi Mountain in a sunny day. For indoor cultural relics, we use a turntable with digital single lens reflex (DSLR) camera and multiple light sources to capture high-resolution images in 180 degrees. Based on these image data, we propose a simple and efficient multi-view stereo 3D reconstruction method for high-resolution images, which consists of a normal-aware PatchMatch stereo for the high-quality normal recovery to represent the detailed surface of the cultural relics, and a GPU-friendly incremental depth map fusion method which can fuse a large amount of depth maps by leveraging a small size of GPU memory. The high-resolution input images are essential for representing the geometric details in historical sites and cultural relics. However, the state-of-the-art depth map fusion method needs to import all depth maps and normal maps into the GPU memory, and then globally fuse the depth points into the 3D point clouds for each reference image. Nevertheless, the space complexity almost linearly increases when the amount of data and image resolution increase. For instance, doubling image size will result in a fourfold increase in GPU memory. Due to limitation of GPU memory, this kind of global fusion strategy cannot address high-resolution input image data. The proposed incremental depth map fusion method in this paper mainly consists of three steps: (1) We first set a reference view and a counter map for cross-view consistency check; (2) Then, we import α neighboring images of the reference view into GPU memory each time, and perform the cross-view consistency check for the depth points in reference view. And then, depth points are accumulated, and counter map is also updated. We then release the memory of these α images and import another α images into GPU and repeat the above operations; (3) When all neighboring views are processed, we can fuse the depth points whose values in counter map are larger than a threshold. The quantitative and qualitative experiment results on the public multi-view stereo benchmark as well as our captured datasets clearly highlight that the proposed method can recover the detailed surfaces while keeping a good scalability for the large-scale image data. The reconstructed high-quality 3D models of historical sites and cultural relics by our method can effectively support immersive virtual reality applications, playing a positive role in the dissemination of Silk Road culture. © 2022, Science Press. All right reserved.