A Simulation System for Scene Synthesis in Virtual Reality

Real-world scene synthesis can be realized through view synthesis or 3D reconstruction methods. While industrial and commercial demands emerge for real-world scene synthesis in virtual reality (VR) using head-mounted displays (HMDs), the methods in the literature generally do not target specific display devices. To meet the rising demands, we propose a simulation system to evaluate scene synthesis methods in VR. Our system aims at providing the full pipeline of scene capturing, processing, rendering, and evaluation. The capturing module provides various input dataset formulations. The processing and rendering module integrates three representative scene synthesis methods with a voluntary performance-aid option. Finally, the evaluation module supports traditional metrics as well as perception-based metrics. An experiment demonstrates the use of our system for identifying the best capturing strategy among the three degrees of foveation tested. As can be expected, the FovVideoVDP metric (based on a model of the human visual system) finds the highest degree of foveation giving best results. The three other quality metrics from the evaluation module (which use features to measure similarity) confirm that result. The synthetic scenes in the experiment can run in VR with an average latency of 5.9 ms for the two selected scenarios across the tested methods on Nvidia GTX 2080 ti.