Rendering Optimizations for Virtual Reality Using Eye-Tracking

Optimizing rendering in virtual reality is an open problem in computer science. The nature of modern VR display technology (high refresh rate and increasing pixel density), coupled with the relatively slow growth modern compute capability, is leading to a bottleneck in VR performance. As we further research methodologies for improving rendering performance and accuracy for VR, it is important to understand the historical approaches and where they succeeded or failed in their approaches. Some implementations will double computing because of the need of stereoscopy, and thus have higher overhead for rendering. This can be improved with Multi-View Rendering where the GPU hardware can assist in duplicating rasterization for multiple views with differing projections. More recently, perception-based rendering has gained traction, which can be further accelerated using Variable Shading Rate or Multi-Rate Shading technology found on more recent GPUs. There has also been some success in using deep neural networks to assist with transmitting foveated content over a network. The advances in the field leave many open research questions, including sparse pixel rendering, driving user attention, and techniques and methodologies for combining variable shading rate images. This review focuses research associated with rendering optimizations for virtual reality using eye tracking, since it is becoming a feature present in consumer-level head-mounted displays. From our review, affordable off-the-shelf virtual reality and eye tracking are both leading to freeing up rendering resources towards improved performance and visual fidelity, as well as providing new and exciting opportunities for human-computer interaction.