Real-Time Layered View Synthesis for Free-Viewpoint Video from Unreliable Depth Information

In this work we present a novel approach for the generation in real time of synthetic views for free-viewpoint video. Our system is based on purely passive stereo cameras which, under the constraints of real-time operation yield unreliable depth maps, especially in the background areas. To solve this issue we propose a layered synthesis algorithm that combines offline and online sources of colour and depth information to increase the quality of the synthesized image. Foreground objects are synthesized using online depth and colour information. The background, however, is assumed to be stationary so its geometry can be reconstructed offline. However, foreground objects affect the colour of the background, so to render the background we carefully mix online and offline colour information. Finally, foreground and background are combined to form the final image.