3D Gaussian Splatting for Real-Time Radiance Field Rendering

被引:1170
作者
Kerbl, Bernhard [1 ]
Kopanas, Georgios [1 ]
Leimkuehler, Thomas [2 ]
Drettakis, George [1 ]
机构
[1] Univ Cote dAzur, INRIA, Nice, France
[2] Max Planck Inst Informat, Saarbrucken, Germany
来源
ACM TRANSACTIONS ON GRAPHICS | 2023年 / 42卷 / 04期
关键词
novel view synthesis; radiance fields; 3D gaussians; real-time rendering;
D O I
10.1145/3592433
中图分类号
TP31 [计算机软件];
学科分类号
081202 ; 0835 ;
摘要
Radiance Field methods have recently revolutionized novel-view synthesis of scenes captured with multiple photos or videos. However, achieving high visual quality still requires neural networks that are costly to train and render, while recent faster methods inevitably trade off speed for quality. For unbounded and complete scenes (rather than isolated objects) and 1080p resolution rendering, no current method can achieve real-time display rates. We introduce three key elements that allow us to achieve state-of-the-art visual quality while maintaining competitive training times and importantly allow high-quality real-time (>= 30 fps) novel-view synthesis at 1080p resolution. First, starting from sparse points produced during camera calibration, we represent the scene with 3D Gaussians that preserve desirable properties of continuous volumetric radiance fields for scene optimization while avoiding unnecessary computation in empty space; Second, we perform interleaved optimization/density control of the 3D Gaussians, notably optimizing anisotropic covariance to achieve an accurate representation of the scene; Third, we develop a fast visibility-aware rendering algorithm that supports anisotropic splatting and both accelerates training and allows real-time rendering. We demonstrate state-of-the-art visual quality and real-time rendering on several established datasets.
引用
收藏
页数:14
相关论文
共 60 条
[1]   Neural Point-Based Graphics [J].
Aliev, Kara-Ali ;
Sevastopolsky, Artem ;
Kolos, Maria ;
Ulyanov, Dmitry ;
Lempitsky, Victor .
COMPUTER VISION - ECCV 2020, PT XXII, 2020, 12367 :696-712
[2]  
AngtianWang PengWang, 2023, 11 INT C LEARNING RE
[3]   Mip-NeRF 360: Unbounded Anti-Aliased Neural Radiance Fields [J].
Barron, Jonathan T. ;
Mildenhall, Ben ;
Verbin, Dor ;
Srinivasan, Pratul P. ;
Hedman, Peter .
2022 IEEE/CVF CONFERENCE ON COMPUTER VISION AND PATTERN RECOGNITION (CVPR 2022), 2022, :5460-5469
[4]   Mip-NeRF: A Multiscale Representation for Anti-Aliasing Neural Radiance Fields [J].
Barron, Jonathan T. ;
Mildenhall, Ben ;
Tancik, Matthew ;
Hedman, Peter ;
Martin-Brualla, Ricardo ;
Srinivasan, Pratul P. .
2021 IEEE/CVF INTERNATIONAL CONFERENCE ON COMPUTER VISION (ICCV 2021), 2021, :5835-5844
[5]  
Bonopera S., 2020, SIBR SYSTEM IMAGE BA
[6]  
Botsch Mario, 2005, Proceedings Eurographics/IEEE VGTC Symposium Point-Based Graphics, P17, DOI DOI 10.1109/PBG.2005.194059
[7]  
Buehler C, 2001, COMP GRAPH, P425, DOI 10.1145/383259.383309
[8]   Depth Synthesis and Local Warps for Plausible Image-Based Navigation [J].
Chaurasia, Gaurav ;
Duchene, Sylvain ;
Sorkine-Hornung, Olga ;
Drettakis, George .
ACM TRANSACTIONS ON GRAPHICS, 2013, 32 (03)
[9]  
Chen Anpei, 2022, EUROPEAN C COMPUTER
[10]  
Chen ZQ, 2022, Arxiv, DOI arXiv:2208.00277