Learning-based Inverse Rendering of Complex Indoor Scenes with Differentiable Monte Carlo Raytracing

被引:14
|
作者
Zhu, Jingsen [1 ]
Luan, Fujun [2 ]
Huo, Yuchi [1 ]
Lin, Zihao [1 ]
Zhong, Zhihua [1 ]
Xi, Dianbing [1 ]
Zheng, Jiaxiang [3 ]
Tang, Rui [3 ]
Bao, Hujun [1 ]
Wang, Rui [1 ]
机构
[1] Zhejiang Univ, State Key Lab CAD&CG, Hangzhou, Peoples R China
[2] Adobe Res, San Francisco, CA USA
[3] Manycore, KooLab, Beijing, Peoples R China
来源
PROCEEDINGS SIGGRAPH ASIA 2022 | 2022年
关键词
ray tracing; lighting estimation; inverse rendering;
D O I
10.1145/3550469.3555407
中图分类号
TP18 [人工智能理论];
学科分类号
081104 ; 0812 ; 0835 ; 1405 ;
摘要
Indoor scenes typically exhibit complex, spatially-varying appearance from global illumination, making inverse rendering a challenging ill-posed problem. This work presents an end-to-end, learning-based inverse rendering framework incorporating differentiable Monte Carlo raytracing with importance sampling. The framework takes a single image as input to jointly recover the underlying geometry, spatially-varying lighting, and photorealistic materials. Specifically, we introduce a physically-based differentiable rendering layer with screen-space ray tracing, resulting in more realistic specular reflections that match the input photo. In addition, we create a large-scale, photorealistic indoor scene dataset with significantly richer details like complex furniture and dedicated decorations. Further, we design a novel out-of-view lighting network with uncertainty-aware refinement leveraging hypernetwork-based neural radiance fields to predict lighting outside the view of the input photo. Through extensive evaluations on common benchmark datasets, we demonstrate superior inverse rendering quality of our method compared to state-of-the-art baselines, enabling various applications such as complex object insertion and material
引用
收藏
页数:8
相关论文
共 50 条
  • [21] Monte Carlo Noise Reduction Algorithm Based on Deep Neural Network in Efficient Indoor Scene Rendering System
    Chen, Xiwen
    Shen, Jianfei
    ADVANCES IN MULTIMEDIA, 2022, 2022
  • [22] Pricing Chinese Convertible Bonds with Learning-Based Monte Carlo Simulation Model
    Zhu, Jiangshan
    Wen, Conghua
    Li, Rong
    AXIOMS, 2024, 13 (04)
  • [23] Density-based Outlier Rejection in Monte Carlo Rendering
    DeCoro, Christopher
    Weyrich, Tim
    Rusinkiewicz, Szymon
    COMPUTER GRAPHICS FORUM, 2010, 29 (07) : 2119 - 2125
  • [24] Sampling-Distribution-Based Evaluation for Monte Carlo Rendering
    Freude, Christian
    Sakai, Hiroyuki
    Zsolnai-Feher, Karoly
    Wimmer, Michael
    PROCEEDINGS OF THE 18TH INTERNATIONAL JOINT CONFERENCE ON COMPUTER VISION, IMAGING AND COMPUTER GRAPHICS THEORY AND APPLICATIONS, VISIGRAPP 2023, 2023, : 119 - 130
  • [25] Langevin Monte Carlo Rendering with Gradient-based Adaptation
    Luan, Fujun
    Zhao, Shuang
    Bala, Kavita
    Gkioulekas, Ioannis
    ACM TRANSACTIONS ON GRAPHICS, 2020, 39 (04):
  • [26] Multi-view Inverse Rendering for Large-scale Real-world Indoor Scenes
    Li, Zhen
    Wang, Lingli
    Cheng, Mofang
    Pan, Cihui
    Yang, Jiaqi
    2023 IEEE/CVF CONFERENCE ON COMPUTER VISION AND PATTERN RECOGNITION (CVPR), 2023, : 12499 - 12509
  • [27] Rendering optical effects based on spectra representation in complex scenes
    Dong, Weiming
    ADVANCES IN COMPUTER GRAPHICS, PROCEEDINGS, 2006, 4035 : 719 - 726
  • [28] HMD-Guided Image-Based Modeling and Rendering of Indoor Scenes
    Andersen, Daniel
    Popescu, Voicu
    VIRTUAL REALITY AND AUGMENTED REALITY, EUROVR 2018, 2018, 11162 : 73 - 93
  • [29] Deep learning-based prediction of Monte Carlo dose distribution for heavy ion therapy
    He, Rui
    Zhang, Hui
    Wang, Jian
    Shen, Guosheng
    Luo, Ying
    Zhang, Xinyang
    Ma, Yuanyuan
    Liu, Xinguo
    Li, Yazhou
    Peng, Haibo
    He, Pengbo
    Li, Qiang
    PHYSICS & IMAGING IN RADIATION ONCOLOGY, 2025, 34
  • [30] Back-calculation of keratometer index based on OCT data and raytracing - a Monte Carlo simulation
    Langenbucher, Achim
    Szentmary, Nora
    Weisensee, Johannes
    Cayless, Alan
    Menapace, Rupert
    Hoffmann, Peter
    ACTA OPHTHALMOLOGICA, 2021, 99 (08) : 843 - 849