Deep learning based multi-view stereo matching and 3D scene reconstruction from oblique aerial images

被引:25
作者
Liu, Jin [1 ]
Gao, Jian [1 ]
Ji, Shunping [1 ]
Zeng, Chang [1 ]
Zhang, Shaoyi [1 ]
Gong, Jianya [1 ]
机构
[1] Wuhan Univ, Sch Remote Sensing & Informat Engn, 129 Luoyu Rd, Wuhan 430079, Peoples R China
基金
中国国家自然科学基金;
关键词
3D scene reconstruction; Multi-view stereo; Oblique aerial images; Deep learning; Dense image matching;
D O I
10.1016/j.isprsjprs.2023.08.015
中图分类号
P9 [自然地理学];
学科分类号
0705 ; 070501 ;
摘要
In this paper, we propose a practical three-dimensional (3D) real-scene reconstruction framework named Deep3D, which is paired with a deep learning based multi-view stereo (MVS) matching model named the adaptive multi-view aggregation matching (Ada-MVS) model, to obtain a 3D textured mesh model from multi view oblique aerial images. Deep3D is the first deep learning based framework for 3D scene reconstruction, in which aerial triangulation and view selection are first performed on the input images, and the depth map of each image is then inferred using the pretrained Ada-MVS model. All the inferred depth maps are then fused into a dense point cloud after filtering the outliers. Finally, the 3D textured mesh is extracted from the dense 3D points as the final product. In the Ada-MVS model, a novel adaptive inter-view aggregation module is specially proposed to address the inconsistent information among oblique views and to fuse the multi-view costs into a robust cost volume. A lightweight recurrent regularization module is also designed for high-efficiency processing of high-capacity aerial images with large depth variations. Moreover, as oblique aerial image datasets are currently lacking, we built a large-scale synthetic multi-view oblique aerial image dataset (WHU-OMVS dataset) for deep learning based model training and methodology evaluation for the task of 3D scene reconstruction. The experimental results show that, firstly, the proposed Ada-MVS model has obvious advantages when used with high capacity oblique aerial images, compared with several relevant learning-based MVS methods. Secondly, through a comprehensive comparison with popular commercial software packages and open-source solutions, it is shown that the proposed Deep3D framework outperforms all the other solutions in terms of reconstruction quality, and outperforms all the open-source solutions and some of the software packages in terms of efficiency on the WHU-OMVS dataset. Thirdly, the Deep3D framework shows a stable generalization ability and excellent performance when applied to other oblique or nadir aerial images, without any further fine-tuning. The dataset and code will be available at http://gpcv.whu.edu.cn/data.
引用
收藏
页码:42 / 60
页数:19
相关论文
共 79 条
  • [1] Large-Scale Data for Multiple-View Stereopsis
    Aanaes, Henrik
    Jensen, Rasmus Ramsbol
    Vogiatzis, George
    Tola, Engin
    Dahl, Anders Bjorholm
    [J]. INTERNATIONAL JOURNAL OF COMPUTER VISION, 2016, 120 (02) : 153 - 168
  • [2] Agisoft, 2022, Agisoft Metashape: Agisoft Metashape
  • [3] Bentley, 2018, ContextCapture
  • [4] PatchMatch Stereo - Stereo Matching with Slanted Support Windows
    Bleyer, Michael
    Rhemann, Christoph
    Rother, Carsten
    [J]. PROCEEDINGS OF THE BRITISH MACHINE VISION CONFERENCE 2011, 2011,
  • [5] A Naturalistic Open Source Movie for Optical Flow Evaluation
    Butler, Daniel J.
    Wulff, Jonas
    Stanley, Garrett B.
    Black, Michael J.
    [J]. COMPUTER VISION - ECCV 2012, PT VI, 2012, 7577 : 611 - 625
  • [6] Cao M., 2006, Bull. Surv. Mapp, V10, P55
  • [7] Cavegn S., 2014, Int. Arch. Photogramm., Remote Sens. Spat. Inf. Sci., V40, P45
  • [8] Pyramid Stereo Matching Network
    Chang, Jia-Ren
    Chen, Yong-Sheng
    [J]. 2018 IEEE/CVF CONFERENCE ON COMPUTER VISION AND PATTERN RECOGNITION (CVPR), 2018, : 5410 - 5418
  • [9] MVSNet plus plus : Learning Depth-Based Attention Pyramid Features for Multi-View Stereo
    Chen, Po-Heng
    Yang, Hsiao-Chien
    Chen, Kuan-Wen
    Chen, Yong-Sheng
    [J]. IEEE TRANSACTIONS ON IMAGE PROCESSING, 2020, 29 : 7261 - 7273
  • [10] Deep Stereo using Adaptive Thin Volume Representation with Uncertainty Awareness
    Cheng, Shuo
    Xu, Zexiang
    Zhu, Shilin
    Li, Zhuwen
    Li, Li Erran
    Ramamoorthi, Ravi
    Su, Hao
    [J]. 2020 IEEE/CVF CONFERENCE ON COMPUTER VISION AND PATTERN RECOGNITION (CVPR), 2020, : 2521 - 2531