Semantic Stereo using Semi Global Matching and Convolutional Neural Networks

3D reconstruction from satellite stereo images is important for numerous applications like 3D city modelling and urban mapping. The Semi Global Matching (SGM) is the state-of-the-art algorithm for dense image matching of satellite stereo images. However, the results of dense image matching are controlled by many factors like, occlusions (presence of some features only in one image), radiometric differences, shadows, texture less surfaces and perspective distortions, which results in significant amount of outliers and missing data in the resulting disparity maps. This problem can be reduced if image segmentation is available. This work explores the fusion of the image segmentation with 3D reconstruction not only to improve the 3D reconstruction results but also to provide the semantic label for each pixel of the image. The image segmentation is itself a challenging problem. Here, we leverage the recent advancements in deep learning to train a Convolution Neural Network (CNN) for segmentation of satellite images in to buildings, roads, water and vegetation classes. The CNN architecture used here is based on the popular U-Net architecture for semantic segmentation. The data from 2019 IEEE GRSS data fusion contest track 2 is used for training of the CNN. The trained network is used for semantic segmentation of the satellite images. These segmentation masks are used to refine the SGM based disparity maps by filtering the disparities and void filling only in the connected component of each segment, which helped in computing smoother disparity maps with fewer outliers and missing data. The evaluation of results of refined disparity and image segmentation was done according to GRSS semantic Labeling, on the bases of IoU. Our net mIoU was 0.7612. As compared to results of simple disparities from SGM results of refined disparity maps were better.