FusioNet: A Two-Stream Convolutional Neural Network for Urban Scene Classification using PolSAR and Hyperspectral Data

Urban Scene classification using single source data is massively studied in remote sensing field. However, single source only provides one certain perspective of the complicated urban scene while the fusion of multimodal dataset can provide complementary knowledge. We aim at fusing the spectrum information of the hyperspectral image and the scattering mechanisms of PolSAR data for urban scene classification. For the joint usage of the two data sets, a simple concatenation would lead to extraction of insufficient information and weakens the influence of the lower dimensional data. In this work, the end-to-end convolutional neural network is utilized to automatically learn how to effectively extract features and to fuse the hyperspectral image and the PolSAR data. More specifically, we propose a novel two-stream convolutional network architecture. It creates identical but separated convolutional stream for each data. Subsequently, the two streams are merged with comparable numbers of dimensionality within the fusion layer. This architecture ensures the effectively extraction of informative features from both data for the classification purpose and the fusion of the two data in a balanced manner. Experimental results suggest significantly superior performance of the proposed framework, while comparing to other existing fusion methods. To our knowledge, it is the first time that deep convolutional neural network accomplishes the fusion of hyperspectral image and SAR data.