Multi-attentive hierarchical dense fusion net for fusion classification of hyperspectral and LiDAR data

With recent advance in Earth Observation techniques, the availability of multi-sensor data acquired in the same geographical area has been increasing greatly, which makes it possible to jointly depict the underlying landcover phenomenon using different sensor data. In this paper, a novel multi-attentive hierarchical fusion net (MAHiDFNet) is proposed to realize the feature-level fusion and classification of hyperspectral image (HSI) with Light Detection and Ranging (LiDAR) data. More specifically, a triple branch HSI-LiDAR Convolutional Neural Network (CNN) backbone is first developed to simultaneously extract the spatial features, spectral features and elevation features of the land-cover objects. On this basis, hierarchical fusion strategy is adopted to fuse the oriented feature embeddings. In the shallow feature fusion stage, we propose a novel modality attention (MA) module to generate the modality integrated features. By fully considering the correlation and heterogeneity between different sensor data, feature interaction and integration is released by the proposed MA module. At the same time, self-attention modules are also adopted to highlight the modality specific features. In the deep feature fusion stage, the obtained modality specific features and modality integrated features are fused to construct the hierarchical feature fusion framework. Experiments on three real HSI-LiDAR datasets demonstrate the effectiveness of the proposed framework. The code will be public on https://github.com/SYFYN0317/-MAHiDFNet.