Joint Classification of Hyperspectral and LiDAR Data Using Hierarchical Multimodal Feature Aggregation-Based Multihead Axial Attention Transformer

The rapid development of sensor and multimodal technology has provided more possibilities for multisource remote sensing image classification. However, some existing joint classification methods are limited to single-level feature fusion and fail to fully explore the deep correlation between cross-level features, thus limiting the effective interaction and complementarity of information between different modal data. To alleviate this issue, this article proposes a hierarchical multimodal feature aggregation-based multihead axial attention transformer (HMAT) for joint classification of hyperspectral and light detection and ranging (LiDAR) data. First, a hierarchical multimodal feature aggregation module (HMFA) is proposed to more effectively fuse spatial-spectral features of hyperspectral images (HSIs) and elevation features of LiDAR data and generate more discriminative low-dimensional feature representations. Second, a pyramid-inverted pyramid convolution module (PIP) is designed. Through the complementary feature extraction structure, PIP can more fully capture the multiscale local features in the fused feature map of hyperspectral and LiDAR data. Finally, a multihead axial attention (MHAA) component is constructed to capture information at different scales in the fused feature maps, thereby accurately modeling global dependencies. The proposed HMAT has been extensively tested on three publicly available datasets. The experimental results demonstrate that the classification performance of the proposed method outperforms that of several state-of-the-art methods.