Coupled adversarial learning for fusion classification of hyperspectral and LiDAR data

被引:57
作者
Lu, Ting [1 ]
Ding, Kexin [1 ]
Fu, Wei [2 ]
Li, Shutao [1 ]
Guo, Anjing [1 ]
机构
[1] Hunan Univ, Coll Elect & Informat Engn, Changsha, Peoples R China
[2] Hunan Univ, Coll Comp Sci & Elect Engn, Changsha, Peoples R China
基金
中国国家自然科学基金; 中国博士后科学基金;
关键词
Hyperspectral image; Light detection and ranging; Multimodal data classification; Adversarial learning; Feature fusion; REMOTE-SENSING IMAGES; INFORMATION FUSION; NETWORK; SPARSE;
D O I
10.1016/j.inffus.2022.12.020
中图分类号
TP18 [人工智能理论];
学科分类号
081104 ; 0812 ; 0835 ; 1405 ;
摘要
Hyperspectral image (HSI) provides rich spectral-spatial information and the light detection and ranging (LiDAR) data reflect the elevation information, which can be jointly exploited for better land-cover clas-sification. However, due to different imaging mechanisms, HSI and LiDAR data always present significant image difference, current pixel-wise feature fusion classification methods relying on concatenation or weighted fusion are not effective. To achieve accurate classification result, it is important to extract and fuse similar high-order semantic information and complementary discriminative information contained in multimodal data. In this paper, we propose a novel coupled adversarial learning based classification (CALC) method for fusion classification of HSI and LiDAR data. In specific, a coupled adversarial feature learning (CAFL) sub-network is first trained, to effectively learn the high-order semantic features from HSI and LiDAR data in an unsupervised manner. On one hand, the proposed CAFL sub-network establishes an adversarial game between dual generators and discriminators, so that the learnt features can preserve detail information in HSI and LiDAR data, respectively. On the other hand, by designing weight-sharing and linear fusion structure in the dual generators, we can simultaneously extract similar high-order semantic information and modal-specific complementary information. Meanwhile, a supervised multi-level feature fusion classification (MFFC) sub-network is trained, to further improve the classification performance via adaptive probability fusion strategy. In brief, the low-level, mid-level and high-level features learnt by the CAFL sub-network lead to multiple class estimation probabilities, which are then adaptively combined to generate a final accurate classification result. Both the CAFL and MFFC sub-networks are collaboratively trained by optimizing a designed joint loss function, which consists of unsupervised adversarial loss and supervised classification loss. Overall, by optimizing the joint loss function, the proposed CALC network is pushed to learn highly discriminative fusion features from multimodal data, leading to higher classification accuracies. Extensive experiments on three well-known HSI and LiDAR data sets demonstrate the superior classification performance by the proposed CALC method than several state-of-the-art methods. The source code of the proposed method will be made publicly available at https://github.com/Ding-Kexin/CALC.
引用
收藏
页码:118 / 131
页数:14
相关论文
共 63 条
[1]   A review of state-of-the-art in Face Presentation Attack Detection: From early development to advanced deep learning and multi-modal fusion methods [J].
Abdullakutty, Faseela ;
Elyan, Eyad ;
Johnston, Pamela .
INFORMATION FUSION, 2021, 75 :55-69
[2]  
[Anonymous], ICLR 2016
[3]  
Arjovsky M., 2017, P INT C LEARNING RE
[4]  
Arjovsky M, 2017, PR MACH LEARN RES, V70
[5]  
Basnyat B, 2021, 2021 IEEE INTERNATIONAL CONFERENCE ON PERVASIVE COMPUTING AND COMMUNICATIONS WORKSHOPS AND OTHER AFFILIATED EVENTS (PERCOM WORKSHOPS), P135, DOI [10.1109/PerComWorkshops51409.2021.9430985, 10.1109/PERCOMWORKSHOPS51409.2021.9430985]
[6]  
Cao Q, 2018, INT GEOSCI REMOTE SE, P8869
[7]   Classification of remote sensing images from urban areas using a fuzzy possibilistic model [J].
Chanussot, J ;
Benediktsson, JA ;
Fauvel, M .
IEEE GEOSCIENCE AND REMOTE SENSING LETTERS, 2006, 3 (01) :40-44
[8]   Using HSI Color Space to Improve the Multispectral Lidar Classification Error Caused by Measurement Geometry [J].
Chen, Biwu ;
Shi, Shuo ;
Sun, Jia ;
Chen, Bowen ;
Guo, Kuanghui ;
Du, Lin ;
Yang, Jian ;
Xu, Qian ;
Song, Shalei ;
Gong, Wei .
IEEE TRANSACTIONS ON GEOSCIENCE AND REMOTE SENSING, 2021, 59 (04) :3567-3579
[9]  
Chen C, 2019, INT GEOSCI REMOTE SE, P2475, DOI [10.1109/IGARSS.2019.8898443, 10.1109/igarss.2019.8898443]
[10]   DEEP FUSION OF HYPERSPECTRAL AND LIDAR DATA FOR THEMATIC CLASSIFICATION [J].
Chen, Yushi ;
Li, Chunyang ;
Ghamisi, Pedram ;
Shi, Chunyu ;
Gu, Yanfeng .
2016 IEEE INTERNATIONAL GEOSCIENCE AND REMOTE SENSING SYMPOSIUM (IGARSS), 2016, :3591-3594