Effective Surrogate Gradient Learning With High-Order Information Bottleneck for Spike-Based Machine Intelligence

被引:43
作者
Yang, Shuangming [1 ]
Chen, Badong [2 ]
机构
[1] Tianjin Univ, Sch Elect & Informat Engn, Tianjin 300072, Peoples R China
[2] Xi An Jiao Tong Univ, Inst Artificial Intelligence & Robot, Xian 710049, Peoples R China
基金
中国国家自然科学基金;
关键词
Training; Neurons; Robustness; Mutual information; Biological neural networks; Power demand; Kernel; Information bottleneck (IB); information-theoretic learning (ITL); neuromorphic computing; spike-driven learning; spiking neural network (SNN); NEURAL-NETWORKS;
D O I
10.1109/TNNLS.2023.3329525
中图分类号
TP18 [人工智能理论];
学科分类号
081104 ; 0812 ; 0835 ; 1405 ;
摘要
Brain-inspired computing technique presents a promising approach to prompt the rapid development of artificial general intelligence (AGI). As one of the most critical aspects, spiking neural networks (SNNs) have demonstrated superiority for AGI, such as low power consumption. Effective training of SNNs with high generalization ability, high robustness, and low power consumption simultaneously is a significantly challenging problem for the development and success of applications of spike-based machine intelligence. In this research, we present a novel and flexible learning framework termed high-order spike-based information bottleneck (HOSIB) leveraging the surrogate gradient technique. The presented HOSIB framework, including second-order and third-order formation, i.e., second-order information bottleneck (SOIB) and third-order information bottleneck (TOIB), comprehensively explores the common latent architecture and the spike-based intrinsic information and discards the superfluous information in the data, which improves the generalization capability and robustness of SNN models. Specifically, HOSIB relies on the information bottleneck (IB) principle to prompt the sparse spike-based information representation and flexibly balance its exploitation and loss. Extensive classification experiments are conducted to empirically show the promising generalization ability of HOSIB. Furthermore, we apply the SOIB and TOIB algorithms in deep spiking convolutional networks to demonstrate their improvement in robustness with various categories of noise. The experimental results prove the HOSIB framework, especially TOIB, can achieve better generalization ability, robustness and power efficiency in comparison with the current representative studies.
引用
收藏
页码:1734 / 1748
页数:15
相关论文
共 64 条
[1]  
Bu T, 2023, Arxiv, DOI arXiv:2303.04347
[2]   Spiking Deep Convolutional Neural Networks for Energy-Efficient Object Recognition [J].
Cao, Yongqiang ;
Chen, Yang ;
Khosla, Deepak .
INTERNATIONAL JOURNAL OF COMPUTER VISION, 2015, 113 (01) :54-66
[3]   Mixture correntropy for robust learning [J].
Chen, Badong ;
Wang, Xin ;
Lu, Na ;
Wang, Shiyuan ;
Cao, Jiuwen ;
Qin, Jing .
PATTERN RECOGNITION, 2018, 79 :318-327
[4]  
Chen Jianbo, 2018, PR MACH LEARN RES, V80
[5]  
Cover T.M., 2005, ELEMENTS INFORM THEO, P347
[6]  
Deng S., 2022, ARXIV
[7]   Deep Learning for Visual SLAM in Transportation Robotics: A review [J].
Duan, Chao ;
Junginger, Steffen ;
Huang, Jiahao ;
Jin, Kairong ;
Thurow, Kerstin .
TRANSPORTATION SAFETY AND ENVIRONMENT, 2019, 1 (03) :177-184
[8]  
Dubois Yann, 2020, ADV NEUR IN, V33
[9]  
Estella A., 2018, PROC INT ZURICH SEM, P35
[10]   Distributed Variational Representation Learning [J].
Estella-Aguerri, Inaki ;
Zaidi, Abdellatif .
IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, 2021, 43 (01) :120-138