Adaptive learning rate algorithms based on the improved Barzilai-Borwein method

被引:1
作者
Wang, Zhi-Jun [1 ,2 ,3 ]
Li, Hong [1 ]
Xu, Zhou-Xiang [1 ]
Zhao, Shuai-Ye [1 ]
Wang, Peng-Jun [4 ]
Gao, He-Bei [2 ]
机构
[1] Wenzhou Univ, Coll Comp Sci & Artificial Intelligence, Wenzhou 325035, Zhejiang, Peoples R China
[2] Wenzhou Med Univ, Eye Hosp, Oujiang Lab, Zhejiang Lab Regenerat Med Vis & Brain Hlth, Wenzhou 325000, Zhejiang, Peoples R China
[3] East China Normal Univ, Sch Comp Sci & Technol, Shanghai 200333, Peoples R China
[4] Wenzhou Univ, Coll Elect & Elect Engn, Wenzhou 325035, Zhejiang, Peoples R China
基金
中国国家自然科学基金;
关键词
Barzilai-Borwein step size; Momentum method; Unconstrained optimization; Deep learning; GRADIENT; STEP;
D O I
10.1016/j.patcog.2024.111179
中图分类号
TP18 [人工智能理论];
学科分类号
081104 ; 0812 ; 0835 ; 1405 ;
摘要
Objective: The Barzilai-Borwein(BB) method is essential in solving unconstrained optimization problems. The momentum method accelerates optimization algorithms with exponentially weighted moving average. In order to design reliable deep learning optimization algorithms, this paper proposes applying the BB method in four variants to the optimization algorithm of deep learning. Findings: The momentum method generates the BB step size under different step range limits. We also apply the momentum method and its variants to the stochastic gradient descent with the BB step size. Novelty: The algorithm's robustness has been demonstrated through experiments on the initial learning rate and random seeds. The algorithm's sensitivity is tested by choosing different momentum factors until a suitable momentum factor is found. Moreover, we compare our algorithms with popular algorithms in various neural networks. The results show that the new algorithms improve the efficiency of the BB step size in deep learning and provide a variety of optimization algorithm choices.
引用
收藏
页数:9
相关论文
共 50 条
[41]   A novel adaptive learning deep belief network based on automatic growing and pruning algorithms [J].
Song, Wei ;
Zhang, Shiyu ;
Wen, Zijian ;
Zhou, Junhao .
APPLIED SOFT COMPUTING, 2021, 104
[42]   Effective neural network training with adaptive learning rate based on training loss [J].
Takase, Tomoumi ;
Oyama, Satoshi ;
Kurihara, Masahito .
NEURAL NETWORKS, 2018, 101 :68-78
[43]   Rapid Reconstruction of Acoustic Holograms Using Improved IU-Net With Adaptive Learning Rate Scheduling [J].
Cui, Mingzhe ;
Li, Yang ;
Wang, Xuewei ;
Wang, Jia ;
Rong, Haoxuan .
IEEE ACCESS, 2024, 12 :178199-178208
[44]   Adaptive Traffic Signal Control Method Based on Offline Reinforcement Learning [J].
Wang, Lei ;
Wang, Yu-Xuan ;
Li, Jian-Kang ;
Liu, Yi ;
Pi, Jia-Tian .
APPLIED SCIENCES-BASEL, 2024, 14 (22)
[45]   A Channel Adaptive WiFi Indoor Localization Method based on Deep Learning [J].
Hao, Lifei ;
Huang, Baoqi ;
Hong, Hao ;
Jia, Bing ;
Li, Wuyungerile .
2021 IEEE WIRELESS COMMUNICATIONS AND NETWORKING CONFERENCE (WCNC), 2021,
[46]   Adaptive Road Extraction Method in Different Scene Based on Deep Learning [J].
Ding Z.-L. ;
Hu Y.-H. ;
Gong J.-W. ;
Xiong G.-M. ;
Lü C. .
Beijing Ligong Daxue Xuebao/Transaction of Beijing Institute of Technology, 2019, 39 (11) :1133-1137
[47]   Adaptive Construction Method for Binary Measurement Matrix Based on Deep Learning [J].
Han Jiefei ;
Lian Bobo ;
Sun Liying .
LASER & OPTOELECTRONICS PROGRESS, 2021, 58 (22)
[48]   An Adaptive Medical Ultrasound Images Despeckling Method Based on Deep Learning [J].
Fu Xiaowei ;
Yang Xuefei ;
Chen Fang ;
Li Xi .
JOURNAL OF ELECTRONICS & INFORMATION TECHNOLOGY, 2020, 42 (07) :1782-1789
[49]   An Improved Infrared and Visible Image Fusion Using an Adaptive Contrast Enhancement Method and Deep Learning Network with Transfer Learning [J].
Bhutto, Jameel Ahmed ;
Tian, Lianfang ;
Du, Qiliang ;
Sun, Zhengzheng ;
Yu, Lubin ;
Soomro, Toufique Ahmed .
REMOTE SENSING, 2022, 14 (04)
[50]   Intelligent seismic facies classification based on an improved deep learning method [J].
Yan X. ;
Gu H. ;
Luo H. ;
Yan Y. .
Shiyou Diqiu Wuli Kantan/Oil Geophysical Prospecting, 2020, 55 (06) :1169-1177