Fast training method of deep-learning model fused with prior knowledge

被引：0

作者：

Wang P. ^{[1
]}

He M. ^{[2
]}

Wang H. ^{[1
]}

机构：

[1] College of Computer Science and Technology, Harbin Engineering University, Harbin

[2] College of Computer Science and Technology, Heilongjiang University of Science and Technology, Harbin

来源：

Harbin Gongcheng Daxue Xuebao/Journal of Harbin Engineering University | 2021年 / 42卷 / 04期

关键词：

Deep learning; Gradient descent; Loss shock; Neural network; Parameter isomorphism; Prior knowledge; Regression model; Training optimization;

D O I：

10.11990/jheu.202005025

中图分类号：

学科分类号：

摘要：

This paper proposes a new method of optimization while training the neural network to improve the training efficiency of deep learning neural networks based on the prior knowledge of model training. Convolutional layer parameters of the neural network multiple iterations are used to train a regression model using deep learning and to guide changes in neural network parameters. Experiments show that the proposed method can reduce the vibration caused by the model during the training process greatly without modifying the original network model structure. Moreover, this method can reduce the model training time by more than 10% while maintaining the model classification accuracy. The deeper the neural network applied, the more obvious the time optimization effect will be. Copyright ©2021 Journal of Harbin Engineering University.

引用

页码：561 / 566

页数：5

共 14 条

[1]

DAUPHIN Y N, PASCANU R, GULCEHRE C, Et al., Identifying and attacking the saddle point problem in high-dimensional non-convex optimization, Proceedings of the 27th International Conference on Neural Information Processing Systems, pp. 2933-2941, (2014)

[2]

HINTON G E, OSINDERO S, THE Y W., A fast learning algorithm for deep belief nets, Neural computation, 18, 7, pp. 1527-1554, (2006)

[3]

HE Kaiming, ZHANG Xiangyu, REN Shaoqing, Et al., Delving deep into rectifiers: surpassing human-level performance on ImageNet classification, Proceedings of the 2015 IEEE International Conference on Computer Vision, pp. 1026-1034, (2015)

[4]

GLOROT X, BENGIO Y., Understanding the difficulty of training deep feedforward neural networks, Proceedings of the 13th International Conference on Artificial Intelligence and Statistics, pp. 249-256, (2010)

[5]

KINGMA D P, BA J L., Adam: a method for stochastic optimization, Proceedings of the 3rd International Conference on Learning Representations, (2015)

[6]

DUCHI J, HAZAN E, SINGER Y., Adaptive subgradient methods for online learning and stochastic optimization, The Journal of machine learning research, 12, 7, pp. 2121-2159, (2011)

[7]

GOODFELLOW I, BENGIO Y, COURVILLE A., Deep learning, (2016)

[8]

IOFFE S, SZEGEDY C., Batch normalization: accelerating deep network training by reducing internal covariate shift, Proceedings of the 32nd International Conference on International Conference on Machine Learning, pp. 448-456, (2015)

[9]

QIU Jianing, WANG Xiaoxia, WANG Peng, Et al., Research on convolutional neural network algorithm combined with transfer learning model, Computer engineering and applications, 56, 5, pp. 43-48, (2020)

[10]

LI Maoying, YANG Liu, HU Qinghua, A survey on theories and algorithms about homogeneous transfer learning, Journal of Nanjing University of Information Science and Technology (natural science edition), 11, 3, pp. 269-277, (2019)

← 1 2 →