An Algorithm Based on Modified Momentum Using Restricted Boltzmann Machine

被引：0
作者：
Shen H.-H. [1 ,2 ,3 ]
Liu G.-W. [2 ]
Fu L.-H. [1 ]
Liu Z.-H. [1 ]
Li H.-W. [1 ,3 ]
机构：
[1] School of Mathematics and Physics, China University of Geosciences, Wuhan, 430074, Hubei
[2] School of Statistics & Information Management, Hubei University of Economics, Wuhan, 430205, Hubei
[3] Hubei Subsurface Multi-scale Imaging Key Laboratory, China University of Geosciences, Wuhan
来源：
Tien Tzu Hsueh Pao/Acta Electronica Sinica | 2019年 / 47卷 / 09期
关键词：
Deep learning; Generalization ability; Gibbs sampling; Gradient approximation algorithm; Momentum acceleration; Restricted Boltzmann machine; Unsupervised learning;
D O I：
10.3969/j.issn.0372-2112.2019.09.020
中图分类号：
学科分类号：
摘要：
Restricted Boltzmann machine (RBM) is a stochastic neural network and probabilistic graphical model, which is one of the most effective unsupervised learning methods in deep learning. Focusing on the gradient approximation algorithm of RBM insensitivity to momentum acceleration and recognition effectiveness, we propose the algorithm based on modified momentum using RBM. Combined with the gradient approximation algorithm of RBM, when the rule to update the hidden states adopted the probability value instead of sampling a binary value, leading to the undesirable recognition effect and limited momentum acceleration. Therefore, we modify the updating rule of the hidden bias to avoid these problems. Simultaneously, we use the rapidly ascending momentum method to improve learning speed in the RBM pre-training phase. An improved slowly descending momentum method is also used in the fine-tuning stage to accurately find the best point, which is far from becoming trapped in poor local optima and improves the classification effect. Through the recognition experiments on MNIST dataset, extended Yale B and CMU-PIE face dataset, the achieved results show that the proposed algorithm can enhance the computation efficiency and improve the generalization ability of networks. The algorithm not only extends the application field of RBM, but also provides a new research idea and reference for the application method of deep learning. © 2019, Chinese Institute of Electronics. All right reserved.
引用
收藏
页码：1957 / 1964
页数：7
相关论文
共 31 条
  • [1] Hinton G.E., Training products of experts by minimizing contrastive divergence, Neural Computation, 14, 8, pp. 1711-1800, (2002)
  • [2] Hinton G.E., Osindero S., Teh Y.W., A fast learning algorithm for deep belief nets, Neural Computation, 18, 7, pp. 1527-1554, (2006)
  • [3] Goodfellow I., Bengio Y., Courville A.
  • [4] Polyak T., Some methods of speeding up the convergence of iteration methods, USSR Computational Mathematics and Mathematical Physics, 4, 5, pp. 1-17, (1964)
  • [5] Rumelhart D.E., Hinton G.E., Williams R.J., Learning representations by back-propagating errors, Nature, 323, pp. 533-536, (1986)
  • [6] Qian N., On the momentum term in gradient descent learning algorithms, Neural Networks, 12, 1, pp. 145-151, (1999)
  • [7] Attoh-Okine N.O., Analysis of learning rate and momentum term in back-propagation neural network algorithm trained to predict pavement performance, Advances in Engineering Software, 30, 4, pp. 291-302, (1999)
  • [8] Mayraz G., Hinton G.E., Recognizing handwritten digits using hierarchical products of experts, IEEE Transactions on Pattern Analysis and Machine Intelligence, 24, 2, pp. 189-197, (2002)
  • [9] Hinton G.E., APractical Guide to Training Restricted Boltzmann Machines, pp. 599-619, (2012)
  • [10] Hamid N.A., Nawi N.M., Ghazali R., Mohd Salleh M.N., Accelerating learning performance of back propagation algorithm by using adaptive gain together with adaptive momentum and adaptive learning rate on classification problems, International Journal of Software Engineering and Its Applications, 5, 4, pp. 31-44, (2011)
1234