Restricted Boltzmann machine: Recent advances and mean-field theory*

被引:46
作者
Decelle, Aurelien [1 ,2 ,3 ]
Furtlehner, Cyril [2 ,3 ]
机构
[1] Univ Complutense, Dept Fis Teor 1, Madrid 28040, Spain
[2] Univ Paris Saclay, INRIA Saclay, TAU Team, F-91405 Orsay, France
[3] Univ Paris Saclay, LISN, F-91405 Orsay, France
关键词
restricted Boltzmann machine (RBM); machine learning; statistical physics; STATISTICAL-MECHANICS; NEURAL-NETWORKS; MODEL; STORAGE;
D O I
10.1088/1674-1056/abd160
中图分类号
O4 [物理学];
学科分类号
0702 ;
摘要
This review deals with restricted Boltzmann machine (RBM) under the light of statistical physics. The RBM is a classical family of machine learning (ML) models which played a central role in the development of deep learning. Viewing it as a spin glass model and exhibiting various links with other models of statistical physics, we gather recent results dealing with mean-field theory in this context. First the functioning of the RBM can be analyzed via the phase diagrams obtained for various statistical ensembles of RBM, leading in particular to identify a compositional phase where a small number of features or modes are combined to form complex patterns. Then we discuss recent works either able to devise mean-field based learning algorithms; either able to reproduce generic aspects of the learning process from some ensemble dynamics equations or/and from linear stability arguments.
引用
收藏
页数:24
相关论文
共 89 条
[41]   Variational mean-field theory for training restricted Boltzmann machines with binary synapses [J].
Huang, Haiping .
PHYSICAL REVIEW E, 2020, 102 (03)
[42]   Role of zero synapses in unsupervised feature learning [J].
Huang, Haiping .
JOURNAL OF PHYSICS A-MATHEMATICAL AND THEORETICAL, 2018, 51 (08)
[43]   Statistical mechanics of unsupervised feature learning in a restricted Boltzmann machine with binary synapses [J].
Huang, Haiping .
JOURNAL OF STATISTICAL MECHANICS-THEORY AND EXPERIMENT, 2017,
[44]   Unsupervised feature learning from finite data by message passing: Discontinuous versus continuous phase transition [J].
Huang, Haiping ;
Toyoizumi, Taro .
PHYSICAL REVIEW E, 2016, 94 (06)
[45]   Advanced mean-field theory of the restricted Boltzmann machine [J].
Huang, Haiping ;
Toyoizumi, Taro .
PHYSICAL REVIEW E, 2015, 91 (05)
[46]   Exchange Monte Carlo method and application to spin glass simulations [J].
Hukushima, K ;
Nemoto, K .
JOURNAL OF THE PHYSICAL SOCIETY OF JAPAN, 1996, 65 (06) :1604-1608
[47]   Independent component analysis:: algorithms and applications [J].
Hyvärinen, A ;
Oja, E .
NEURAL NETWORKS, 2000, 13 (4-5) :411-430
[48]  
Karakida R., 2014, DEEP LEARN REPR LEAR
[49]   Dynamical analysis of contrastive divergence learning: Restricted Boltzmann machines with Gaussian visible units [J].
Karakida, Ryo ;
Okada, Masato ;
Amari, Shun-ichi .
NEURAL NETWORKS, 2016, 79 :78-87
[50]   INFINITE-RANGED MODELS OF SPIN-GLASSES [J].
KIRKPATRICK, S ;
SHERRINGTON, D .
PHYSICAL REVIEW B, 1978, 17 (11) :4384-4403