Wide neural networks with bottlenecks are deep gaussian processes

被引：0

作者：

Agrawal, Devanshu ^{[1
]}

Papamarkou, Theodore ^{[2
]}

Hinkle, Jacob ^{[2
]}

机构：

[1] Bredesen Center, University of Tennessee, Knoxville,TN,37996-3394, United States

[2] Computational Sciences and Engineering Division, Oak Ridge National Lab, Oak Ridge,TN,37830-8050, United States

来源：

Journal of Machine Learning Research | 2020年 / 21卷

关键词：

There has recently been much work on the wide limitof neural networks; where Bayesian neural networks (BNNs) are shown to converge to a Gaussian process (GP) as all hidden layers are sent to in_nite width. However; these results do not apply to architectures that require one or more of the hidden layers to remain narrow. In this paper; we consider the wide limit of BNNs where some hidden layers; called bottlenecks; are held at finite width. The result is a composition of GPs that we term a bottleneck neural network Gaussian process(bottleneck NNGP). Although intuitive; the subtlety of the proof is in showing that the wide limit of a composition of networks is in fact the composition of the limiting GPs. We also analyze theoretically a single-bottleneck NNGP; finding that the bottleneck induces dependence between the outputs of a multi-output network that persists through extreme post-bottleneck depths; and prevents the kernel of the network from losing discriminative power at extreme post-bottleneck depths. © 2020 Devanshu Agrawal; Theodore Papamarkou; Jacob Hinkle;

D O I：

暂无

中图分类号：

学科分类号：

摘要：

引用