On size-independent sample complexity of ReLU networks

被引：0

作者：

Sellke, Mark ^{[1
]}

机构：

[1] Harvard Stat, Cambridge, MA USA

来源：

INFORMATION PROCESSING LETTERS | 2024年 / 186卷

关键词：

Neural networks; Rademacher complexity; Generalization; Theory of computation;

D O I：

10.1016/j.ipl.2024.106482

中图分类号：

TP [自动化技术、计算机技术];

学科分类号：

0812 ;

摘要：

We study the sample complexity of learning ReLU neural networks from the point of view of generalization. Given norm constraints on the weight matrices, a common approach is to estimate the Rademacher complexity of the associated function class. Previously [9] obtained a bound independent of the network size (scaling with a product of Frobenius norms) except for a factor of the square -root depth. We give a refinement which often has no explicit depth -dependence at all.

引用

页数：3

共 50 条

[41] DISCUSSION OF: "NONPARAMETRIC REGRESSION USING DEEP NEURAL NETWORKS WITH RELU ACTIVATION FUNCTION"
Kutyniok, Gitta
ANNALS OF STATISTICS, 2020, 48 (04) : 1902 - 1905
[42] Mean-field Analysis of Piecewise Linear Solutions for Wide ReLU Networks
Shevchenko, Alexander
Kungurtsev, Vyacheslav
Mondelli, Marco
JOURNAL OF MACHINE LEARNING RESEARCH, 2022, 23
[43] Nonparametric Regression Using Over-parameterized Shallow ReLU Neural Networks
Yang, Yunfei
Zhou, Ding-Xuan
JOURNAL OF MACHINE LEARNING RESEARCH, 2024, 25 : 1 - 35
[44] Classification of Data Generated by Gaussian Mixture Models Using Deep ReLU Networks
Zhou, Tian-Yi
Huo, Xiaoming
JOURNAL OF MACHINE LEARNING RESEARCH, 2024, 25 : 1 - 54
[45] Expressive power of ReLU and step networks under floating-point operations
Park, Yeachan
Hwang, Geonho
Lee, Wonyeol
Park, Sejun
NEURAL NETWORKS, 2024, 175
[46] Gradient flow dynamics of shallow ReLU networks for square loss and orthogonal inputs
Boursier, Etienne
Pillaud-Vivien, Loucas
Flammarion, Nicolas
ADVANCES IN NEURAL INFORMATION PROCESSING SYSTEMS 35 (NEURIPS 2022), 2022,
[47] Information theoretic perspective on sample complexity
Pereg, Deborah
NEURAL NETWORKS, 2023, 167 : 445 - 449
[48] Strengthened Circle and Popov Criteria for the Stability Analysis of Feedback Systems With ReLU Neural Networks
Richardson, Carl R. R.
Turner, Matthew C. C.
Gunn, Steve R. R.
IEEE CONTROL SYSTEMS LETTERS, 2023, 7 : 2635 - 2640
[49] Hidden unit specialization in layered neural networks: ReLU vs. sigmoidal activation
Oostwal, Elisa
Straat, Michiel
Biehl, Michael
PHYSICA A-STATISTICAL MECHANICS AND ITS APPLICATIONS, 2021, 564
[50] Synthesizing ReLU Neural Networks with Two Hidden Layers as Barrier Certificates for Hybrid Systems
Zhao, Qingye
Chen, Xin
Zhang, Yifan
Sha, Meng
Yang, Zhengfeng
Lin, Wang
Tang, Enyi
Chen, Qiguang
Li, Xuandong
HSCC2021: PROCEEDINGS OF THE 24TH INTERNATIONAL CONFERENCE ON HYBRID SYSTEMS: COMPUTATION AND CONTROL (PART OF CPS-IOT WEEK), 2021,

← 1 2 3 4 5 →