arXiv Analytics

Sign in

arXiv:2202.03841 [cs.LG]AbstractReferencesReviewsResources

Width is Less Important than Depth in ReLU Neural Networks

Gal Vardi, Gilad Yehudai, Ohad Shamir

Published 2022-02-08Version 1

We solve an open question from Lu et al. (2017), by showing that any target network with inputs in $\mathbb{R}^d$ can be approximated by a width $O(d)$ network (independent of the target network's architecture), whose number of parameters is essentially larger only by a linear factor. In light of previous depth separation theorems, which imply that a similar result cannot hold when the roles of width and depth are interchanged, it follows that depth plays a more significant role than width in the expressive power of neural networks. We extend our results to constructing networks with bounded weights, and to constructing networks with width at most $d+2$, which is close to the minimal possible width due to previous lower bounds. Both of these constructions cause an extra polynomial factor in the number of parameters over the target network. We also show an exact representation of wide and shallow networks using deep and narrow networks which, in certain cases, does not increase the number of parameters over the target network.

Related articles: Most relevant | Search more
arXiv:1809.07122 [cs.LG] (Published 2018-09-19)
Capacity Control of ReLU Neural Networks by Basis-path Norm
arXiv:2306.13264 [cs.LG] (Published 2023-06-23)
FedSelect: Customized Selection of Parameters for Fine-Tuning during Personalized Federated Learning
arXiv:2405.15006 [cs.LG] (Published 2024-05-23)
Path-metrics, pruning, and generalization