arXiv:2003.09671 [cs.LG]AbstractReferencesReviewsResources
On Information Plane Analyses of Neural Network Classifiers -- A Review
Published 2020-03-21Version 1
We review the current literature concerned with information plane analyses of neural network classifiers. While the underlying information bottleneck theory and the claim that information-theoretic compression is causally linked to generalization are plausible, empirical evidence was found to be both supporting and conflicting. We review this evidence together with a detailed analysis how the respective information quantities were estimated. Our analysis suggests that compression visualized in information planes is not information-theoretic, but is rather compatible with geometric compression of the activations.
Comments: 8 pages; under review
Related articles: Most relevant | Search more
arXiv:1911.12780 [cs.LG] (Published 2019-11-28)
Detection and Mitigation of Rare Subclasses in Neural Network Classifiers
arXiv:2105.04405 [cs.LG] (Published 2021-05-07)
A Critical Review of Information Bottleneck Theory and its Applications to Deep Learning
arXiv:2305.11387 [cs.LG] (Published 2023-05-19)
Justices for Information Bottleneck Theory