arXiv Analytics

Sign in

arXiv:1906.05895 [cs.LG]AbstractReferencesReviewsResources

Learning to Forget for Meta-Learning

Sungyong Baik, Seokil Hong, Kyoung Mu Lee

Published 2019-06-13Version 1

Few-shot learning is a challenging problem where the system is required to achieve generalization from only few examples. Meta-learning tackles the problem by learning prior knowledge shared across a distribution of tasks, which is then used to quickly adapt to unseen tasks. Model-agnostic meta-learning (MAML) algorithm formulates prior knowledge as a common initialization across tasks. However, forcibly sharing an initialization brings about conflicts between tasks and thus compromises the quality of the initialization. In this work, by observing that the extent of compromise differs among tasks and between layers of a neural network, we propose a new initialization idea that employs task-dependent layer-wise attenuation, which we call selective forgetting. The proposed attenuation scheme dynamically controls how much of prior knowledge each layer will exploit for a given task. The experimental results demonstrate that the proposed method mitigates the conflicts and provides outstanding performance as a result. We further show that the proposed method, named L2F, can be applied and improve other state-of-the-art MAML-based frameworks, illustrating its generalizability.

Related articles: Most relevant | Search more
arXiv:2209.07263 [cs.LG] (Published 2022-09-15)
Robustness in deep learning: The good (width), the bad (depth), and the ugly (initialization)
arXiv:2009.08576 [cs.LG] (Published 2020-09-18)
Pruning Neural Networks at Initialization: Why are We Missing the Mark?
arXiv:1907.06065 [cs.LG] (Published 2019-07-13)
Bringing Giant Neural Networks Down to Earth with Unlabeled Data