arXiv Analytics

Sign in

arXiv:2007.15646 [cs.CV]AbstractReferencesReviewsResources

Rewriting a Deep Generative Model

David Bau, Steven Liu, Tongzhou Wang, Jun-Yan Zhu, Antonio Torralba

Published 2020-07-30Version 1

A deep generative model such as a GAN learns to model a rich set of semantic and physical rules about the target distribution, but up to now, it has been obscure how such rules are encoded in the network, or how a rule could be changed. In this paper, we introduce a new problem setting: manipulation of specific rules encoded by a deep generative model. To address the problem, we propose a formulation in which the desired rule is changed by manipulating a layer of a deep network as a linear associative memory. We derive an algorithm for modifying one entry of the associative memory, and we demonstrate that several interesting structural rules can be located and modified within the layers of state-of-the-art generative models. We present a user interface to enable users to interactively change the rules of a generative model to achieve desired effects, and we show several proof-of-concept applications. Finally, results on multiple datasets demonstrate the advantage of our method against standard fine-tuning methods and edit transfer algorithms.

Comments: ECCV 2020 (oral). Code at https://github.com/davidbau/rewriting. For videos and demos see https://rewriting.csail.mit.edu/
Categories: cs.CV, cs.GR, cs.LG
Subjects: I.2.6, I.3.3
Related articles: Most relevant | Search more
arXiv:2209.04747 [cs.CV] (Published 2022-09-10)
Diffusion Models in Vision: A Survey
arXiv:1611.09502 [cs.CV] (Published 2016-11-29)
Deep Quantization: Encoding Convolutional Activations with Deep Generative Model
arXiv:2308.04091 [cs.CV] (Published 2023-08-08)
From Unimodal to Multimodal: improving the sEMG-Based Pattern Recognition via deep generative models