arXiv Analytics

Sign in

arXiv:1904.01326 [cs.CV]AbstractReferencesReviewsResources

HoloGAN: Unsupervised learning of 3D representations from natural images

Thu Nguyen-Phuoc, Chuan Li, Lucas Theis, Christian Richardt, Yong-Liang Yang

Published 2019-04-02Version 1

We propose a novel generative adversarial network (GAN) for the task of unsupervised learning of 3D representations from natural images. Most generative models rely on 2D kernels to generate images and make few assumptions about the 3D world. These models therefore tend to create blurry images or artefacts in tasks that require a strong 3D understanding, such as novel-view synthesis. HoloGAN instead learns a 3D representation of the world, and to render this representation in a realistic manner. Unlike other GANs, HoloGAN provides explicit control over the pose of generated objects through rigid-body transformations of the learnt 3D features. Our experiments show that using explicit 3D features enables HoloGAN to disentangle 3D pose and identity, which is further decomposed into shape and appearance, while still being able to generate images with similar or higher visual quality than other generative models. HoloGAN can be trained end-to-end from unlabelled 2D images only. Particularly, we do not require pose labels, 3D shapes, or multiple views of the same objects. This shows that HoloGAN is the first generative model that learns 3D representations from natural images in an entirely unsupervised manner.

Related articles: Most relevant | Search more
arXiv:1902.03361 [cs.CV] (Published 2019-02-09)
Image Decomposition and Classification through a Generative Model
arXiv:2106.05258 [cs.CV] (Published 2021-06-09)
Generative Models as a Data Source for Multiview Representation Learning
arXiv:1508.04035 [cs.CV] (Published 2015-08-17)
A Generative Model for Multi-Dialect Representation