arXiv Analytics

Sign in

arXiv:2007.10618 [cs.CV]AbstractReferencesReviewsResources

Novel View Synthesis on Unpaired Data by Conditional Deformable Variational Auto-Encoder

Mingyu Yin, Li Sun, Qingli Li

Published 2020-07-21Version 1

Novel view synthesis often needs the paired data from both the source and target views. This paper proposes a view translation model under cVAE-GAN framework without requiring the paired data. We design a conditional deformable module (CDM) which uses the view condition vectors as the filters to convolve the feature maps of the main branch in VAE. It generates several pairs of displacement maps to deform the features, like the 2D optical flows. The results are fed into the deformed feature based normalization module (DFNM), which scales and offsets the main branch feature, given its deformed one as the input from the side branch. Taking the advantage of the CDM and DFNM, the encoder outputs a view-irrelevant posterior, while the decoder takes the code drawn from it to synthesize the reconstructed and the viewtranslated images. To further ensure the disentanglement between the views and other factors, we add adversarial training on the code. The results and ablation studies on MultiPIE and 3D chair datasets validate the effectiveness of the framework in cVAE and the designed module.

Related articles: Most relevant | Search more
arXiv:2312.08892 [cs.CV] (Published 2023-12-14)
VaLID: Variable-Length Input Diffusion for Novel View Synthesis
arXiv:2303.15880 [cs.CV] (Published 2023-03-28)
Novel View Synthesis of Humans using Differentiable Rendering
arXiv:2006.15617 [cs.CV] (Published 2020-06-28)
Shadow Removal by a Lightness-Guided Network with Training on Unpaired Data