arXiv Analytics

Sign in

arXiv:2211.08610 [cs.CV]AbstractReferencesReviewsResources

CoNFies: Controllable Neural Face Avatars

Heng Yu, Koichiro Niinuma, Laszlo A. Jeni

Published 2022-11-16Version 1

Neural Radiance Fields (NeRF) are compelling techniques for modeling dynamic 3D scenes from 2D image collections. These volumetric representations would be well suited for synthesizing novel facial expressions but for two problems. First, deformable NeRFs are object agnostic and model holistic movement of the scene: they can replay how the motion changes over time, but they cannot alter it in an interpretable way. Second, controllable volumetric representations typically require either time-consuming manual annotations or 3D supervision to provide semantic meaning to the scene. We propose a controllable neural representation for face self-portraits (CoNFies), that solves both of these problems within a common framework, and it can rely on automated processing. We use automated facial action recognition (AFAR) to characterize facial expressions as a combination of action units (AU) and their intensities. AUs provide both the semantic locations and control labels for the system. CoNFies outperformed competing methods for novel view and expression synthesis in terms of visual and anatomic fidelity of expressions.

Related articles:
arXiv:2302.01330 [cs.CV] (Published 2023-02-02)
SceneDreamer: Unbounded 3D Scene Generation from 2D Image Collections
arXiv:2309.15830 [cs.CV] (Published 2023-09-27)
OrthoPlanes: A Novel Representation for Better 3D-Awareness of GANs
arXiv:2405.14276 [cs.CV] (Published 2024-05-23, updated 2024-05-24)
D-MiSo: Editing Dynamic 3D Scenes using Multi-Gaussians Soup