arXiv Analytics

Sign in

arXiv:2211.17084 [cs.CV]AbstractReferencesReviewsResources

High-Fidelity Guided Image Synthesis with Latent Diffusion Models

Jaskirat Singh, Stephen Gould, Liang Zheng

Published 2022-11-30Version 1

Controllable image synthesis with user scribbles has gained huge public interest with the recent advent of text-conditioned latent diffusion models. The user scribbles control the color composition while the text prompt provides control over the overall image semantics. However, we note that prior works in this direction suffer from an intrinsic domain shift problem, wherein the generated outputs often lack details and resemble simplistic representations of the target domain. In this paper, we propose a novel guided image synthesis framework, which addresses this problem by modeling the output image as the solution of a constrained optimization problem. We show that while computing an exact solution to the optimization is infeasible, an approximation of the same can be achieved while just requiring a single pass of the reverse diffusion process. Additionally, we show that by simply defining a cross-attention based correspondence between the input text tokens and the user stroke-painting, the user is also able to control the semantics of different painted regions without requiring any conditional training or finetuning. Human user study results show that the proposed approach outperforms the previous state-of-the-art by over 85.32% on the overall user satisfaction scores. Project page for our paper is available at https://1jsingh.github.io/gradop.

Related articles: Most relevant | Search more
arXiv:2312.09792 [cs.CV] (Published 2023-12-15)
Latent Diffusion Models with Image-Derived Annotations for Enhanced AI-Assisted Cancer Diagnosis in Histopathology
Pedro Osorio et al.
arXiv:2308.12453 [cs.CV] (Published 2023-08-23)
Augmenting medical image classifiers with synthetic data from latent diffusion models
arXiv:2209.14697 [cs.CV] (Published 2022-09-29)
Creative Painting with Latent Diffusion Models