Abstract:
Recent advances in generative AI are reshaping how visual media is produced. These tools are beginning to make content creation accessible beyond expert practitioners and have the potential to support highly personalized use cases for everyday users. For example, they can enable educators to design custom illustrations for their curricula, small-scale retailers to visualize products in new settings without professional photoshoots, and independent creators to streamline artistic and design workflows. Realizing this potential, however, requires generative models that are not only controllable and customizable to user needs but also safe to deploy, such as by having support mechanisms for removing undesirable concepts like copyrighted styles, memorized identities, or unsafe content.
In this dissertation, we focus on customizing pretrained generative models, primarily text-to-image diffusion models, for such downstream tasks. A central challenge is that paired input-output supervision for these tasks is often scarce, expensive, or unavailable. To address this, we explore strategies for obtaining useful supervision under progressively weaker data assumptions: from few-shot optimization, to synthetic paired-data construction, to feedback from pretrained vision understanding models.
Part I studies optimization-based customization of text-to-image models for two complementary goals: acquiring new concepts from a few reference images and removing undesirable concepts, such as copyrighted styles or memorized identities. We first analyze task-specific fine-tuning and find that cross-attention layers undergo the most significant relative weight changes. Building on this observation, we develop \textit{Custom Diffusion}, which learns new concepts by fine-tuning only this targeted subset of parameters. The same principle also enables concept removal, combined with the objective of steering generations associated with a target concept that we want to remove toward a broader anchor class.
Part II moves from per-instance optimization, which can be time consuming, to scalable dataset construction. We present a synthetic dataset construction pipeline for generating images of the same object across diverse contexts and viewpoints, enabling large-scale training for applications such as instant product recontextualization. The key insight is that, with appropriate constraints, generative models can synthesize identity-consistent examples at a scale that would be difficult or expensive to achieve through real-world data collection.
While synthetic data enables scalable training, it also introduces new dependencies: the resulting model can inherit artifacts, biases, or limitations from the data-generation pipeline, and the dataset may become outdated as stronger generative models emerge.
Part III, therefore, investigates the use of pretrained vision encoders as a source of differentiable supervision for customizing generative image models. We demonstrate this paradigm across two settings. First, we show that ensembles of pretrained vision encoders, such as CLIP and DINO, can serve as effective discriminators for GAN training, enabling training with a substantially limited dataset. Second, we extend this principle to diffusion-based image editing, converting a text-to-image model into a reference-image-conditioned editing model without any paired before-and-after editing data. We use vision-language models' understanding capabilities to provide differentiable feedback for the editing task. In addition, to ensure generated images remain on the real-image manifold, we apply a distribution matching loss.
Collectively, these contributions demonstrate that generative image models can be effectively customized for diverse downstream applications through targeted supervision strategies, overcoming the scarcity of paired data while preserving the general capabilities of the pretrained model.
In this dissertation, we focus on customizing pretrained generative models, primarily text-to-image diffusion models, for such downstream tasks. A central challenge is that paired input-output supervision for these tasks is often scarce, expensive, or unavailable. To address this, we explore strategies for obtaining useful supervision under progressively weaker data assumptions: from few-shot optimization, to synthetic paired-data construction, to feedback from pretrained vision understanding models.
Part I studies optimization-based customization of text-to-image models for two complementary goals: acquiring new concepts from a few reference images and removing undesirable concepts, such as copyrighted styles or memorized identities. We first analyze task-specific fine-tuning and find that cross-attention layers undergo the most significant relative weight changes. Building on this observation, we develop \textit{Custom Diffusion}, which learns new concepts by fine-tuning only this targeted subset of parameters. The same principle also enables concept removal, combined with the objective of steering generations associated with a target concept that we want to remove toward a broader anchor class.
Part II moves from per-instance optimization, which can be time consuming, to scalable dataset construction. We present a synthetic dataset construction pipeline for generating images of the same object across diverse contexts and viewpoints, enabling large-scale training for applications such as instant product recontextualization. The key insight is that, with appropriate constraints, generative models can synthesize identity-consistent examples at a scale that would be difficult or expensive to achieve through real-world data collection.
While synthetic data enables scalable training, it also introduces new dependencies: the resulting model can inherit artifacts, biases, or limitations from the data-generation pipeline, and the dataset may become outdated as stronger generative models emerge.
Part III, therefore, investigates the use of pretrained vision encoders as a source of differentiable supervision for customizing generative image models. We demonstrate this paradigm across two settings. First, we show that ensembles of pretrained vision encoders, such as CLIP and DINO, can serve as effective discriminators for GAN training, enabling training with a substantially limited dataset. Second, we extend this principle to diffusion-based image editing, converting a text-to-image model into a reference-image-conditioned editing model without any paired before-and-after editing data. We use vision-language models' understanding capabilities to provide differentiable feedback for the editing task. In addition, to ensure generated images remain on the real-image manifold, we apply a distribution matching loss.
Collectively, these contributions demonstrate that generative image models can be effectively customized for diverse downstream applications through targeted supervision strategies, overcoming the scarcity of paired data while preserving the general capabilities of the pretrained model.