September
2026
Toward Human–Model Collaboration: From Model Editing to Data Attribution
Authors:
Abstract:
In standard machine learning pipelines, the relationship between human creativity and a generative model is strictly divided. Human data shapes the model during training, but once training is complete, that link is cut off. The model's parameters encode this visual knowledge, but they remain largely inaccessible to the creators using the system. A text prompt can steer a single output in text-to-image models, but it cannot edit the model's underlying priors for deeper creative control, nor can it trace an output back to its original training sources. This limits two important parts of visual creation. A creator’s ideas may be constrained by the model’s existing capabilities. For example, it is difficult to ask a generated cat to have a rabbit-like face shape because the model has never learned that combination from its training data. At the same time, attributing sources of inspiration is a common, and often essential, practice in creative work, but the model’s outputs provide little information about which training images influenced them.
To reconnect creators with models in both directions, this thesis studies the model’s learned parameters to change and trace learned visual relationships. First, model editing updates the parameters so that a creator’s sparse visual example becomes a reusable rule for future samples. Second, training data attribution analyzes targeted parameter changes to estimate which training images influenced a generated output.
Part I: Editing Generative Models. A creator often has a visual idea that a text prompt cannot express, such as a specific pose, shape, or change to a facial feature. We turn a small number of sparse visual demonstrations, including sketches and warped examples, into a rule that a pretrained generator applies to new samples. The edited model preserves the appearance, diversity, and ability to generate new instances of the original model. We then extend model editing from GANs to text-to-image diffusion models.
Part II: Training Data Attribution. An influential training image is one whose removal would make the model less able to produce a given output. Iteratively removing images and retraining the model to identify influential sets is infeasible, while attribution methods typically lack ground truth for evaluation. We first build a benchmark with known influence by fine-tuning a model on new concepts. We then estimate influence by unlearning a synthesized image and measuring which training images become less represented in the model alongside that image. Removing the top 500 images ranked by this method, about 0.42% of the training set, is sufficient to prevent a retrained model from reproducing the query. To make the method usable at scale, we distill its scores into a feature space that supports fast retrieval. Finally, to analyze the visual relationships between influential training images and generated outputs, we introduce a text-prompted perceptual metric that conditions visual similarity on a named visual aspect. The metric identifies the aspects in which an influential training image resembles a generated output.
To reconnect creators with models in both directions, this thesis studies the model’s learned parameters to change and trace learned visual relationships. First, model editing updates the parameters so that a creator’s sparse visual example becomes a reusable rule for future samples. Second, training data attribution analyzes targeted parameter changes to estimate which training images influenced a generated output.
Part I: Editing Generative Models. A creator often has a visual idea that a text prompt cannot express, such as a specific pose, shape, or change to a facial feature. We turn a small number of sparse visual demonstrations, including sketches and warped examples, into a rule that a pretrained generator applies to new samples. The edited model preserves the appearance, diversity, and ability to generate new instances of the original model. We then extend model editing from GANs to text-to-image diffusion models.
Part II: Training Data Attribution. An influential training image is one whose removal would make the model less able to produce a given output. Iteratively removing images and retraining the model to identify influential sets is infeasible, while attribution methods typically lack ground truth for evaluation. We first build a benchmark with known influence by fine-tuning a model on new concepts. We then estimate influence by unlearning a synthesized image and measuring which training images become less represented in the model alongside that image. Removing the top 500 images ranked by this method, about 0.42% of the training set, is sufficient to prevent a retrained model from reproducing the query. To make the method usable at scale, we distill its scores into a feature space that supports fast retrieval. Finally, to analyze the visual relationships between influential training images and generated outputs, we introduce a text-prompted perceptual metric that conditions visual similarity on a named visual aspect. The metric identifies the aspects in which an influential training image resembles a generated output.