Reference-Prompted Instance Segmentation for Growing Retail Catalogs

July 2026

Reference-Prompted Instance Segmentation for Growing Retail Catalogs

Authors:

Jeffrey Ke

Abstract:

Retail and warehouse perception systems work against a catalog that
never stops changing: a segmenter deployed today will eventually be
asked to find products that did not exist when it was trained. A closed-
set segmenter can only grow its vocabulary by widening its classification
head and fine-tuning it, which risks the classes it already handled, or
by retraining on the whole accumulated catalog, which costs more with
every product added. This thesis proposes reference-prompted segmen-
tation, which specifies the target product with a reference prompt, a
fronto-parallel view of the product, rather than a class label or a text de-
scription, so that a product’s identity enters the model as an input rather
than as a slot in its output. We fine-tune a reference-prompted segmenter
one new product at a time — twenty-seven sequential steps growing a
three-product catalog to thirty — using Learning without Forgetting,
and compare it against a closed-set segmenter given the identical recipe,
budget, and data. Our method retains its earlier products substantially
better than the closed-set baseline. Every dataset in this thesis comes
from isaac_datagen, a standalone Isaac Sim synthetic-data engine of-
fered as a reusable contribution in its own right. Under this protocol and
at this scale, the reference-prompted segmenter retains 5× the panoptic
quality of an equally-trained closed-set segmenter after sequentially ab-
sorbing 30 products (0.275 vs. 0.054 PQ), letting a perception system
keep pace with a growing catalog without ever retraining on all of it.

Notes:

@mastersthesis{Ke-2026-88338,
author = {Jeffrey Ke},
title = {Reference-Prompted Instance Segmentation for Growing Retail Catalogs},
year = {2026},
month = {July},
school = {Carnegie Mellon University},
address = {Pittsburgh, PA},
number = {CMU-RI-TR-92},
keywords = {instance segmentation, retail, reference-prompting, fine-tuning, less forgetting},
}
Copyright notice: This material is presented to ensure timely dissemination of scholarly and technical work. Copyright and all rights therein are retained by authors or by other copyright holders. All persons copying this information are expected to adhere to the terms and constraints invoked by each author's copyright. These works may not be reposted without the explicit permission of the copyright holder.