Imitation learning has emerged as a leading paradigm for teaching manipulation skills to robots, but its success
depends on the costly endeavour of collecting robot demonstrations through teleoperation. Generalizing to
novel objects, environments, and task variations typically requires massive datasets that are expensive to
scale. This thesis investigates an alternative lever: hierarchy—explicitly factorizing manipulation policies
into high-level reasoning about where to go and low-level control about how to get there—and asks whether
such a factorization can both improve data efficiency from robot demonstrations alone and enable the use of
cheaper sources of data such as human video.
We present GHOST, a framework for learning visuomotor manipulation policies that generalize beyond
the training distribution. GHOST factorizes control into (i) a high-level policy that predicts the next sub-goal
as a distribution over 3D end-effector poses from multi-view RGB-D observations, and (ii) a low-level
goal-conditioned controller that executes embodiment-specific actions. To condition image-based policies
on 3D goals, we introduce a simple spatial interface that projects predicted goals into the image plane and
represents them as end-effector heatmaps. Across a suite of manipulation tasks, this hierarchical factorization
consistently improves performance and robustness compared to a flat Diffusion Policy.
We further show that this hierarchical interface makes it easy to incorporate human demonstrations
without relying on (noisy) action retargeting. Because sub-goals are largely embodiment-agnostic, we train
the high-level policy on human video to specify how learned skills should be applied and composed, while
keeping the low-level policy trained purely on robot data. This hierarchy enables adaptation to novel objects
and task variations using only a small number of human demonstrations.
We evaluate GHOST on a suite of manipulation tasks spanning pick-and-place, cloth folding, and tool-use,
demonstrating both improved in-distribution performance and meaningful cross-embodiment generalization
from human video. We conduct ablations to further isolate the contributions of the hierarchical factorization,
and identify the visual domain gap between human and robot observations as a significant bottleneck for
long-horizon generalization.