Hierarchical imitation learning has become an effective recipe for robotic manipulation: a high-level policy predicts a sub-goal end-effector pose, and a low-level policy produces the actions that realize it. This decomposition improves generalization to unseen objects, and it exposes an interpretable interface at the sub-goal. Yet the design of the high-level goal predictor is far from settled—what it should predict, how the low-level policy should consume it, what capabilities it must encode, and how a user might steer it all remain open. This thesis investigates these questions through three studies that share one object of study:
the high-level goal predictor.

First, on the MimicGen benchmark, we study what the high level should predict. We compare conditioning a 3D diffusion policy on dense predicted goal point clouds, produced by a cross-displacement model (TAX3D), against conditioning it on four predicted endeffector points from a weighted displacement model. The sparse four-point representation is a substantially more reliable conditioning signal across four differenttasks.

Second, on RLBench, we expand the capabilities of the four-point predictor and pair it with a motion planner in place of a learned low-level policy. We add language conditioning and RGB features, and additionally predict discrete gripper open/close actions and
collision-ignore decisions, reaching performance competitive with state-of-the-art keyframeaction methods. We identify a key limitation: because the high level regresses its sub-goal, it predicts the mean of valid targets and cannot represent multi-modal ones, especially when a target admits multiple valid orientations.

Third, in a sim-to-real setting, we make the high level steerable via prompting, letting a user disambiguate among multiple valid targets on the same object—for example, which drawer to open. The prompted high level transfers zero-shot to a real robot, and in simulation it matches the opening success rate of the original single-target policy while adding target selection.

Together, these studies map the design space of high-level goal prediction and offer concrete lessons about representation, capability, and controllability for hierarchical manipulation policies.