July
2026
Imagining Before Acting: Learning and Using World Models for Robot Manipulation
Authors:
Abstract:
Imitation learning has become the dominant paradigm for robot manipulation, yet it suffers from two structural weaknesses: it can exploit only scarce, expensive expert demonstrations, and its policies act reactively, with no mechanism for predicting or evaluating the outcomes of their own actions. This thesis argues that both weaknesses share a single remedy, a learned world model, and develops that remedy in two directions: learning world models from the cheapest available data, and using them to give imitation policies foresight at decision time. First, ViPRA (Video Prediction for Robot Actions) turns a video prediction model into a continuous-control robot policy without large-scale action labels: it extracts motion-centric latent actions from unlabeled human and robot videos, pretrains a video--language model to jointly predict future observations and latent action chunks, and grounds the latents into smooth, high-frequency control with a chunked flow-matching decoder using only 100--200 teleoperated demonstrations, outperforming vision-language-action baselines by 16% on the SIMPLER benchmark and 13% on real-world Franka manipulation. Second, ForesightIL augments a fixed imitation policy with test-time planning: candidate action sequences sampled from the policy are rolled forward in a frozen latent world model and ranked by a temporally straightened, goal-conditioned value function whose latent distances count steps-to-go rather than visual similarity; an uncertainty-gated planner then predicts the world model's own error from its internal activations and escalates only untrustworthy steps to full planning, matching always-plan performance at roughly half the planning rate and up to twice lower cost, while improving substantially over the behavior-cloning baseline across four simulated suites and two real-robot tasks. Finally, the thesis summarizes Nano World Models, a minimalist, reproducible substrate for studying world-model design, whose findings, notably that world models can silently ignore actions while producing excellent predictions, sharpen the design principles underlying the other two projects. Together, these contributions trace the lifecycle of a world model for manipulation: acquiring predictive knowledge from cheap data, deploying it economically at decision time, and understanding the choices that make it work.
Notes:
copied = false, 2000);
">
@mastersthesis{Pan-2026-88329,
author = {Hengkai Pan},
title = {Imagining Before Acting: Learning and Using World Models for Robot Manipulation},
year = {2026},
month = {July},
school = {Carnegie Mellon University},
address = {Pittsburgh, PA},
number = {CMU-RI-TR-26-77},
keywords = {world models, imitation learning, planning},
}
author = {Hengkai Pan},
title = {Imagining Before Acting: Learning and Using World Models for Robot Manipulation},
year = {2026},
month = {July},
school = {Carnegie Mellon University},
address = {Pittsburgh, PA},
number = {CMU-RI-TR-26-77},
keywords = {world models, imitation learning, planning},
}