Imagining Before Acting: Learning and Using World Models for Robot Manipulation

July 2026

Imagining Before Acting: Learning and Using World Models for Robot Manipulation

Authors:

Hengkai Pan

Abstract:

Imitation learning has become the dominant paradigm for robot manipulation, yet it suffers from two structural weaknesses: it can exploit only scarce, expensive expert demonstrations, and its policies act reactively, with no mechanism for predicting or evaluating the outcomes of their own actions. This thesis argues that both weaknesses share a single remedy, a learned world model, and develops that remedy in two directions: learning world models from the cheapest available data, and using them to give imitation policies foresight at decision time. First, ViPRA (Video Prediction for Robot Actions) turns a video prediction model into a continuous-control robot policy without large-scale action labels: it extracts motion-centric latent actions from unlabeled human and robot videos, pretrains a video--language model to jointly predict future observations and latent action chunks, and grounds the latents into smooth, high-frequency control with a chunked flow-matching decoder using only 100--200 teleoperated demonstrations, outperforming vision-language-action baselines by 16% on the SIMPLER benchmark and 13% on real-world Franka manipulation. Second, ForesightIL augments a fixed imitation policy with test-time planning: candidate action sequences sampled from the policy are rolled forward in a frozen latent world model and ranked by a temporally straightened, goal-conditioned value function whose latent distances count steps-to-go rather than visual similarity; an uncertainty-gated planner then predicts the world model's own error from its internal activations and escalates only untrustworthy steps to full planning, matching always-plan performance at roughly half the planning rate and up to twice lower cost, while improving substantially over the behavior-cloning baseline across four simulated suites and two real-robot tasks. Finally, the thesis summarizes Nano World Models, a minimalist, reproducible substrate for studying world-model design, whose findings, notably that world models can silently ignore actions while producing excellent predictions, sharpen the design principles underlying the other two projects. Together, these contributions trace the lifecycle of a world model for manipulation: acquiring predictive knowledge from cheap data, deploying it economically at decision time, and understanding the choices that make it work.

Notes:

@mastersthesis{Pan-2026-88329,
author = {Hengkai Pan},
title = {Imagining Before Acting: Learning and Using World Models for Robot Manipulation},
year = {2026},
month = {July},
school = {Carnegie Mellon University},
address = {Pittsburgh, PA},
number = {CMU-RI-TR-26-77},
keywords = {world models, imitation learning, planning},
}
Copyright notice: This material is presented to ensure timely dissemination of scholarly and technical work. Copyright and all rights therein are retained by authors or by other copyright holders. All persons copying this information are expected to adhere to the terms and constraints invoked by each author's copyright. These works may not be reposted without the explicit permission of the copyright holder.