The promise of robot manipulation is to automate tasks that are too dull, dirty, or dangerous for humans. Realizing this promise requires remarkable resource efficiency: robots must adapt to new tasks with limited data and compute, while meeting stringent industrial performance requirements. Current learning-based systems can succeed in dexterous tasks, but require large amounts of resources to reach industrial standards. A possible explanation for the inefficiency is that frontier models learn from images. Content that is irrelevant to robotics dominates the images. Previous work has improved efficiency through task-specific feature engineering. Can we construct similarly focused representations that remain scalable and broadly applicable? This thesis explores spatiotemporal representations as a possible alternative through spatiotemporal reconstruction, learning generative priors, and world-action models.

The first part of this thesis addresses spatiotemporal reconstruction. Robots may benefit from observing past experiences in a precise spatial (3D reconstruction) and temporal (3D point tracks) context. A crucial prerequisite for obtaining these contexts is geometric camera calibration. I propose a method for robustly calibrating wide-angle lenses in a new easy-to-use calibration toolbox, TartanCalib, which researchers use around the world. TartanCalib requires dedicated calibration captures, which are not always available. I propose MASt3R-SfM, which uses a pairwise reconstruction foundation model (MASt3R) to achieve unconstrained structure-from-motion. Using a set of calibrated captures, I then explore neural rendering techniques for multi-view reconstruction. Residual-NeRF uses a known background to reduce ambiguity when reconstructing transparent objects. DeformGS extends to dynamic scenes and estimates dense scene motion in highly deformable environments. Cloth-Splatting reconstructs cloth faster and with greater quality, using a pre-trained cloth dynamics model.

The second part investigates generative spatiotemporal priors for robot manipulation. Humans understand viable geometry and motion from partial observations, and robots benefit from a similar prior. I propose RaySt3R, a foundation model that completes objects from a single view. It achieves this by casting object completion as a novel-view depth prediction task. However, acquiring this kind of geometric supervision at scale is expensive. To learn from arbitrary images, I propose a post-training recipe (Modality Forcing) to generate monocular depth maps from a text-to-image (T2I) model. I discover that depth quality scales with T2I pre-training, and contribute a state-of-the-art monocular depth estimator. However, this image-based prior does not yet capture temporal structure. In PointZero, I study 3D point track completion as a pre-training objective for 3D world-action models (WAMs) without requiring robot action labels. I show that PointZero improves performance in action-conditioned dynamics prediction and imitation learning.

The third part considers spatiotemporal world-action models. In 3PoinTr, I study 3D point tracks as an alternative to video in world-action models (WAMs). To condition a closed-loop policy on compact task plans, 3PoinTr predicts 3D point tracks instead of video. These plans outperform 2D track plans. However, complementary modalities may improve on 3D point tracks alone. With ModAR, I investigate world-action models that autoregressively predict multiple future modalities before generating actions. I contribute the first controlled study across data, modalities and WAM formulations. I find that learned visual features from DINO (a self-supervised visual encoder), point tracks, and depth are beneficial while predicting future RGB images provides no consistent additional benefit at the evaluated scale.

Together, these works connect spatiotemporal reconstruction, generative priors, and action generation to study how spatiotemporal representations can support scalable robot learning.