Robots in everyday human environments are expected to perform robust manipulation across cluttered, constrained, and physically interactive settings. This thesis studies how scalable simulation-based data generation can train vision-based imitation learning policies for robust zero-shot transfer to the real world.
In the first part of this thesis, we study robotic manipulation in cluttered shelf environments, where limited free space, dense object arrangements, and complex visual backgrounds make policy learning challenging. We present a scalable learning pipeline that leverages Dynamic Movement Primitives (DMPs) to expand a small set of teleoperated demonstrations into a large and diverse synthetic dataset. We train an imitation learning policy across six scenarios and three manipulation strategies, demonstrating robust generalization across diverse object shapes and zero-shot transfer to a physical manipulator.
In the second part of this thesis, we address autonomous physical human-robot interaction (pHRI), where large-scale real-world training data is difficult to collect. We introduce a zero-shot ``text2sim2real'' generative simulation framework that synthesizes diverse pHRI scenarios from high-level natural-language prompts, including soft-body human models, scene layouts, and robot motion trajectories. Using the generated demonstrations, we train vision-based imitation learning policies on segmented point clouds and show zero-shot sim-to-real transfer on scratching and bathing tasks, with success rates exceeding 80%.
Together, these two works demonstrate that scalable simulation data generation can serve as a practical foundation for robust robot policies in both constrained manipulation and physical human-robot interaction.