Tasks involving locally unstable or discontinuous dynamics (such as bifurcations and collisions) remain challenging in robotics, because small variations in the environment can have a significant impact on task outcomes. In this thesis, we present a robot system that we developed to evaluate learning algorithms on real-world physical problem solving tasks which incorporate these challenges. For such tasks, learning a single deterministic policy that is robust to slight or imperceptible changes in environment state and dynamics is difficult. Learning such a policy from scratch on the real robot can also be prohibitively expensive. We provide a framework for learning structured exploration policies in simulation based on a mixture of experts (MoE) policy representation. We also present a method for efficiently adapting the policy in the real world. The mixture of experts policy is composed of stochastic sub-policies that allow exploration of multiple distinct regions of the action space (or strategies) and a high-level selection policy to guide exploration towards the most promising regions. We demonstrate that representing multiple strategies promotes efficient adaptation in new environments and strategies learned under different dynamics can still provide useful information about where to look for good solutions.