Household and service industry robots will become much more useful when they can effectively communicate with humans and better understand the world. A natural speech interface and understanding of the world could enable robots to take on more general purpose use, as users could interactively specify tasks, and robots could provide feedback and semantic knowledge of the world in an intuitive way. In this paper, we describe an initial formulation and results for a natural speech interface on the ballbot, an agile and safe robot that balances on a single spherical wheel. We develop an end-to-end system for perceiving the world, responding to user questions, and receiving natural language commands to navigate in the world. We then demonstrate this system through an example indoor service scenario.