We introduce Neural Dynamical Systems (NDS), a method of learning dynamical models which incorporates prior knowledge in the form of systems of ordinary differential equations. NDS uses neural models to estimate free parameters of the system, predicts residual terms, and numerically integrates over time to predict future states. It also natively handles irregularly sampled data and implicitly learns values of interpretable system parameters. We find that NDS learns dynamics with higher accuracy and fewer samples than a variety of deep learning methods that do not incorporate the prior knowledge. We demonstrate these advantages first on synthetic dynamical systems and then on real data captured from deuterium shots from a nuclear fusion reactor.