Inverse rendering is the process of recovering the shape, materials, and lighting conditions of an environment from a set of images. Both this process as a whole and its individual components are fundamental to applications ranging from medical imaging to astronomy, and from AR/VR to embodied intelligence. Unfortunately, inverse rendering is an inherently ill-posed problem, and often requires many measurements from conventional cameras to perform accurately. In this thesis, we demonstrate that the combination of physics-based light transport modeling, neural scene representations, and specialized cameras---in particular, active sensors, which send illumination into an environment in addition to capturing images of it---can be extremely fruitful for developing inverse rendering algorithms that are robust to complex visual phenomena encountered in real-world settings. We first derive a differentiable volume rendering procedure for the raw measurements from continuous-wave time-of-flight cameras, which produces higher quality 3D reconstructions than baselines when imaging low-albedo objects, scenes with large depth ranges, and surfaces that exhibit multi-path interference. Next, we design a neural scene representation that accelerates volume rendering by directly predicting sparse sample points along each ray; and further show that it has the ability to model a broad class of challenging light transport effects, including reflections, refractions, and scene dynamics. Finally, we leverage this representation in order to build a framework for physics-based inverse rendering of geometry, materials, and lighting under strong global illumination for both active and passive sensors. With this framework, we showcase novel applications such as accurate 3D reconstruction in the presence of near-field reflections, time-resolved relighting, and time-of-flight imaging without time-of-flight cameras.