The largest remaining problem in autonomous vehicle (AV) development is ensuring reasonable and safe decision-making in the long-tail of edge cases that are poorly represented in training data. In this thesis, we discuss data-centric methods for building safer policies in the structured, data-rich domain of on-road driving and the unstructured, data-scarce domain of off-road driving. We first introduce RefAV, a benchmark for retrieving interesting and safety critical scenarios from uncurated driving logs. We revisit spatio-temporal scenario mining through the lens of recent vision-language models (VLMs) to detect whether a described scenario occurs in a driving log and, if so, precisely localize it in both time and space. Notably, we find that naively repurposing existing VLMs yields poor performance, suggesting that scenario mining presents unique challenges. We discuss the competition held around the RefAV benchmark and share insights from the community. The second half of this thesis explores counterfactual auto-labeling as a method to make vision-language-action (VLA) models more controllable during off-road navigation. We find that training naively on VLM-generated commands is not enough to elicit general language following, as a sample's image observation is often more predictive of the future trajectory than the language command. Instead, we augment the observation with alternative trajectories collected from other times the robot visited a particular location. We perform closed loop evaluation in 3D reconstructions of previously unseen environments. We show our dataset annotation method improves the ability of an agent to navigate autonomously to waypoints hundreds of meters away while enabling language-based interventions at important decision points.