A central challenge in the development of autonomous vehicles (AVs) is identifying and responding to anomalous scenarios. Due to the complexity of urban streets, environmental conditions, and the behavior of other actors, AVs encounter out-of-distribution events at a non-trivial rate (often termed the long-tail problem). Anomalous situations may be statically perceivable (e.g. an overturned truck, or a mounted policeman) or require nuanced behavioral inference (e.g. vehicles swerving to avoid broken glass on the roadway). Despite the importance of robustly identifying these situations, there are few resources for developing or benchmarking new methods in the research community. Here we introduce a framework for generating anomalous and nominal scenarios using a domain-specific scenario description language (Scenic) and a driving simulation environment (CARLA) from multiple viewpoints. We introduce four classes of roadway anomalies and establish a benchmark and baselines for the development of new methods. Crucially, our benchmark supports the development of both sensor-based and semantic approaches. The code and related documentation for this work is available at https://github.com/t27/carla-scenic-data-collector