PVS: A system for large scale outdoor perception performance evaluation

May 2011

PVS: A system for large scale outdoor perception performance evaluation

Authors:

Cristian Dima, Carl Wellington, Stewart J. Moorehead, Levi Lister, Joan Campoy, Carlos Vallespi, Boyoon Jung, Michio Kise, and Zachary Bonefas
Conference Paper
Proceedings of (ICRA) International Conference on Robotics and Automation

Abstract:

This paper describes the motivation, design and implementation of a Perception Validation System (PVS), a system for measuring the outdoor perception performance of an autonomous vehicle. The PVS relies on using large amounts of real world data and ground truth information to quantify performance aspects such as the rate of false positive or false negative detections of an obstacle detection system. Our system relies on a relational database infrastructure to achieve a high degree of flexibility in the type of analyses it can support. We discuss the main steps required for going from raw data to numerical estimates describing the performance of the perception system, including the generation of ground truth information and the safe speed metric we found to be most useful for comparing the perception system's outputs to the ground truth data. We present results illustrating some of the analyses that can be completed using the Perception Validation System.
@conference{Dima-2011-122465,
author = {Cristian Dima And Carl Wellington And Stewart J. Moorehead And Levi Lister And Joan Campoy And Carlos Vallespi And Boyoon Jung And Michio Kise And Zachary Bonefas},
title = {PVS: A system for large scale outdoor perception performance evaluation},
booktitle = {Proceedings of (ICRA) International Conference on Robotics and Automation},
year = {2011},
month = {May},
pages = {834 - 841},
}
Copyright notice: This material is presented to ensure timely dissemination of scholarly and technical work. Copyright and all rights therein are retained by authors or by other copyright holders. All persons copying this information are expected to adhere to the terms and constraints invoked by each author's copyright. These works may not be reposted without the explicit permission of the copyright holder.