Introspective Evaluation of Perception Performance for Parameter Tuning without Ground Truth

July 2017

Introspective Evaluation of Perception Performance for Parameter Tuning without Ground Truth

Authors:

Humphrey Hu and George Kantor

Abstract:

Modern perception systems are notoriously complex, featuring dozens of interacting parameters that must be tuned to achieve good performance. Conventional tuning approaches require expensive ground truth, while heuristic methods are difficult to generalize. In this work, we propose an introspective ground-truth-free approach to evaluating the performance of a generic perception system. By using the posterior distribution estimate generated by a Bayesian estimator, we show that the expected performance can be estimated efficiently and without ground truth. Our simulated and physical experiments in a demonstrative indoor ground robot state estimation application show that our approach can order parameters similarly to using a ground-truth system, and is able to accurately identify top-performing parameters in varying contexts. In contrast, baseline approaches that reason only about observation log-likelihood fail in the face of challenging perceptual phenomena.
@conference{Hu-2017-105701,
author = {Humphrey Hu And George Kantor},
title = {Introspective Evaluation of Perception Performance for Parameter Tuning without Ground Truth},
booktitle = {Proceedings of Robotics: Science and Systems (RSS '17)},
year = {2017},
month = {July},
}
Copyright notice: This material is presented to ensure timely dissemination of scholarly and technical work. Copyright and all rights therein are retained by authors or by other copyright holders. All persons copying this information are expected to adhere to the terms and constraints invoked by each author's copyright. These works may not be reposted without the explicit permission of the copyright holder.