Predicting Failures of Vision Systems

June 2014

Predicting Failures of Vision Systems

Authors:

Peng Zhang, Jiuling Wang, Ali Farhadi, Martial Hebert, and Devi Parikh
Conference Paper
Proceedings of (CVPR) Computer Vision and Pattern Recognition

Abstract:

Computer vision systems today fail frequently. They also fail abruptly without warning or explanation. Alleviating the former has been the primary focus of the community. In this work, we hope to draw the community’s attention to the latter, which is arguably equally problematic for real applications. We promote two metrics to evaluate failure prediction. We show that a surprisingly straightforward and general approach, that we call ALERT, can predict the likely accuracy (or failure) of a variety of computer vision systems - semantic segmentation, vanishing point and camera parameter estimation, and image memorability prediction - on individual input images. We also explore attribute prediction, where classifiers are typically meant to generalize to new unseen categories. We show that ALERT can be useful in predicting failures of this transfer. Finally, we leverage ALERT to improve the performance of a downstream application of attribute prediction: zero-shot learning. We show that ALERT can outperform several strong baselines for zero-shot learning on four datasets.
@conference{Zhang-2014-7841,
author = {Peng Zhang And Jiuling Wang And Ali Farhadi And Martial Hebert And Devi Parikh},
title = {Predicting Failures of Vision Systems},
booktitle = {Proceedings of (CVPR) Computer Vision and Pattern Recognition},
year = {2014},
month = {June},
pages = {3566 - 3573},
}
Copyright notice: This material is presented to ensure timely dissemination of scholarly and technical work. Copyright and all rights therein are retained by authors or by other copyright holders. All persons copying this information are expected to adhere to the terms and constraints invoked by each author's copyright. These works may not be reposted without the explicit permission of the copyright holder.