Towards Transparent Systems: Semantic Characterization of Failure Modes

September 2014

Towards Transparent Systems: Semantic Characterization of Failure Modes

Authors:

Aayush Bansal, Ali Farhadi, and Devi Parikh
Conference Paper
Proceedings of (ECCV) European Conference on Computer Vision

Abstract:

Today’s computer vision systems are not perfect. They fail frequently. Even worse, they fail abruptly and seemingly inexplicably. We argue that making our systems more transparent via an explicit hu- man understandable characterization of their failure modes is desirable. We propose characterizing the failure modes of a vision system using semantic attributes. For example, a face recognition system may say “If the test image is blurry, or the face is not frontal, or the person to be recognized is a young white woman with heavy make up, I am likely to fail.” This information can be used at training time by researchers to design better features, models or collect more focused training data. It can also be used by a downstream machine or human user at test time to know when to ignore the output of the system, in turn making it more reliable. To generate such a “specification sheet”, we discrimina- tively cluster incorrectly classified images in the semantic attribute space using L1-regularized weighted logistic regression. We show that our spec- ification sheets can predict oncoming failures for face and animal species recognition better than several strong baselines. We also show that lay people can easily follow our specification sheets.
@conference{Bansal-2014-17149,
author = {Aayush Bansal And Ali Farhadi And Devi Parikh},
title = {Towards Transparent Systems: Semantic Characterization of Failure Modes},
booktitle = {Proceedings of (ECCV) European Conference on Computer Vision},
year = {2014},
month = {September},
pages = {366 - 381},
}
Copyright notice: This material is presented to ensure timely dissemination of scholarly and technical work. Copyright and all rights therein are retained by authors or by other copyright holders. All persons copying this information are expected to adhere to the terms and constraints invoked by each author's copyright. These works may not be reposted without the explicit permission of the copyright holder.