Creating Benchmarking Problems in Machine Vision: Scientific Challenge Problems

April 1993

Creating Benchmarking Problems in Machine Vision: Scientific Challenge Problems

Authors:

O. Firschein, M. Fischler, and Takeo Kanade

Abstract:

We discuss the need for a new series of benchmarks in the vision field. to provide a direct quantitative measure of progress understandable to sponsors of research as well as a guide to practitioners in the field. A first set of benchmarks in two categories is proposed (1) static scenes containing manmade objects, and (2) static naturalloutdoor scenes. The tests are "end-to-end" and involve determining how well a system can identify instances (an item or condition is present or absent) in selected regwns of an image. The scoring would be set up so that the automatic setting of adjustable parameters is rewarded and manual tuning is penalized. To show how far machine vision has yet to go, a Benchmark 2000 problem is also suggested using children's "what is wrong" puzzles in which defective objects in a line drawing of a scene must be found.
@workshop{Firschein-1993-13470,
author = {O. Firschein And M. Fischler And Takeo Kanade},
title = {Creating Benchmarking Problems in Machine Vision: Scientific Challenge Problems},
booktitle = {Proceedings of DARPA Image Understanding Workshop (IUW '93)},
year = {1993},
month = {April},
}
Copyright notice: This material is presented to ensure timely dissemination of scholarly and technical work. Copyright and all rights therein are retained by authors or by other copyright holders. All persons copying this information are expected to adhere to the terms and constraints invoked by each author's copyright. These works may not be reposted without the explicit permission of the copyright holder.