Robustness Inside Out Testing

June 2020

Robustness Inside Out Testing

Authors:

Deborah S. Katz, Milda Zizyte, Casidhe Hutchison, David Guttendorf, Patrick E. Lanigan, Eric Sample, Philip Koopman, Michael Wagner, and Claire Le Goues

Abstract:

Robustness testing is an important technique to reveal defects and vulnerabilities in software, especially software for Unmanned Autonomous Systems (UAS). We present Robustness Inside Out Testing (RIOT) as a technique directed at finding failures in autonomy systems that are able to be activated from external interfaces. The technique consists of four main steps: unit-level robustness testing, generalization, permeability analysis, and activation. Each of these steps yields a valuable deliverable in the testing process, and, when applied in succession, expands a unit-level bug to an external interface. RIOT has the following advantages over traditional robustness testing: it finds faults faster, it can find faults missed by traditional approaches, it identifies faults that can be triggered from inputs at an external interface, and it produces useful artifacts to aid in fault diagnosis and repair. In this paper, we outline each step of the RIOT process and provide an example of RIOT finding a bug on a real system that would not have been discovered using existing techniques.
@conference{Katz-2020-126334,
author = {Deborah S. Katz And Milda Zizyte And Casidhe Hutchison And David Guttendorf And Patrick E. Lanigan And Eric Sample And Philip Koopman And Michael Wagner And Claire Le Goues},
title = {Robustness Inside Out Testing},
booktitle = {Proceedings of 50th Annual IEEE/IFIP International Conference on Dependable Systems and Networks},
year = {2020},
month = {June},
pages = {1 - 4},
}
Copyright notice: This material is presented to ensure timely dissemination of scholarly and technical work. Copyright and all rights therein are retained by authors or by other copyright holders. All persons copying this information are expected to adhere to the terms and constraints invoked by each author's copyright. These works may not be reposted without the explicit permission of the copyright holder.