Menu
NREC developed the Stress Testing for Autonomy Architectures (ASTAA) tool to find safety problems in autonomy systems that are unlikely to be discovered by other types of tests or extended field testing. The ASTAA tool can be used to test both entire systems and individual components at any point during the development process.
Developing safe, robust autonomy software is challenging. While field testing is a valuable part of validating overall system safety, it cannot uncover all safety issues of an autonomy system. It’s impossible to field test autonomous systems under all combinations of operating conditions, vehicle behavior, sensor inputs and environmental factors. These factors usually can’t be controlled or replicated, which precludes regression testing and benchmarking to compare successive versions of the autonomy software.
With ASTAA, there’s no need to rely exclusively on system field tests. Safety testing can be done at any point during the development process. ASTAA can be used to test individual components as well as the overall system. Access to component source code isn’t necessary because ASTAA tests via external interfaces. Problems can be discovered early on when they are easier to fix.
The ASTAA toolkit feeds potentially abnormal inputs into a system until it exhibits unsafe behavior. These inputs could include an out-of-range sensor and vehicle data, inputs that cause a divide by zero errors, and other data that may generate unusual behavior or computational errors. The ASTAA tool then generates a report of the problems it has found. Tracking them down and fixing them improves the robustness and safety of autonomy systems.
ASTAA can uncover software errors that traditional test methods are unlikely to find and could result in unsafe behaviors after a system is deployed.
ASTAA creates a benchmark for testing autonomy software. This data is especially valuable for acceptance and safety testing of unmanned autonomous systems. These systems currently undergo standardized tests that focus primarily on their physical characteristics (mobility, operating temperature range, endurance and so forth). ASTAA allows autonomy software to be tested with the same level of rigor. This gives testers a better tool for predicting the safety of deployed autonomous systems.
This project was funded by the Test Resource Management Center (TRMC) Test and Evaluation / Science & Technology (T&E/S&T) Program through the U.S. Army Program Executive Office for Simulation, Training and Instrumentation (PEO STRI) under Contract No. W900KK-11-C-0025, “Stress Testing for Autonomy Architectures (ASTAA).”
National Robotics Engineering Center
10 40th Street
Pittsburgh, PA 15201
+1 (412) 681-6900
Carnegie Mellon University
Legal Info | www.cmu.edu
