Test Autonomy Levels

Test Autonomy Levels

Software testing, powered by Artificial Intelligence (AI) and Machine Learning (ML), is undergoing a transition similar to cars (manual driving to fully autonomous). It is useful for the testing profession to adopt similar autonomy level definitions. These level definitions will help categorize different engineering efforts and put them in context of the ultimate goal of full testing autonomy.

Test Autonomy level definitions:

• Level 0: Manual Testing (“Unassisted”) Manual exploratory and regression testing physically carried out by people.

• Level 1: Scripted Automation (“Hands On”) Handcrafted test automation scripts that can repeatedly execute a test case, sometimes with basic self-healing attributes if the application structure changes.

• Level 2: Exploratory Bots (“Hands Off”) Automated semi-intelligent exploration of apps and some measurement of performance/stability without human intervention.

• Level 3: Human-Directed Regression (“Eyes Off”) Humans describe the high-level intent of a test case e.g. “Add two items to shopping cart, delete one, and make sure there isn’t a crash.” Automated machines autonomously decide out how to execute the intent of the regression test case on one or more applications.

• Level 4: Generative (“Mind Off”) Machines generate and execute most test coverage across exploration, and regression testing efforts.

• Level 5: Fully Autonomous (“Human Tester Optional”) Without human assistance, machines are able to evaluate an application, decide what when, where and how testing should be performed, and summarize the results for humans (or machines) to make a release/no-release decision based on test results.

— Jason Arbon CEO Appdiff.com / test.ai, and co-chair of AI for Software testing org (http://www.aitesting.org)