Modelization of Automated Regression Testing, the ART of testing

Regression testing is costly and time consuming. This pretty much sums up the general feel about testing and constitutes one of the main reasons why regression testing is done poorly, if at all. One of the key root causes to test cost and lead time is the manual nature of most – if not all – testing activities and of two in particular: initially creating test cases (test design) and subsequently keeping the test cases and their expected results up-to-date for further use during the application life cycle (regression testing). What if we could automate these two testing activities?! Make testing lean and cost effective, keeping it of the project’s critical path?

With Model Based Testing, we can…! Before deep diving into Model Based Test Design (MBTD) and Automated Regression Testing (ART), let’s first consider the concept of Model Based Testing (MBT). Traditionally, MBT aimed at automated executing of the full test cycle based on a few types of models, specifically designed by and for testing. The promise of this approach is very appealing, but in practice we just didn’t manage to get it done. Too intricate and too brittle to succeed. The spark of brilliance is not to focus on the complete test cycle from test goal to test report, but to gradually, step-by-step modelize distinct activities in testing. A proper model is unambiguous and can therefore be interpreted by a computer, resulting in a consistent, high quality set of test cases. And there are requirements and design models in abundance! So let’s use those instead of creating dedicated test models. Process flow diagram, decision tables, business rules, state transition diagrams and many more are all perfectly useful to generate test cases automatically. We call this Model Based Test Design and have been doing it successfully for many years. But with a suite of generated test cases, we’re still pretty far from fully Automated Regression Testing, because we can’t model the expected result of an individual test case in a flow chart or transition diagram. But we do have a way of generating the proper expected results: execute the test suite on a production-like environment. If we store the answers for later comparison with the answers of the regression test run, then we don’t have to maintain any result at all. They are at our disposal where and whenever we need it! With current state of the art test tools this can generate test cases geared to be executed by a test tool, run a baseline test cycle, deploy new or adjusted software, run the regression test cycle, compare the two test cycle outcomes and report on differences fully automated! There is no reason anymore not to execute full regression testing.