July 20, 2015

Test Automation: Is it about Recording or Engineering?

BY :     July 20, 2015

test automationTest automation is a promising way to increase the efficiency of repetitive testing tasks. However, it poses few challenges in ensuring its effectiveness and expected benefits.

When thinking about test automation, the first thing to remember is that testing includes much more than just a set of repetitive tasks. Therefore, not everything can (and should) be automated. For example, key testing tasks such as the definition of test objectives or the design of a test plan are engineering tasks that constitute the base for testing, regardless of whether the tests are manually or automatically executed. On the other hand, not all test case components or test types are needed to be repeatable, while others (loading tasks, regression tests, etc.) may really maximize the benefits of automation.

The decision on whether test cases are to be or not to be automated needs to be supported by the expected Return on Investment (ROI) analysis. It’s done by considering several aspects such as the effort for the creation of automated tests, the execution time, the feedback provided and the maintenance effort according to expected changes. In other words, we cannot limit the analysis to the conception that automation is a one-shot task, because obtained test case scripts need to be maintained.

It is known that in software engineering, the maintenance effort may be significantly greater than that required for development of new functionalities. We also know that maintainability relies on the ability to change/modify the implementation, and that this ability depends on the architecture, which organizes the code and makes it more or less modular, changeable, understandable and robust. This is exactly what happens in automation: Not every automation approach will lead us to the same ROI. This is why we promote it in an automation project, where we need to align the objectives and the environment characteristics (frequency of software changes, data availability and integrity, etc.) with the definition of a suitable architecture, aimed at obtaining well-organized and structured test implementations to minimize the risks (maintainability effort, data variability, requirements changes).

This discussion poses a question: Is test automation simply a ‘recording & reproducing’ process or a more complex engineering process? There exist some tools that help us record scripts, which are able to interact with graphical user interfaces and reproduce the exact recorded interactions. However, what happens if something changes? Do we again record all the scripts or do we modify, script by script, the generated code in order to adapt it to the changes? And, if we modify the code (maintenance), would it be worth to have it implemented (with the assistance of recording to generate code chunks) based on a modular, understandable and maintainable architecture? The report – The Forrester Wave: Modern Application Functional Test Automation Tools, Q2 2015 –  enforces this idea and states that “Fast feedback loops frequently break UI-only test suites”. Consequently, it suggests “Ratchet down UI test automation in favor of API test automation. Modern applications require a layered and decoupled architecture approach.”

Albert Tort

About

Albert Tort is a Software Control & Testing specialist in Sogeti España. Previously, he served as a professor and researcher at the Services and Information Systems Engineering Department of the Universitat Politècnica de Catalunya-Barcelona Tech.

More on Albert Tort.

Related Posts

Your email address will not be published. Required fields are marked *

4 + 9 =


  1. Sunny · August 3, 2015 Reply

    Yes…I went through that nightmare where the applications were changing and the automated tests were getting outdated. API based testing as well as GUI based testing would be required.

  2. Frank Høvin · August 18, 2015 Reply

    I have trouble seing how this can ever be a case of either the one or the other being correct. It depends on the type of application,the scope and need for future use. Recording is a fine way of creating the test to begin with, since it can save you a lot of time, but I don’t see how one can avoid having to use scripting if one is to parameterize or generalize the test(s). And this would be a necessity if the tests are to be easily modifiable as things change/evolve. So, a combination on both would be the answer in most cases, I think.

*Opinions expressed on this blog reflect the writer’s views and not the position of the Sogeti Group