English | EN 
The book “Quality for DevOps teams” and the renewed TMAP body of knowledge support all people involved in IT delivery, to deliver business value using IT systems that are at the right quality level.
In this blog, you will learn how mutation testing, which relates closely to unit testing, can determine whether your unit tests are working properly.
Unit testing is probably one of the most discussed topics among members of DevOps teams. Every member of the team has an opinion about it. Typical discussions in a DevOps team are about the definition of a unit or what aspects of a unit should be tested. Another point of discussion is code coverage, especially managers seem to love this metric. But the exact boundaries of a unit or the percentage of coverage, are no indicators of the quality of the testing. So instead of these time-consuming discussions, the DevOps team should assess the quality of their tests, mutation testing is a good way to do this.
Mutation testing is a type of testing where certain statements in the source code are changed (mutated) to check if test cases will identify the fault that was introduced this way. This is a manner to verify the quality of the test set (instead of the object under test).
Mutation testing is a useful way to check if common faults would be detected by the test set and this way helps to improve a test set in case the injected faults would not be detected. It is, however, no replacement for proper test design.
How does mutation testing work?
Write a piece of code. Write tests for that code. Execute the tests. If all tests pass this first version of the test, the test is ready.
Create a mutant in the code and again execute the tests. If a test fails, the mutant is detected, and the tests are OK. If none of the tests fail, the mutant is not detected; the tests are then incomplete, and an extra test needs to be added or the code must be refactored. So basically each mutation should lead to at least one test to fail.
As with any activity in DevOps, we strive to automate as much of this process as possible. Good mutation testing tools are available. Keep in mind however that the analysis of mutants that are not detected often requires human action.
A brief example:
In case the IT system contains a condition that someone should be at least 120 cm long to be admitted, the example pseudo-code is:
IF length ≥ 120 cm
THEN MOVE “admitted” TO status
ELSE MOVE “not admitted” TO status
ENDIF
With 2-value boundary value analysis the test cases would be:
TC1: input – length = 119, expected outcome – “not admitted”
TC2: input – length = 120, expected outcome – “admitted”
Both test cases will pass for the pseudocode above.
Now the IF statement in the pseudocode is changed with a common mutant (changing “greater-than-or equal” to “equal”):
IF length = 120 cm
For this mutant TC1 and TC2 will both pass, but a test case should have failed. The mutation test has detected a fault in the test set that can be fixed by adding a third test case:
TC3: input – length = 121, expected outcome – “admitted”
So instead of wasting time discussing how and what a DevOps team should test, mutation testing enables the team members to focus on the quality of their (unit-)tests. And this, in the end, leads to achieving the right level of quality, that will deliver the pursued business value to the business stakeholders and users.
The new TMAP body of knowledge
and the book “Quality for DevOps teams” are launched on Tuesday 17 March 2020
in Utrecht, the Netherlands.
The TMAP body of knowledge is
freely accessible on www.tmap.net.
The book is available in
hardcover and ePub versions on www.ict-books.com.
For TMAP training courses please
go to academy.sogeti.nl.
Would you like to know more? Don’t hesitate to contact your Sogeti representative.
This blog was written by Berend van Veenendaal, one of the authors of the above-mentioned book.
