QA and test should be made a smarter, more automated process. That’s according to 62% of respondents in the 2020-2021 World Quality Report from Capgemini and Sogeti, in partnership with Micro Focus, published on November 5, 2020.
It appears that although automation has become core to QA transformation, there’s still some way to go before it reaches maturity, with just 58% of organizations saying they have the right level of test automation. And, although as many as 68% of respondents said they have the required automation tools, only 37% of them feel they get a return on that investment.
The perceived benefit of test automation is typically cost savings, although this is now shifting to faster time to market. Indeed, 69% of this year’s WQR survey respondents said reduction of test cycle time is a key benefit of automation. This sits ahead of both cost reduction and reduced security risk as perceived benefits of automation. The greatest benefit this year was better control and transparency of testing activities.
Test coverage as a key metric
It is surprising that better test coverage didn’t rate as highly (52%) because we feel that it is an important outcome of automation. After all, the broader the scope of your test coverage, the less likely you’ll hit glitches in a real-life scenario. As a metric for tracking application quality, it is interesting to note that functional test overage is the most important indicator, so our surprise is understandable.
Does coverage as a metric sit behind the distinct message that automation is very much about the user interface? Certainly, there is a big change this year in that performance testing is no longer a priority of automated testing. Rather testing APIs and user acceptance are given the highest scores in terms of the percentage of all activities that are automated. We believe that this reflects more business involvement in QA, with an assured UI representing a tangible business outcome.
In search of fewer defects
Surely, a primary ambition of test automation should be to uncover defects early, something that a shift-left approach enables. This is especially so if testing is to meet the business need for UI quality. Yet, defect ‘detection’ is ranked by our survey participants as one of the lowest benefits of test automation. This reflects a lack of maturity in the KPIs for automation, which we believe need to be measured in a new way for the full benefits of automation to be realised. So, for example, we should shift from simply automating penetration testing to the effectiveness of the broader automation suite, for example with a KPI of ‘defect yield’.
Artificial intelligence (AI) as an enabler of new KPIs could be helpful but is not currently widely used in this respect. Nonetheless, new, intelligent technologies are coming into the mix, with machine learning techniques being applied to object recognition, and to determine the scope of optimal automation tests in a bid to reduce growth in test scripts. We’re also seeing AI-based self-healing scripts used to automatically modify scripts during run time. However, both AI and ML are still at early adoption stage in the automation of QA and testing. To a large extent, this is because there aren’t many AI-infused automation tools available – as yet.
Where are the skills?
The lack of AI maturity for test automation has had a knock-on effect on AI skillsets in that they too remain immature. It is interesting to note, however, that for the past three years there has been little change in the response to whether AI changes the skills needed from QA and test professionals, with just 34% saying these skills are lacking with more required. We find this surprising in that we believe AI does need a different mindset and skills.
Once we see more toolsets exploiting the AI models then we should see skills increasing in this area. At present, what’s needed are skills in DevOps, continuous testing and test automation. In this year’s WQR, we point out that the gaps between advanced automation tools and required skillsets are here to stay. As such, we recommend that the way forward is to develop a strategy either to lessen dependency on skills or to make automation inclusive.
Finally, a universal truth is that, for various reasons, the testing phase of the development cycle will continue to be squeezed. As such, we assert that introducing more automation – and pursuing it vigorously – is the only way out of this evergreen challenge.