Test Automation Meters

How to measure something that cannot be seen? The importance of test automation for an organization is often implicitly seen as an improvement in software quality and can be difficult to quantify in numbers. How to measure something that cannot be seen? The effect of test automation can be difficult to measure. The importance of test automation for an organization is often implicit – an improvement in software quality that is difficult to quantify in numbers. What is the increase in bug rates about software quality? It can tell you that software developers have begun carelessness and therefore bugs run more into the testing phase. However, it can also tell you that the bugs are being found more efficiently, they have not been detected before. Over-staring an individual meter can mislead. Test automation is preventive in nature. It is supposed to prevent bugs from getting into production before they can be detected. In the best case, test automation is invisible to the end user. So how do you evaluate something that shouldn’t even appear? Test Automation KPIs At Robot Framework User Group on 28.6.2018 (https://www.meetup.com/Robot-Framework-Helsinki/events/251475886/) Tommi Oinonen gave an interesting presentation on KPIs and test automation meters. What are KPIs (Key Performance Indicators) for Test Automation? Which converters best measure the performance of test automation? Tommi presented a list of interesting questions to evaluate different aspects of test automation. He divided the questions into four categories: confidence-building, early detection of bugs, test feedback, and maintainability. These questions provide a better picture of what all the factors affect the test automation process and what should be considered when developing test automation. Confidence issues, such as the extent of test coverage and the reliability of tests and test environments, create confidence in the software that is being developed. The number and quality of bugs found can be compared with the time and how many bugs pass through the testing phase to the next. Test feedback can be evaluated by considering how long it takes to run the test to get results and how quickly disrupted tests are corrected. Test maintainability is a subset that is influenced by the number of tests, the complexity of the test framework used, and how difficult it is to analyze the test results. Maintenance of a complex system and its complex tests may require a full-time test developer work input. Sum of multiple variables All of the above categories contain important questions. Test coverage, reliability, number, and maintainability are important indicators. Tests must be comprehensive and reliable, testing the right things in the right way and having enough. There are many different metrics available for test automation, but one thing is certain: good quality software is the result of many meters. Sources: – Tommi Oinonen, “Do You Have KPI Phobia?”, KPIs and RobotFramework + Robot Framework RPA, Robot Framework Helsinki Meetup, 28.06.2018, Espoo Finland Image source