The (MIS)Estimation of the Bewilderment Factor in Test Automation Projects

What happens in a test automation project when input test case designs to be automated and their associated requirements are not trustworthy or they change, evolve or become inconsistent, even when the project is already started? One of the main consequences of such kind of situations is bewilderment. This answer poses a challenge: how to deal with the bewilderment factor in projects? The bewilderment factor is the extra effort produced as a result of one or more of the following circumstances:

• Getting over contradictory or unclear indications, requirements, specifications or test case designs.
• Ambiguity or missing information regarding the input elements which serve as input for the project.
• Unclear objectives to be achieved.
• Working in a non-clearly defined environment.
• Unexpected/non-communicated changes in the application.

Such circumstances are especially risky in test automation projects, since automated test cases perform a non-ambiguous sequence of actions on the system under test. If such specifications are not clear, then they need to be clarified to become unambiguous. Sometimes, this effort also creates communication issues, responsibility conflicts and misalignments between people expectations. The consequence is, again, an increased bewilderment. We promote to include the bewilderment factor in estimation techniques. It is a subjective factor, but it usually raises the effort to be done in many projects. This factor may aggregate many of the social effects that are not usually considered when estimating a project, although their consequences become tangible in terms of time, money and results. The bewilderment is especially critical when the kickoff of the project does not ensure that work environment, objectives and necessary inputs are clearly stated. Sometimes, this ambiguity cannot be totally resolved, but if we know the risk, why not taking it into account? As a social factor, we need to evaluate bewilderment by using empirical techniques based on theconsequences experienced in previous projects, evaluated on a common set of project environment variables. Particularly, in test automation projects, it is essential to evaluate if:

• The automation scope is well-specified.
• Input manual test cases to be automated are well-specified with an adequate level of detail. Otherwise, test cases need to be (re)designed first according to the scope and the testing criteria(coverage, risks, key functionalities, etc.)
• The execution criteria and the reporting formats are decided. It includes aspects such as thetesting environment that will be used to execute the test cases, how the results will be generated and managed, etc.
• There exists a test environment that fits data requirements and controlled changes.
• A framework for reducing maintenance effort and code comprehensibility is defined and explicitly specified.

Social cross-wise abilities and attitudes of the project team, such as experience in programming, autonomous learning, communication abilities and management skills may contribute to mitigate this factor. Many times, in software engineering or automation testing projects, estimations may be overtaken by reality when the bewilderment factor is not taken into account. Bewilderment is a non-countable factor at first sight, and therefore it is more difficult to be estimated and conceived as real effort from the customer point of view. However, not considering it increases the risk of failing effort estimations, since it may be a frequent cause of extra effort in many contexts. In summary, we should take into account that people matter, and that dealing with social effects focused on people (like bewilderment) in projects planning and management may count (or discount) in the results.