KEEPING AN EYE ON THE WHOLE WHILE DIVING INTO THE DEEP

Testing is about exploring to gain information about the system under test. In theory, this can be done with a map of defined requirements, but in reality, most of it has to be done in uncharted territory. There is no way to thoroughly describe a system and most of the information needed lies in between what’s described and in uncharted territory.

It’s easy to get lost in the details during testing. You test every aspect of the written requirements, but you miss that the entire process is mismatched or the unarticulated needs of the stakeholders. When working with agile methodologies, you get lost in the user stories to be delivered and miss that they actually don’t work together.

At the same time, you could miss a lot of bugs if you just test the overall aspects of a system. To find all relevant information, you have to dive deep and explore the unknown.

A great tester learns to keep an eye on the whole while diving into the deep.

In this blog, we will discuss strategies for navigating uncharted territories while also keeping an eye on the whole and diving deep.

Use explicit and implicit tests

One important strategy is to use both explicit and implicit testing. What is the difference between them? In this blog I’m using the TMAP definitions. Explicit testing involves defining and executing specific test situations and cases for testing a specific situation. An implicit test is observing the behavior of a test object during execution of test cases that were not designed for testing that specific behavior. In other words, explicit testing is when we are only observing what’s included in the test’s purpose, while implicit testing is when we’re observing outside the scope.

Automatic testing is ONLY explicit. Manual scripted testing should be both explicit and implicit. We need both.

So, observe not only what’s within the scope of the test. When something weird happens, explore it!

Use different perspectives

To find problems, we need to test from different perspectives. It’s not enough to just use one or two. Of course, what perspectives you need to base your tests on varies depending on the system. The following are just some common examples:

• User – The user will have a unique perspective. If you’re lucky, actual users can support your testing. If not, you can solve it by interviewing users and/or observing their work to gain more insight into their needs. Always spend some time testing from the user’s perspectives.
• Business – The business is not always the same as the users of the system. For instance, the business can be the organization that sets the rules for the work. Involve people from the business as early as possible to help with the testing.
• Technical – Depending on how the system is designed and what technology is used, there may be risks we need to test for. Talk to architects and developers about how the system works to identify and design tests.
• Operation – A system needs to function properly in operation. This can, for example, include testing if the system uses excessive resources, if logging works and if we can backup and restore databases.
• Interconnected systems – Most systems today have a lot of dependencies. Test based on the perspective of each interconnected system. Does your system behave as the interconnected system expects?

Some perspectives will be detailed focused, while others are more interested in the overall view. By using enough perspectives, it becomes easier to keep an eye on the whole while diving deep.

Use different depth

It’s easy to get lost in details, and lose the overall view, but at the same time we need to look at details and not just get an overview of the system. It’s hard to do both at the same time. The solution is to use different levels of depth in specific tests. Some are more aimed at flows and general view, while others focus on details.

Understand the system and its purpose

To be able to keep an eye on the whole, you need to understand the system and its purpose. Requirements and/or user stories aren’t enough to achieve this. You need other sources. Some examples are:

• Other documentation such as architectural descriptions, design documents, and user manuals
• Common sense
• People who know how it should work
• People who know how it is used
• Similar types of systems
• Previous versions of the system

The above are just some examples. So, use whatever else you can find as well.

Test more than requirements

A common problem in testing is when the tests are only focused on requirements and/or user stories. In theory, there is a map of defined requirements. In reality, most of it is actually uncharted territory. There is no way to thoroughly describe a system. Most bugs lie in-between what’s described and in uncharted territory. There are always unformulated requirements. Often, they are self-evident for stakeholders, so they aren’t mentioned even though they are necessary to get a usable system. Requirements are also often described as points, but you need to be able to walk between the points.

Therefore, don’t restrict yourself to a box! Test more than requirements. Some examples:

• System works as a whole
• Negative testing
• Error handling, for instance, what should happen when another system doesn’t respond?
• Stability
• Operability
• Testability

The above are just some examples of what needs to be tested besides requirements and/or user stories.

Use tests with different purposes

It’s not just one type of test we need, and it’s hard to do everything at the same time. If we use tests with different purpose, it’s easier to make sure we have all the different types we need. Depending on the purpose, the test can be better suited to be exploratory, or scripted.

So, use tests with different purposes to make sure you keep an eye on the whole while diving into the deep. Specific tests should reflect different perspectives. The purpose of the test is used to differentiate between types of tests. The purpose determines what to include in the test, how to formulate it if it’s scripted, and the level of detail to observe.

Use a Test Area Mind Map

To be able to keep an eye on the whole, you need an overview of the test object divided into more details. A Test Area Mind Map can help provide an overview of the system, what to test and the test progress.

A test area describes at an overall level what needs to be tested. It is a way of grouping tests that have something in common in order to get a better view of the whole. The division is usually made based on actors or functions and properties. When all test areas have been identified, each test should be able to be placed within a specific area. Often, the division of test areas and sub-areas is gradually adjusted when more information becomes available.

A Test Area Mind Map is a great way to get the system overview needed.

Summary

Testing is about exploring to gain information about the system under test. In theory, this can be done with a map of defined requirements, but in reality, most of it has to be done in uncharted territory. There is no way to thoroughly describe a system and most of the information needed lies in-between what’s described and in uncharted territory.

During testing, we need to keep an eye on the whole while diving into the deep. To be able to do this, we should:

• Use Explicit and Implicit tests
• Use Different perspectives
• Use Different depth
• Understand the system and its purpose
• Test more than Requirements
• Use tests with different purpose
• Use a Test Area Mind Map

There are many different tests we need to conduct. One person can’t be good at everything. Therefore, we need to work together as a team, using the strategies in this blog. Of course, there are more strategies we can use. I’ve highlighted those I deem most important.

What do you think? Do you have additional strategies you want to share? Please, comment. I’d love to have your feedback.