Juggling test environments

A couple of weeks ago at Sogeti Netherlands we organized the QX day: an event around quality. My contribution to this event was on something that crossed my path many times before and will probably remain there for a long time to come: managing test environments. If you can’t wait until the end I’ll just give you the conclusion:

Any effort you put in maintaining a test environment is wasted effort.

And here’s why.

What makes a test environment?

The best description of what makes up a test environment I (completely coincidentally) found at tmap.net: https://www.tmap.net/building-blocks/test-environment

The parts that stand out for me are:

• It is what the tester needs, not what the organization has available
• Teams often lack the skills and knowledge to direct the setup of the test environment
• It consists of systems, applications, data, network, but just as much a part of this is the process of cleaning up, managing access and orchestration.

All in all, the construction and maintenance of the test environment should be driven by the needs as described in the test plan (in whatever form). Important here is the scope that is set, everything that is outside of the scope of the tests is waste and should be minimized. These can include:

• User management outside of the accounts used to run tests.
• Availability outside of the time when tests are running.
• External systems outside of the scope of the systems that are tested.
• Cleaning up collected data.

Scoping space and time

Knowing what makes a test environment tick is all about scope. If you have dependencies outside of your test scope, simulate them. Simulations (mocks / emulators / stubs / …) are much easier to use, since you control them. Make them simple, it’s only a simulation. The smaller your data set for your mock, the easier it is to control.

As for time: this is very easily overlooked. I’ve seen insane amounts of time used to create test accounts, delete old records, disks that needed to be cleaned up because test data filled a test server, management of team constraints and frustration around using the same environment and influencing each other’s test results. And why?

• Because you don’t delete the environment when you’re done
• Because you are not in control of the environment
• Because you haven’t automated installation
• Because you have manual procedures and do your own sql queries.

Getting in control

You waste time and money (an insane amount of it) when you’re not in control. It can be any reason and there are many justifications for the past, but don’t let it control your future. Getting back in control can be achieved by:

• Automate the installation of your test environment
• Any external dependency, remove it and replace it with a simulator
• Write down what your test sets are, for all your test cases, and generate that data
• Delete your environment. Yes. Delete it. You generate a new environment every time you run your tests.

CI/CD Enablers

To enable your test automation, CI/CD and the cloud are your friends. The agent on which they are running have many capabilities, read up on them and learn to use them. Some capabilities that are extremely useful are (taken from Azure Devops agent but it can be any system):

• Cloud clients for managing Azure, Google cloud, Amazon, Alibaba, etc.
• On-agent docker-compose available
• Kubernetes client
• Dotnet cli, Nodejs, Java, etc.
• ARM, terraform, puppet and chef

Find the components you need and make your automation work for you today!

And if you’d like to share any information and / or initiatives the comments are always welcome…