Juggling test environments
A couple of weeks ago at Sogeti Netherlands we organized the QX day: an event around quality. My contribution to this event was on something that crossed my path many times before and will probably remain there for a long time to come: managing test environments. If you can’t wait until the end I’ll just give you the conclusion:
Any effort you put in maintaining a test environment is wasted effort.
And here’s why.
What makes a test environment?
The best description of what makes up a test environment I (completely coincidentally) found at tmap.net: https://www.tmap.net/building-blocks/test-environment
The parts that stand out for me are:
- It is what the tester needs, not what the organization has available
- Teams often lack the skills and knowledge to direct the setup of the test environment
- It consists of systems, applications, data, network, but just as much a part of this is the process of cleaning up, managing access and orchestration.
All in all, the construction and maintenance of the test environment should be driven by the needs as described in the test plan (in whatever form). Important here is the scope that is set, everything that is outside of the scope of the tests is waste and should be minimized. These can include:
- User management outside of the accounts used to run tests.
- Availability outside of the time when tests are running.
- External systems outside of the scope of the systems that are tested.
- Cleaning up collected data.
Scoping space and time
Knowing what makes a test environment tick is all about scope. If you have dependencies outside of your test scope, simulate them. Simulations (mocks / emulators / stubs / …) are much easier to use, since you control them. Make them simple, it’s only a simulation. The smaller your data set for your mock, the easier it is to control.
As for time: this is very easily overlooked. I’ve seen insane amounts of time used to create test accounts, delete old records, disks that needed to be cleaned up because test data filled a test server, management of team constraints and frustration around using the same environment and influencing each other’s test results. And why?
- Because you don’t delete the environment when you’re done
- Because you are not in control of the environment
- Because you haven’t automated installation
- Because you have manual procedures and do your own sql queries.
Getting in control
You waste time and money (an insane amount of it) when you’re not in control. It can be any reason and there are many justifications for the past, but don’t let it control your future. Getting back in control can be achieved by:
- Automate the installation of your test environment
- Any external dependency, remove it and replace it with a simulator
- Write down what your test sets are, for all your test cases, and generate that data
- Delete your environment. Yes. Delete it. You generate a new environment every time you run your tests.
CI/CD Enablers
To enable your test automation, CI/CD and the cloud are your friends. The agent on which they are running have many capabilities, read up on them and learn to use them. Some capabilities that are extremely useful are (taken from Azure Devops agent but it can be any system):
- Cloud clients for managing Azure, Google cloud, Amazon, Alibaba, etc.
- On-agent docker-compose available
- Kubernetes client
- Dotnet cli, Nodejs, Java, etc.
- ARM, terraform, puppet and chef
Find the components you need and make your automation work for you today!
And if you’d like to share any information and / or initiatives the comments are always welcome…
About Edwin van der Thiel
Freedom has been a guiding factor throughout Edwin's career, it drives his ambition and gives meaning, in different ways. In the form of Personal Freedom it's the ability for every person to be themselves. He embrace it in himself, in his development plan, building his career. To others he strives to be as open, fair and welcoming as possible, and defend each one's right to be themselves. In software freedom, through Open Source and Open Standards, He's a big advocate of the ideology. Not only in the open source community, but bringing the culture of openness and sharing wherever he works. Openness is the basis of trust, and it can be a guide to the future. Through freedom of teams, He's a big fan of Agile working. His vision on Agile is centered around the triangle of Trust, Responsibility and Commitment. It brings guidance on collaboration and enablement rather than focussing on processes, managers and templates. Freedom of data, or rather the ability to own your own data. Currently this centers heavily on Decentralization, Web3, Blockchain and the Metaverse. To him the important issue is on building an internet where everyone can be owner of their data, they control it, no governing platform, institution or country can take it. Much like the ideas in your head, you hold what's yours. He loves the south-american culture, in particular Brasil. In his spare time he is a husband to Jacy, father to Amy, a dancer and Capoeira instructor, and loves to explore the world.
More on Edwin van der Thiel.
Awesome, yes test environment is consuming a lot of time. I often refer to them as a ‘simulated environment’ – this often helps people look upon them in another light and often unblocks people’s obsession about making it a holy grail of quality assurance.