English | EN 
Testing in production is often perceived as risky and controversial. However, when approached strategically, it can deliver significant benefits.
Recently, I was asked to advise on a testing strategy for a critical system operating in a production environment. My recommendation was to adopt a controlled and calculated approach, as extensive testing in production carries inherent risks that can impact system stability, customer experience, and business operations.
No pre-production environment can fully replicate the complexities of a live system. Real user behaviour, live/transactional data, infrastructure dynamics, and interactions between multiple subsystems are challenging to simulate accurately.
Testing in production allows teams to validate assumptions, uncover edge cases, and ensure that systems perform as expected under real-world conditions. However, to minimise risks while maximising insights, it requires a structured and well-planned approach.
Additionally, periodic Business Continuity and Disaster Recovery (BCDR) tests, along with routine health checks, should be conducted in production to ensure resilience and reliability.
While production testing offers valuable insights, it also presents challenges:
- System Stability: Poorly designed tests can overload the system, leading to downtime or degraded performance.
- Customer Impact: Testing can inadvertently trigger notifications or alerts to customers, causing confusion or dissatisfaction.
- Data Integrity: Test cases may create or modify production data, distorting reports, metrics, and SLAs.
- Security Risks: Security testing, such as penetration tests, can expose vulnerabilities if not carefully managed.
To leverage the benefits of production testing while mitigating its risks, consider the following strategies:
- Limit the Scope of Testing: Focus on specific features, components, or scenarios. Test new features with a small subset of users and conduct tests during off-peak hours.
- Use Feature Flags: Implement feature toggles to enable or disable functionality in real-time, allowing for quick rollbacks if needed.
- Monitor Extensively: Deploy robust monitoring and logging mechanisms to detect issues early. Real-time dashboards and alerts can help identify performance bottlenecks, errors, or anomalies.
- Isolate Test Data: Ensure that test data is clearly marked and isolated from production data to prevent contamination of reports and business metrics.
- Communicate with Stakeholders: Keep key stakeholders, including customers, support teams, and business leaders, informed about the testing plan. Transparency helps manage expectations and ensures preparedness for potential disruptions.
When executed correctly, testing in production provides immense value. It helps uncover issues that may only emerge under real-world conditions, accelerates feedback loops, and ensures that solutions are validated in the actual environment where they will operate.
However, production testing should complement, rather than replace, thorough testing in pre-production environments. It serves as an additional layer of validation, strengthening overall quality assurance efforts.
Testing in production is not inherently a bad idea—it is a powerful strategy when managed effectively. By adopting a calculated and controlled approach, teams can unlock its benefits while mitigating risks. The key is to strike a balance between innovation and caution, ensuring that testing enhances system reliability without compromising stability or customer trust.