We have heard a lot about Continuous Integration and Continuous Deployment A.K.A. CI/CD but, how and why do we, as QA people, help in these settings on our project?
First, we must understand that the quickest and most well-defined answer will always be: it depends. All the settings of a project will be accordingly to how the team and the business work, so we, as QAs, the people who perform different kinds of testing on the lower environments and the production environment must help our team in connect the business necessities with an approach that assure us that we are reviewing what we need, in the required environment with the required changes. In practice, Continuous Delivery means we deploy the new changes as soon as we have them approved in the repo and we can do this with an answer that is simple to type but hard to set: pipelines, we do not focus on a specific tool for this, we need to focus on the general concept of “How can I guarantee that I am testing (functional or non-functional) the right environment with the latest updates?”
What has helped me a lot is understanding our application and not just the business side, but also how the deployment cycle is working, we usually have more than one testing environment, let's use as an example these environments:
- Test
- Stage
- Preview
- Prod
We have so many of them because we use different settings and connections between these environments, let´s say that Test will have a connection with the testing database. Still, Stage has the latest changes from the API team that we don´t have on test yet and Preview consumes similar APIs and databases as Prod so, we can do performance tests without interrupting Prod behavior. With all these common configurations on our testing environments, is easier to realize why we need to understand our pipelines so we can set different types of automated tests on them based on the system and necessities, for example:
- We can create an accessibility test for the Test environment.
- We can add e2e test on Stage because we know that it always has the latest changes on the APIs.
- We can execute Load tests on the Preview environment as this use similar services to Prod.
- We can add a performance test on Prod to confirm that the User Experience will not be affected (of course this will be after deploying the changes to not interrupt the pipeline flow)
As you can see here, we have different types of testing for the different environments, and you are wondering, how can we handle this in the pipeline? Well, there are several approaches that we can use for solving this, a DevOps engineer will help us more in this configuration but one that I really like to use is splitting, having one pipeline for each environment, and setting the YAML file based on each of them, we can also connect our repo to execute the task for all of them but Prod at the same time, these means, when we do a PR on the development branch we can trigger the pipeline for Test, Stage, and Preview and have the master branch to trigger the Pipeline for Prod
As I always tell you, this is the experience that I got from working on different types of projects and are patterns that have helped me and my team, if you have any comment or suggestion please let us know in the comments below.
Top comments (2)
Don't run performance tests on production. Do have performance monitoring.
Update your test environment. If the environment does not have the latest APIs then those APIs aren't ready to test.
As QA I'm an advocate for change. I want my developers to feel like they can do code refactors and dependency updates without fear of harming the product.
Yeah you are totally right Jesse!
Performance should not be executed on prod but in this case we did it because we were noticing that the User Experience was really slow and they were dropping from our flow, so we added after deployment on the pipeline so we can confirm the the Time to Interaction, and Speed Index without blocking the deployment
And about the test environment, same thing, in this project we had several test environments but some teams used just some of them so we adapted to be able to properly test the changes before deploying
Thanks for your support!