DEV Community

Cover image for Beyond GitOps: Continuous Delivery and Release Management for SaaS
Jürgen Reiß
Jürgen Reiß

Posted on

Beyond GitOps: Continuous Delivery and Release Management for SaaS

Any SaaS platform derives its value from

  1. the annual recurring revenue (ARR) and its growth.
  2. the number of customers and the growth in customer numbers.
  3. the customer retention.

To achieve a high value, reliable platform operations is most important. However, the fast delivery of new features and improvements is needed to retain customers and to onboard new customers. Reliable operations and fast changes seem to be contradictory requirements at first glance. However, continuous delivery is the best practice that I know of to achieve fast delivery of new features. And thus reduce the time to market!

The full description of CD could be found in Reliable Software Releases through Build, Test, and Deployment Automation. I would like to focus on what in my experience is needed to make CD successful in a SaaS context:

  1. Experienced software developers - who understand the business domain.
  2. Reasonable test coverage - at least 80% for unit tests and a reasonable number of end-2-end and performance tests.
  3. A delivery pipeline - immutable artifacts are promoted from one stage of the pipeline to the next, or discarded on the way. Note that a delivery pipeline has literally nothing to do with what tools like Jenkins call a pipeline. Very confusing.
  4. A promotion process that keeps whoever is responsible for promotion in the driver seat and makes promotion decisions simple and auditable.
  5. Easy rollback.

One state-of-the-art approach to achieve 3 (delivery pipeline) and 5 (easy rollback) is GitOps: The practice of maintaining all configuration of the various stages in a Git repository. My preferred approach would be a setup with helm charts maintained in Git and deployed to the various stages with Argo CD. So far so good. So far so best practice.

The challenge that I would like to put into focus of this article is 4. (driver seat).

GitOps is Operations Centric

GitOps in many cases is driven by operations (Some organizations call this DevOps). This is the natural thing. Operations decides about things like the number of cores a specific application requests or is limited to in each stage. Or how secrets are injected into the Docker image. Or on which URL the prometheus endpoint is exposed in each of the SaaS applications.

Among all of these configurations there is one configuration that is also maintained in GitOps even though the decision which value the configuration must have is decided somewhere else completely: The Docker image version (in kubernetes) or the binary version (on VM). For the sake of simplicity, I will refer to this configuration as "Docker image version" only.

Let's assume we have three stages in our delivery pipeline:

  • integration
  • acceptance
  • production

Usually, developers decide on the Docker image version in the first stage of a delivery pipeline (integration stage). They do that simply by building the application or by committing (maybe to a certain branch). Usually, testers thoroughly test the application on this first stage. Maybe with a manual test plan for the new features and an automated integration test for the existing features. If testers are happy, they promote the application to the next stage of the delivery pipeline (acceptance). However, it also happens a lot that a tester has to redeploy an older version of Docker image first before executing the test in integration. This is live after all - and live does not always stick to the plan!

So, what testers need to be able to do, is:

  • deploy a version to integration
  • promote a version to acceptance

On acceptance, in some real world cases that I worked with, the application will be tested by a stakeholder or a key user to find out if the new feature really works as expected. Also performance tests might be executed there by the testing team. Maybe there is also a full integration test: usually more than one application is deployed at the deploy window, and we have to test them together.

As a consequence the expectation is that the key user or the performance tester has an easy way to deploy the version they want to test on acceptance. And - depending on the process - promote the image version for production.

Once deployed to production, the team might decide to execute a smoke test. The consequence of a failing smoke test might be a rollback. It might also happen that outside the smoke test protocol, issues are reported and cannot be resolved by second-level support. In this case, the second-level support might decide to roll back the changes.

To summarize, the following roles need access to the GitOps Git repos to set the Docker image version:

  • developers
  • testers
  • product people
  • second-level support.

And if you ever worked in a real environment, you know what operations will tell you if you ask for this kind of access: NO, NO, NO (and this is a nice team!).

Make GitOps Role Centric

We need a solution that considers the needs of all roles:

  • Operations needs full control about all settings that have to do with operations.
  • The other roles need full control over the Docker image version only.

And the solution is NOT to create a ticket for operations each time the Docker image version needs to be updated. While a lot of companies do that, this leads to waste of working time. And no one has time to waste!

So, how to achieve this? Let's configure the helm chart to use three Git repositories:

  1. Helm Templates: In the helm templates Git repository DevOps can develop their helm chart templates for the various applications. Very likely they develop in one branch and have another branch for each stage.
  2. Deployment Configuration: In the deployment configuration Git repository DevOps can maintain the configuration of the application. Here things like CPU, memory, env vars or secrets are configured.
  3. Image Configuration: In the image configuration Git repository all the other roles can manage the Docker image version of the applications.

Git's RBAC (Role Based Access Control) will be used to restrict access. Note: you very likely need additional measures to guarantee that only the Docker image version is maintained in the image configuration repository.

The pros of this approach are

  • that it works,
  • that the management of the grants is well understood
  • every role can edit what they are responsible for.

However, there are some missing pieces. There is no guarantee that a Docker image can only be deployed to production if it was deployed to acceptance and was promoted there. There is not even a guarantee that the rollback goes back to the previous version. Humans edit in Git repositories. And they can make errors. And they will do so. Further, some of the roles might not be very familiar with Git at all. This will make it difficult to scale your team.

Make Docker image selection simple: Release Management

Simple Docker image selection requires some work. Additionally to managing the Docker images, we also need to manage the lifecycle of the image through the delivery pipeline.

So, we need to keep track of:

  • each "release" of an application. A release is usually created when the CI server builds a Docker image and puts it to the registry.
  • A release that is deployed to a stage can either be promoted to the next stage or discarded. A promotion allows the deployment on this future stage.
  • A release that was deployed before can be deployed again (rollback).
  • In bigger SaaS applications, not all roles might have the right to perform these actions on all products! We need Resource-Based Access Control (ReBAC).

At my last company I implemented such a release management process. We used the tools we were familiar with: Jenkins, Git, kubectl. And it worked. However, the solution was not good. A good solution guides a user in the right direction and makes it difficult for them to make mistakes. Our solution did not do that.

Now, with more experience I firmly believe that this problem deserves a proper solution. A solution to the release management in CD will allow your SaaS business to scale, avoid deployment mistakes, and make your work environment a better place. Therefore, I built CDRM (continuous delivery release management). In case you have a similar problem, do not hesitate to use it. It is open source. See CDRM.

Please share your thoughts!

Top comments (0)