Editor’s note: This article first appeared on Scotch.io.
The practice of GitOps is the next incarnation of continuous delivery. It allows developers to reach into the traditional purview of IT Operations – a land of many historical gates – and automatically update applications and the infrastructure they live on in production. In GitOps, the single source of truth for all change management and version control is the software configuration management (SCM). GitOps leaves behind traditional ITIL-type management and treats infrastructure and applications as versioned artifacts complete with the same granular audit trail (commit IDs, time stamps, etc.) that is captured during software development.
I do declare!
The idea of GitOps is that the desired state of the entire system is stored in version control via Git commits. If stripped to its bones, you can think of GitOps as a system for file versioning. A key tenet of GitOps is that the application and environment’s ideal state is described using declarative specifications via the use of a config file.
This means that the configuration is governed by a set of facts, not a list of instructions. Instead of giving you a recipe to build a thing, it gives you a rule to check that the result is correct. You can describe your entire CI/CD pipeline in the code repo this way. To make changes to the desired state, a developer issues a pull request which basically tells everyone about changes that you’ve published to the repo and instructs the repo to pull in the changes. By using Git you get version history, audit logs, rollback capabilities and the ability to see who changed what and when.
Feature flags + GitOps
When we consider GitOps, the use cases that come immediately to mind are container orchestration and cluster management – specifically with the declarative tool Kubernetes. Not many immediately think of feature flags. So why is this important that we consider feature flags? It’s important because the vision of GitOps is holistic control of the entire system. Feature flags are often perceived as living ‘outside the law’. We believe that feature flags – especially at scale – become an immensely powerful tool if they are managed with the same rigor, governance and importance as the rest of your code. If you want to use GitOps to manage feature flags, their desired state must be described using a config file. But that’s not all.
There is a right way to apply GitOps to feature flagging
Feature flags are a sticky wicket. They hold the power to make production changes, but they don’t undergo the same burden of proof for production readiness as the rest of your code. This needs to evolve if it’s going to become a staying part of the software delivery lifecycle.
If we want to manage feature flags with GitOps, the desired state (described by declarative specifications) must be saved to a config file. We use YAML so that it’s human readable and editable. When updates to the desired state need to be made you simply merge the config. This change is submitted via PR which establishes an audit trail and ensures that the correct folks are verifying the change – which is exactly how it happens when someone changes code in the application or updates infrastructure settings. We believe this is the right way to manage feature flags with GitOps. It’s also the truest to the vendor neutral aspirations of the practice.
To our knowledge, only CloudBees Rollout can support this. Some of our competitors also have a config file and their SDK knows how to read and change it. However, it’s not human editable. It’s also not automatically saved in an SCM like GitHub. They force users to circumvent the established process by which they manage code in order to manage feature flags. For example, if a feature rollback is required customers are forced to use a third-party dashboard, not Git where this should be taking place.
Some ‘Git’y use cases for feature flags with CloudBees Rollout
Configuration as code. This term is often used interchangeably with infrastructure as code (IaC), but it’s actually distinct. IaC is about the management and provisioning of the stack whereas CaC is about automating the migration of configurations between environments. This is all about making environment provisioning cookie cutter. No snow flakes allowed. We treat our feature flags the same way that configuration as code treats applications (we use CaC terminology here instead of IaC because feature flags are not part of the infrastructure, they are on the software application). When we’re talking GitOps this means we can document changes and versioning that happens in our feature flags in the same way the PR tracks changes and versioning for application changes in the SCM. Pushing changes to the main branch triggers an event to be consumed by the SDK. The system then understands how to update the YAML file configuration desired state for the feature flag.
CloudBees Rollout saves all feature flag and targeting data as local YAML files which are saved in a Git repository. Making changes to the local YAML files updates live CloudBees Rollout feature flagging data. We take advantage of Git’s characteristics as a decentralized version control system, allowing you full traceability and revision history even when working locally. If you edit your feature flags directly in GitHub and commit changes to the main branch, an event is triggered back to the dashboard and is also reflected in Rollout’s audit logs. If changes are done via the dashboard, the dashboard acts like a Git client and updates the YAML files on GitHub.
Once you are taking a configuration as code approach to your feature flags you can achieve these awesome use cases;
- Governance and accountability: Because all changes are in Git, every commit produces an audit trail. You know who changed what in your feature flags and when. And because everything is managed by PR you can have team members approve your changes for extra accountability.
- Progressive delivery, change and version control: Feature flags allow you to separate feature deployment from code releases. Eliminate long-lived branches by wrapping your features in a feature flag when they are committed to the main branch. They can remain in an ‘off’ state until they are code complete. Cutting down on branching in Git allows you do more progressive releases (increasing release velocity with bite size chunks). A GitOps approach to feature flagging ensures that every change is accounted for.
- Cloning environments: With configuration as code you can clone feature configurations between environments (dev, QA, prod, feature X,Y,Z) by tracking feature toggle changes alongside your existing configuration.
- Feature flag automation: When you have editable config file which describes the system desired state, it’s easy to run automation (for testing or deployment purposes) based on various desired states. You can automate deployment to a subset of the user base or a variety of segmentations using a GitOps approach to feature flags. When feature flags are set up as config files, it’s very easy to move the system to a new desired state. Other alternatives, such as the traditional CI process of using the rest API to change the feature flag, are more complex. Instead of waiting for authentication against the server, waiting for the network to report to the server and on… and on… using GitOps to manage feature flags is as simple as changing the config file in the Git repo to change the state.
- Roll back feature changes with a Git command: There comes a time in every developers life when you need to roll back that commit. You can use feature flags to do that with a simple git revert command. Since CloudBees Rollout stores configuration code in Git, you can use branches to isolate changes to rollback in time and to work in parallel streams without affecting your production/Staging environment.
Even if adopting GitOps is still aspirational for your team, you can use CloudBees Rollout to manage your feature flags. API integration allows you to link to your favorite performance, analytics, monitoring and APM tools making it an easy fit regardless of how you manage the bridge between Dev and Ops.
Want to try it for yourself? Check out the 14-day free trial.
Top comments (2)
How is configuration which changes per environment managed? I'm thinking mock services to staging services to production.
This configuration lives in a git repo separate from the application?