Feature flags provide so much for software organizations: they allow teams to separate code deployment from feature release, test in production, run experiments, and more. However, some rules apply to the feature flagging process that are easy for teams to overlook. I’ve gathered the best practices of feature flags from Split’s team of experts so that your team doesn’t have to think twice about how you manage and maintain your flags.
Feature Flag Management
DO: Scope each flag to a single team
Each feature flag should be owned by a single team to avoid overlap in accountability. Always aim to have each flag owned by a clearly identified team. That team is responsible for driving the feature’s rollout, monitoring its performance, and retiring the flag when it’s time. Then, when the time comes for cleanup, that team will have all of the details necessary to remove it, and there won’t be confusion in ownership.
“It became clear to us from the beginning that we had to find ways to give each team a space to organize their flags and prevent teams from stepping on each other’s toes when it comes to managing feature flags. ”
Patricio Echague, Co-founder & CTO, Split Software
DON’T: Treat all flags the same
You can use feature flags for many purposes, including release toggles for half-finished code, dividers between segments of an A/B test, and as a kill switch for the Ops team to simplify rollbacks. Because of the different roles that a feature flag can have, you shouldn’t treat them the same. If you’re using a flag as a release toggle, developers should manage it. If you’re using the flag to run an A/B test, your product manager should manage it. If you’re using a flag as a kill switch, your Ops team should manage it.
“Not all flags are created equal. Some are meant to be permanent; others are meant to serve as a circuit breaker, and others to enable experimentation”
Patricio Echague
Percentage Rollout
DO: Maintain consistency
Suppose User A visits your site, and your feature-flagging system decides that User A should see a specific feature (variation “on” of the feature). In that case, User A should continue to see this variation of the feature no matter how many times the flag is evaluated. Increasing the exposure to a broader user population should not affect the current specific exposure of variations to users—if a user experienced a feature when it ramped to 40%, that user should continue to see it as it ramps to 60%, 80%, and so on.
DON’T: Get In Your Own Way
Let’s say you are migrating from an old email service provider to a new one. You start sending new traffic to the new provider, continue to roll it out to the rest of the population, and then stop paying for the old service. At some point, flipping back to the old provider is no longer possible because you’ve stopped paying for the service. The flag is still in place, but it’s no longer safe to turn it off. You don’t want to get to the point of no return with a feature flag if you need to go back to the old service.
“Risk mitigation is an important use case for feature flags. Make sure turning off a feature fast is always an option. We’ve had to press the red button(kill switch) quite a few times”
Patricio Echague
Implementing Your Feature Flags
DON’T: Make your feature flagging system a potential single point of failure
Your feature flagging system should not be on the critical path of your application. There needs to be a graceful degradation path so that in case of an outage, your system can continue to function. If the system can’t figure out what the feature flag value should be, it should use the most recent cached value, and if it can’t get that, it’s common to fall back to a default of all the flags off.
DO: Make sure your feature flag changes are backward compatible
Let’s say you are in front of a frozen lake, and you are not sure if it is stable enough to hold you while you walk on it. You stretch out your right leg, gently apply pressure, and then slowly move your left leg to meet your right leg. You do this continuously to build your confidence until you reach the other end of the lake. If, however, you feel the ice crack underneath your leg, you would slowly revert your actions and go back. This idea is called the expand-contract pattern.
Just as you need different versions of your software to work simultaneously, you also need different versions of the database to work at the same time. In reality, you can’t stop the world, update your code, and then start the world again. You need to do it gradually, which means more than one version will have to work with the database, simultaneously. When you migrate to a new database schema, you expand the schema to work with the old version and the latest version and then eventually contract to work only with the new. This allows the gradual rollout of recent changes to be successful while knowing that the old code still works for your users who haven’t gotten it yet. For this process to be effective, each change should be backward compatible so that just in case you need to roll back, your database will still be valid.
DO: Build a wrapper around your feature flagging framework
There are two parts to implementing a feature flag: the evaluation and the logic. You should separate the logic from the evaluation because it prevents the details of your feature flagging implementation from leaking all over your codebase. It is also easier to test, rather than having to mock out the feature flagging system itself. By separating these pieces, it allows you to base a decision on more than just feature flags. For example, You can use a separate A/B testing framework or a separate entitlement system.
DO: Treat your feature flagging code as production code
If you require two code reviews for your pull request before someone gets to merge their code to master, you should also require two reviews for any configuration changes to your feature flags.
Additionally, if you wouldn’t add multiple ugly nested if statements in your regular codebase, you shouldn’t add them to your feature flag code. Although you are not writing code in the feature flagging system itself, the changes you make here directly impact your users in production, so you want to be careful. By having the same approval process for feature flag configuration as you do for production code, you can prevent people from making a change to a flag configuration that they don’t own or that they meant to make for a different flag. Approval flows help to avoid common mistakes.
_“One thing I see a lot of teams doing is not treating feature flag code as if it’s real production code” _
Pete Hodgson, Software Delivery Consultant
DO: Implement flagging decisions close to business logic
Your flagging decisions should be encapsulated as much as possible, thereby avoiding the need for every part of your codebase to be aware of every component. You don’t want to change the scope of your components just to support feature flagging decisions.
DON’T: Manually sync feature flag configurations across environments
There should be a system for automatically promoting changes through environments. When there is automation in place, the chances of making a mistake are low, and you save time in the process.
Deprecating Feature Flags
DO: Have a plan to remove your feature flags
You should be intentional about removing flags once they have served their purpose. Whether you are using the flag to test in production with your internal teammates, run an experiment, or serve as a kill switch, you should know the criteria for the flag to meet its definition of done. Keep in mind that different flags have different sunsetting timelines depending on their purpose.
“The most common challenge I see teams facing with feature flags is that they have too many feature flags”
Pete Hodgson
DON’T: Keep old feature flag code in your codebase
Once a flag has served its purpose, it should be retired following the plan you created. You don’t want to be stuck with hundreds of feature flags that served their purpose and then were just forgotten about. Something super useful you can do when thinking about removing a flag is to set up alerts in your feature flag system that will notify you when a flag is stale or hasn’t had impressions in a while. You want to make sure that you don’t have old feature flags in your code after deleting your flags in the UI.
Do: Use Feature Flags!
Feature flags provide software development teams with the necessary tools to have successful feature releases. They also provide organizations with the foundation for continuous delivery, and metrics analysis to improve upon KPIs. In implementing feature flags, there are several things to keep in mind. The above tips and tricks will help you uplevel your processes for the best user experience and greatest overall business impact.
Learn More About Feature Flags
Whether you have never used feature flags before or have been using them for years, it’s never too late to learn more!
- Implementing Testing in Production
- A Quick Guide to Implementing Feature Flags with Spring Boot
- The Benefits of Feature Flags in Software Development
- Feature Flag Maintenance
- Pros and Cons of Canary Release and Feature Flags in Continuous Delivery
To stay up to date on all things testing in production and feature flagging, follow us on Twitter @splitsoftware, and subscribe to our YouTube channel!
Top comments (4)
Awsome post. I am interested in maintaining a database version to be compatible with multiple version of application. it would be great if you will write more post about this. it is the are I am stuck for longtime. Do you have multiple version of app and database deployed together in the test env and have the testers to test each version combinations? Is it up to the developer to make the backward compatibility happen or is there some tool or methodology to help? Should I bundle database migration script such as flyway or liquibase with the application so that the database schema is upgrade when the application is deployed, or should I have a separate deployment pipeline to upgrade db?
thanks in advance 🥰
Ngaww: Sorry for the slow reply... we thought we had notifications set up right but alas, no! There are indeed tips for working with databases.
Here is a post Talia Nassi wrote in June on this very topic: split.io/blog/database-migration-w...
Again, sorry for the delay on the reply... I'll work out a way to stay more on top of them!
/dtk (Dave Karow - CD Evangelist @ Split)
Update: I ported the blog I mentioned in the above comment over to dev.to, so it's here now: dev.to/split/database-migrations-w...
hi, please don't worry, I appreciate that you take time help me finding the answer 😊