Mustafa ERBAY

Posted on Aug 24 Originally published at mustafaerbay.com.tr

Feature Flag Architecture: Decoupling Deploy from Release

#featureflags #deployment #architecture #trunkbased

A feature reaching a user is actually two separate events, though we spent years treating it as one. Code landing on a server is an engineering event: the build passed, the image changed, the process restarted. A feature becoming visible to users is a business decision: is marketing ready, does support know, is there a rollback plan? A feature flag is the mechanism that separates those two events — Microsoft's definition of feature management says exactly this: a software development practice that "decouples feature release from code deployment."

The separation has a price. Every flag is a fork at runtime and a promissory note of unclear maturity in your codebase. In this post I treat a flag not as an if, but as an architectural component with its own type, lifespan, default, evaluation path, and removal date. I covered the ownership, approval, and audit side of configuration earlier in feature flags and configuration governance; the question here is different — the flag's own lifecycle.

The Dead Platform's Flag: A Case from My Own Repo

This blog's social sharing pipeline runs on boolean fields carried in each article's frontmatter: twitter, linkedin, bluesky, devto. They aren't feature toggles in the classic sense — they're "was this work done" markers. But they behave like flags on the system, and one day they blew up exactly like a badly designed flag.

On May 16, 2026, the Mastodon account got suspended; the API started returning 403 "Your login is currently disabled". The reflexive move looked right: leave the flag off instead of ripping out the integration. The result sits in the code's own notebook:

16 May 2026 — Mastodon tamamen kaldirildi. Hesap (mastodon.social) askiya
alindi (403: "Your login is currently disabled"), mastodon: false flag'i
findNextTrPost'ta ayni eski TR postu sonsuz dongude yakaliyordu → yeni
postlar payaslsimda asla siraya gelmiyordu. Tamamen sokuldu.

(The note says: Mastodon was removed entirely. The suspended account's mastodon: false flag kept trapping the same old post in an infinite loop in findNextTrPost — new posts never got their turn.)

The queue selector worked as "find the oldest post with an unshared platform." mastodon: false matched that definition forever, because that platform could never become true again. Every run the system picked the same old article, failed to publish it, and never moved on. Nobody got an alert — technically nothing was "erroring," new content just quietly stopped being shared.

The defect here isn't the value false; it's that value's semantics. In the system, false meant "not done yet, queued"; what I meant was "never again." Load two meanings into one cell and the flag stops being a switch and becomes a trap. That's why the first question of flag design isn't "on or off" — it's: what is this value telling the system?

Type Determines Lifespan

Not all flags are the same thing, and the most useful distinction is not the "release/experiment/ops" trichotomy most teams use, but how long it's allowed to live. GitLab bakes this directly into the type in its own development process; the documentation puts a maximum lifespan next to each type:

Type	Default	Maximum lifespan
`gitlab_com_derisk`	must not be `true`	2 months after merging into the default branch
`wip`	must not be `true`	4 months
`beta`	can be `true` to "release" a feature to everyone in beta	6 months
`ops`	should be `false` in most cases	Unlimited, but must be evaluated every 12 months

The table shows the four main types; the documentation also lists experiment (6 months) plus special types like worker and markdown_cache. The table itself carries a freshness lesson: GitLab's old development type is deprecated, replaced by gitlab_com_derisk, wip, and beta. Most articles on the internet explaining "GitLab development feature flags" don't know about this split.

The point isn't the names. The moment you assign a flag a type, you've assigned it an expiry date — and flags without dates don't die, they accumulate. GitLab's own rationale is short and sharp: flags "should remain in the codebase for as short a period as possible to reduce the need for feature flag accounting." Accounting is exactly the right word; every open flag is one more "which branch of reality are we on right now" question every new engineer has to keep in their head.

They also give one universal rule: every newly introduced flag should be disabled by default and used with an actor.

The Default Value Is a Product Decision, Not an Error Path

I'll get to actors, but defaults first. Most teams treat the default as a technical detail — "what do we return if the flag can't be read." It isn't.

The OpenFeature specification makes this a hard rule: evaluation methods on the client must not throw exceptions or otherwise abnormally terminate; in the event of abnormal execution they always return the default value (Requirement 1.4.10). So when your flag server is down, the network is gone, or the provider isn't ready yet, your application keeps running — but entirely inside the world your default values describe. That's also why calls made before the provider is ready matter: the spec requires providers to emit PROVIDER_READY when initialization completes and PROVIDER_ERROR when it fails (2.8.2, 2.8.3), and every flag read in between falls back to its default.

The concrete example in my own pipeline is the burncpu field. When Fediverse sharing was added on July 14, 2026, the archive held 974 older posts, none of which had this field in their frontmatter. "What happens when the field is missing" had two possible answers:

false → "none of them were shared" → the bot dumps 974 old posts onto the Fediverse back to back. In other words, spam.
true → "consider them all shared, skip" → only future posts get shared.

The code chose the second. That wasn't an error-path choice, it was a product choice: don't replay the past, look forward. As a general rule I'd put it this way — kill-switch-style flags should default fail-safe (off: without the feature the system merely does less), while permission/security flags should default fail-closed (no access). The most dangerous flag is the one whose default is "whatever prod happened to have last time"; such a flag has no real default, only luck.

Where Is the Flag Read From: Evaluation Architecture

If the flag's value doesn't come from code, it comes from somewhere — and that somewhere is now a critical dependency of your system. OpenFeature — the vendor-neutral flag API accepted into the CNCF in 2022 and promoted to incubating in November 2023 — splits this layer in two: the client your application talks to, and the provider that translates to the flag system behind it. A provider may wrap a vendor SDK, call REST, or read a local file; application code sees no difference.

The evaluation result is standardized too. get<Type>Details calls return a structure carrying value, flag key, variant, reason, and — on failure — error code (1.4.3–1.4.8). That reason field is, I think, the most underrated piece of flag observability: it takes values like STATIC, DEFAULT, TARGETING_MATCH, SPLIT, CACHED, DISABLED, STALE, ERROR, UNKNOWN (2.2.5). The answer to "did the user see the feature" is not true/false; the difference between true because TARGETING_MATCH and true because DEFAULT is the entirety of your 3 a.m. debugging session.

There's one more architectural split to decide up front: the specification defines two paradigms, dynamic-context (server-side) and static-context (client-side). Server-side, every evaluation call carries its own context (1.3.1.1) — because the same process serves thousands of different users. Client-side, the context is set once per user and evaluation methods take no context parameter (1.3.2.1). An abstraction that mixes the two is the fastest route to showing user A the variant assigned to user B in your mobile app.

A Percentage Rollout That Isn't Sticky Isn't a Rollout

Percentage rollouts are the most requested and most misbuilt flag feature. The broken version: generate a random number per request and compare against the percentage. That sends a user to the new UI on one request and the old one on the next, producing the famous support ticket where the feature vanishes the moment they hit submit.

The right version is a decision that sticks to an actor. GitLab's documentation puts it neatly: actors make percentage rollouts sticky, and an actor who had the feature enabled at 1% keeps it enabled at 10%, 50%, and 100%. An actor can be a user, a project, a group, or a CI runner. There's a side benefit too: since the actor is known, you can filter logs and errors by it.

On the Azure App Configuration side, the same idea lives in the Microsoft.Targeting filter. An audience is defined by Users, Groups, DefaultRolloutPercentage, and Exclusion; the feature turns on if the user is listed, falls within a group rollout, or lands inside the default percentage — and the exclusion list overrides everything. When combining filters, requirement_type comes into play: Any (the default when unspecified) needs one filter to pass, All requires every filter to pass — a rule like "50 percent of users during the time window" can only be built with All.

And a subtler trap: if configuration updates live, a flag's value can change in the middle of a single HTTP request — the top half of the page renders the new flow, the bottom half the old one. In .NET the answer is IVariantFeatureManagerSnapshot, which caches the first evaluation and returns it for the rest of the request. If your stack has nothing like it, reading the flag once at request start and carrying it in context does the same job.

Consistency doesn't end at the request boundary either. When does a queued background job read its flag — at enqueue time or at processing time? Mid-rollout those give different answers, leaving you a queue processed half with the old behavior and half with the new; the same ambiguity plays out between two services that each evaluate the flag independently. The practical fix is to write the decision into the work itself: put the flag value into the payload when enqueuing, and have the worker skip re-evaluation. Client-side setups have a leak of their own: if flag definitions are shipped to the browser, the name of an unannounced feature (say, enable_new_pricing) is visible to everyone in the JS bundle — enough for a curious eye to read tomorrow's launch today.

Flipping a Flag Is a Deployment Too

"Flags are instant and risk-free" is my favorite lie in feature flag marketing. Flipping a flag changes prod behavior; the only difference is the change never went through CI — which is usually not good news.

AWS AppConfig models this directly as a deployment problem — and grabs it by exactly the right handle. Feature flags there are a distinct configuration profile type (with the hosted URI), separated from freeform configuration. When you roll out a change you pick a deployment strategy, and the strategy defines four things:

Deployment type: Linear (even steps of the growth factor) or Exponential (G*(2^N) — with a growth factor of 2, it proceeds 2%, 4%, 8%, …).
Step percentage / growth factor: the percentage of callers targeted at each step.
Deployment time: the window during which the deployment is processed in intervals. As the docs stress, this is not a timeout.
Bake time: the period after the configuration reaches 100% of targets during which CloudWatch alarms are monitored. If an alarm fires during this time, AppConfig rolls the deployment back.

Bake time is the star of that list. Most flag incidents grow because someone said "flipped it, looks fine" and moved on five minutes later; the bad news usually arrives in hour two — a queue backs up, or a dependency starts rate-limiting you. A flag switch with no automated rollback attached is just a prod change performed faster. It's the flag-shaped extension of the logic in error budgets as a release gate.

The Kubernetes Example: Maturity Level Determines the Default

One of the largest-scale practitioners of flag discipline is Kubernetes itself. Features live behind feature gates, and the maturity level directly determines the default. A stale piece of folklore circulates widely here, so I'll quote the docs directly: built-in alpha API versions are disabled by default, and built-in beta API versions are also disabled by default — the only exception being beta API versions introduced before Kubernetes 1.22, which shipped enabled. "Beta means on by default" has been repeated for years and is no longer true for new APIs.

One more detail: you enable and disable API groups with --runtime-config, but the change requires restarting the API server and controller manager to take effect. Even in the platform's most mature flag system, not every switch is instant. When designing your own flags, if you can't answer "how long until this change takes effect, and how long to reverse it," that flag isn't a control yet — it's a hope.

Flag Debt and Test Combinatorics

Independent n flags mean, in theory, 2ⁿ distinct system behaviors. At ten flags that's 1,024 combinations, and you are not testing all of them — nobody is. In practice you need to test two states: the combination live in prod today and the combination you're heading toward. Everything in between spins out of control the moment flags become interdependent — when one being on changes what another means.

And honestly, the debt this post warns about sits in my own repo too. Inside generate-content.ts there's this line:

if (process.env.ENABLE_DISABLED_EXTERNAL_COVERS !== '1') return false;

This flag was added on June 30, 2026, when the external cover image strategy was reverted (the Unsplash endpoints were returning 503, the generated covers carried no branding) — instead of deleting the code, it went behind a gate defaulting to off. That code hasn't run in two months and most likely never will again; the only reason it's there is "maybe we'll go back." Not a kill switch — indecision. The honest name for that line isn't ENABLE_DISABLED_EXTERNAL_COVERS, it's OLD_CODE_KEPT_FOR_THE_ARCHIVE — and the archive is called git history.

Decision Framework

The list I run through when adding a new flag, distilled from the sources in this post:

Write the type and the expiry date at the same time. De-risk, work-in-progress, beta, or operational kill switch? The first three are measured in months; the last can be permanent but gets reviewed yearly.
Choose the default as a product decision. The answer to "what if the value can't be read" is the world your users see when the system is down.
Keep the value's semantics single-meaning. If false means both "not yet" and "never again," your queue will eventually eat its own tail.
Stick the percentage to an actor. A rollout that rolls dice per request shows one user two different products on two requests.
Guarantee consistency across a request. Read the flag once at request start; carry it in context.
Automate rollback and leave bake time. If no alarm is wired up, the job isn't done when the rollout hits 100%.
Log the evaluation reason. value alone is not enough; without reason and variant you can't reconstruct the incident.
Attach removal to the same issue, not to a future sprint. If the removal task isn't on the flag's birth certificate, that flag never retires.

Conclusion

Feature flags buy real freedom by separating deploy from release: ship code as often as you like, open the feature when you're ready, close it within minutes when it goes wrong. The price you pay tends to be hunted in the wrong place. It isn't the runtime cost of a few branches — that's too small to measure.

The real price is epistemic: in a flagged system, the code alone no longer tells the truth. You cannot know what prod is doing by reading the repo; the truth lives half in code and half in the flag store. That's why the center of mature flag discipline isn't technology but accounting: which flags are on, who turned them on, when do they come out. Ask your own setup this question — "can I produce, in thirty seconds, the list of flags currently on in prod and the removal date of each?" If the answer is no, you didn't decouple deploy from release; you just opened one more place to lose your system's truth.

DEV Community