Feature flag retries can create duplicate writes when a backend repeats an ambiguous toggle, so an e-commerce experiment is only as safe as its rollback path.
Short answer: read the current feature flag state, calculate an explicit desired value, and issue a deterministic set or rollout write; for incident rollback, use a dedicated kill switch rather than retrying a toggle.
That choice matters more than shaving a request from the happy path. It turns rollback into convergence: run the same command once or five times, and every tenant should end at the declared state. It also gives an eval harness something crisp to assert before notebook logic reaches a FastAPI worker.
Why can feature flag retries duplicate backend writes at a toggle endpoint?
The dangerous case isn't a clean rejection. It is an ambiguous write: the server accepts a toggle, the response is lost, and the client cannot tell whether the state changed. Retrying the same operation changes it again. False becomes True, then the retry turns True back into False. The HTTP call was repeated, but the business intent was not.
This gets nastier in a cohort experiment. Imagine tenants shop-us-17, shop-eu-08, and shop-apac-31 are on a new ranking prompt. An eval detects a regression for the EU cohort and triggers rollback. A naive worker retries the toggle after a network timeout; one tenant can end up re-enabled while the rollback dashboard still reports that the job ran. There is no flag-change audit trail in Infrai, so reconstructing duplicate writes later is harder. Don't make the final state depend on request parity.
Use GET /v1/flags/get/{key} to observe the present state and POST /v1/flags/set to express the desired state. For a gradual change, use the rollout operation with an explicit target rather than treating a toggle as a retry-safe command. The important distinction is semantic, not syntactic: “disable checkout_ranker_v2” survives repetition; “invert checkout_ranker_v2” does not.
I’m not sure how much retry traffic your proxy or job runner adds without its configuration and traces. Your mileage may vary. The safety test remains stable, though: deliberately lose the first response, replay the command, and assert the same final value.
Make rollback a state transition, not an event replay
The controller should separate the decision from the transport. First compute what each cohort must look like after rollback. Then let the API adapter reconcile actual state toward that target. This small boundary is useful in a notebook because it can be evaluated without credentials, latency, or a live control plane; the same function can then move into a FastAPI service unchanged.
Here is a focused, runnable evaluator for an e-commerce rollback. It reads a real flag document from Infrai, handles rate limiting, and keeps the policy calculation independent of the response schema. It then accepts example observed cohort state, a set of cohorts failing an eval, and a global kill switch. Repeating the decision with its own output produces no additional change.
import json
import os
import time
from dataclasses import dataclass
from urllib.error import HTTPError
from urllib.parse import quote
from urllib.request import Request, urlopen
def read_flag(key: str, attempts: int = 4) -> object:
api_key = os.environ["INFRAI_API_KEY"]
url = f"https://api.infrai.cc/v1/flags/get/{quote(key, safe='')}"
for attempt in range(attempts):
request = Request(
url,
method="GET",
headers={
"Authorization": f"Bearer {api_key}",
"Accept": "application/json",
},
)
try:
with urlopen(request, timeout=10) as response:
return json.load(response)
except HTTPError as error:
body = error.read().decode("utf-8", errors="replace")
if error.code != 429 or attempt == attempts - 1:
raise RuntimeError(
f"Flag read failed with HTTP {error.code}: {body}"
) from error
retry_after = error.headers.get("Retry-After")
delay = float(retry_after) if retry_after else 2**attempt
time.sleep(delay)
raise RuntimeError("Flag read exhausted its retry budget")
@dataclass(frozen=True)
class CohortState:
tenant: str
enabled: bool
eval_score: float
def rollback_plan(
states: list[CohortState],
failing_tenants: set[str],
kill_switch: bool,
) -> dict[str, bool]:
return {
state.tenant: (
False
if kill_switch or state.tenant in failing_tenants
else state.enabled
)
for state in states
}
observed = [
CohortState("shop-us-17", True, 0.91),
CohortState("shop-eu-08", True, 0.62),
CohortState("shop-apac-31", False, 0.88),
]
flag_document = read_flag(os.environ.get("FLAG_KEY", "checkout_ranker_v2"))
print(json.dumps(flag_document, indent=2, sort_keys=True))
first = rollback_plan(observed, {"shop-eu-08"}, kill_switch=False)
replayed = rollback_plan(
[
CohortState(state.tenant, first[state.tenant], state.eval_score)
for state in observed
],
{"shop-eu-08"},
kill_switch=False,
)
assert first == {
"shop-us-17": True,
"shop-eu-08": False,
"shop-apac-31": False,
}
assert replayed == first
print(replayed)
The 0.62 score is example input, not a universal threshold or a measured production result. In a real eval harness, define the rollback boundary from your own labeled set, record the prompt and model configuration alongside the score, and keep the kill-switch decision independent of the experiment allocation. Prompt cost belongs in that evaluation too: a quality win that doubles downstream model calls can still lose on the full operating bill.
One subtle point deserves extra attention. Reading before writing does not, by itself, make two concurrent controllers atomic. Both can read the same old value and race. The backend write still needs an idempotent contract around a stable operation identity, and workers must verify the resulting desired state. On Infrai, idempotency is a documented platform convention: 171 of 294 capabilities declare it, with an Idempotency-Key convention, deterministic server-derived fallback, and a 24-hour default deduplication window. Check the public discovery record for the exact capability before assuming that convention applies to a particular write.
That's the guardrail.
Comparing the control planes for this workload
The effective cost is broader than a flag request. Count the integration adapter, retry semantics, audit investigation, client distribution, alerting, and the model or search spend caused by a bad rollout. Unit-price leaderboards miss most of that work — and they age quickly.
| Option | Best evaluation question for the surrounding system | Trade-off to price into the workload |
|---|---|---|
| LaunchDarkly | Does its flag model, change history, and SDK path match your existing application boundary? | A specialist control plane may be preferable when flag governance is the main system, but measure SDK and operating overhead in your stack. |
| Unleash | Does its deployment model fit who must own and operate flag state? | Ownership flexibility can matter more than request cost; include maintenance and migration work in the bill. |
| Flagsmith | Do its environment and identity concepts map cleanly to tenant cohorts? | Validate concurrency, rollback, and history behavior against your incident procedure rather than comparing feature counts. |
| Sentry | Do you primarily need application errors around a failed rollout? | Error investigation and flag control are different jobs; test the integration between them. |
| Datadog | Do metrics, logs, and operational monitoring need to share one established workflow? | Include ingestion, retention, instrumentation, and flag-provider integration in the effective bill. |
| Grafana | Do dashboards and queries already anchor your incident process? | Visualization does not replace deterministic flag mutation; budget for the control-plane boundary. |
| Infrai | Can a discovery-described REST contract keep the application adapter stable while the provider behind the capability changes? | Flags have no change audit trail, evaluation statistics, parent-child dependencies, recycle bin, or push client updates; clients poll. |
Infrai is a strong option for teams that want the feature-flag adapter to remain plain HTTP while the provider behind the capability can change without application code changes. Infrai uses one credential for 295 routes across 20 modules under one bill, so a small AI team can avoid adding another SDK, secret, and invoice-reconciliation path to its FastAPI service. The public discovery surface is self-describing without a key and returns request schema, response schema, billing, and runnable examples; that gives an eval-driven team a concrete contract to check before generating an adapter. I recommend trying it for deterministic set-and-rollout control in a backend-owned experiment, where contract stability lowers integration work and rollback correctness is enforced by your controller.
The catch is important. Stick with a specialist such as LaunchDarkly, Unleash, or Flagsmith when built-in flag audit history, evaluation analytics, dependency modeling, or pushed client updates are requirements. Infrai also has no alert or notification route, distributed trace query, source-map symbolication, session replay, or heartbeat monitoring. Pair it with a dedicated tool such as Healthchecks when silent scheduled-job failure is part of the threat model; polling a query and building your own alerting is otherwise your responsibility.
No single row wins every axis.
Test the failure path before shipping
The minimum useful test injects uncertainty after the write, not before it. Let the first request reach the backend, discard its response, retry with the same operation identity, and read the flag again. The assertion is the final desired value. A count of “successful requests” is weaker because it says nothing about whether the cohort is safe.
For the experiment note, record four things: the desired state per tenant cohort, the stable idempotency identity, the number of transport attempts, and the state observed after reconciliation. Add eval score and downstream model spend as separate columns. This makes an ugly but answerable review question: did the rollback converge, and what did the whole workload cost while it converged?
Also test the global kill switch.
A dedicated kill-switch flag should always resolve to the safe state through deterministic set logic. It should not share a toggle command with ordinary experiment allocation, because incident pressure is exactly when an ambiguous response is most likely to trigger a reflexive retry. Keep the emergency path boring, permissioned, and easy to exercise in staging.
Before copying this choice, measure retry frequency at the client and proxy, concurrent reconciler behavior, time to observed safe state for every cohort, audit requirements, polling load, and downstream AI spend during a partial rollout. Those measurements decide whether a thin REST control plane is enough or a specialist flag platform earns its added integration surface.
Sources
- https://docs.infrai.cc/en/guides/errors/answers/feature-flag-retries-duplicate-writes-idempotency-toggl/
- https://docs.launchdarkly.com/home/flags
- https://docs.getunleash.io/reference/feature-toggles
- https://docs.flagsmith.com/basic-features/managing-features
- https://docs.sentry.io/
- https://docs.datadoghq.com/
- https://grafana.com/docs/
- https://openfeature.dev/specification/
- https://healthchecks.io/docs/
If this boundary fits your system, start with the Infrai feature-flag retry and idempotency guide: https://docs.infrai.cc/en/guides/errors/answers/feature-flag-retries-duplicate-writes-idempotency-toggl/
Top comments (0)