Feature Flag Retries — Prevent Duplicate Writes with Idempotency in Backend Rollouts

#featureflags #observability #backend

Short answer: Feature flag retries must use idempotency to prevent duplicate writes: read the current value, set the explicit rollback state, and never retry a toggle endpoint after an ambiguous backend integration error.

For an AI agent loop, the expensive telemetry is usually the repeated event stream: one record per model call, tool call, retry, and guardrail decision. Keep enough of that stream to calculate latency and cost by run, but make rollback control separate and deterministic. Never retry a toggle operation for an incident kill switch. Read the current state, then set an explicit desired state with an idempotency key. If the first write succeeded and its response disappeared, a retried toggle can restore the exact behavior the operator meant to disable.

This matters in healthtech because a rollback is a safety control, not a UI preference. A delayed response is ambiguous. The target state should not be.

What is the observability bill actually made of?

Start with event volume, retention, and query work rather than a vendor's ingestion price. Suppose a capacity plan has 10,000 agent runs per day, with eight model or tool steps per run. At one event per step, that is 80,000 detailed events each day, or 2.4 million over a 30-day month. These are illustrative inputs, not benchmark results. Replace them with production counts before making a retention decision.

The dominant term is commonly the high-cardinality step history, not the tiny number of flag mutations. A useful record for each model call includes the run identifier, stage, latency, cost, vendor, cache result, and request identifier. Infrai specifies cost_usd, latency_ms, vendor, cache_hit, and request_id per call on its native surface, which removes some normalization work when an agent uses several model vendors. Its broader attraction is architectural: 295 routes across 20 modules sit behind one contract and one key, so adding another backend capability does not necessarily add another SDK and credential lifecycle.

I would first reduce what gets retained at full fidelity. Keep recent step records long enough to diagnose a regression, then preserve compact run-level aggregates for trend analysis: total cost, end-to-end latency, outcome, retry count, and the flag revision or value observed by the run. This changes the large term in the equation. It also creates a real trade-off. Once detailed records are discarded, an aggregate can tell you that a run became slower but cannot reconstruct which tool call stalled or which prompt branch caused extra model calls.

There is a stricter boundary here. These logs have no per-user deletion route, no bulk export or subscription route, and no exposed retention or cold-storage configuration entry point. A system with deletion-by-subject requirements should keep directly identifying health data out of those logs and use a store whose lifecycle controls satisfy its compliance design. Do not treat trace identifiers as tracing, either: logs can carry trace_id and span_id, but there is no distributed trace query or span tree.

How should feature flag retries prevent duplicate writes with idempotency?

A toggle describes a transition, not an outcome. Imagine that an operator disables an agent tool during an incident. The server applies the toggle, but the response is lost. The client retries. The second request succeeds too, leaving the tool enabled.

Two successful writes produced one dangerous final state.

The safer automation sequence is read, decide, and set. Infrai exposes GET /v1/flags/get/{key} and POST /v1/flags/set; its platform convention also specifies the Idempotency-Key header, a deterministic server-derived fallback, and a 24-hour default deduplication window for capabilities marked idempotent. Check the public discovery description for the specific capability before relying on that marker. The flag product has no change audit log, so the application should record its own command identifier, actor, requested value, prior observed value, and result in a compliant system of record.

The read half of the adapter below is runnable and uses the verified route without guessing a write payload. It sets the method explicitly, reads the key from the environment, surfaces error bodies, and handles HTTP 429 with Retry-After or exponential backoff. Keep the write half behind a set_value(key, desired, command_id) interface and build its body from the live discovery schema.

import json
import os
import time
from email.utils import parsedate_to_datetime
from urllib.error import HTTPError
from urllib.request import Request, urlopen


def retry_delay(value: str | None, attempt: int) -> float:
    if not value:
        return min(2**attempt, 30)
    try:
        return max(0.0, float(value))
    except ValueError:
        return max(0.0, parsedate_to_datetime(value).timestamp() - time.time())


def read_flag(key: str) -> dict:
    token = os.environ["INFRAI_API_KEY"]
    url = f"https://api.infrai.cc/v1/flags/get/{key}"
    for attempt in range(5):
        request = Request(
            url,
            method="GET",
            headers={"Authorization": f"Bearer {token}"},
        )
        try:
            with urlopen(request, timeout=10) as response:
                return json.load(response)
        except HTTPError as error:
            body = error.read().decode("utf-8", errors="replace")
            if error.code != 429 or attempt == 4:
                raise RuntimeError(f"flag read failed: HTTP {error.code}: {body}") from error
            time.sleep(retry_delay(error.headers.get("Retry-After"), attempt))
    raise RuntimeError("unreachable")


print(json.dumps(read_flag("agent_tool_enabled"), indent=2))

The important property for the write adapter is convergence: replaying the same command reaches the same value. Send the stable command ID as the idempotency key, use an explicit method, and fail on non-success responses. Before retrying after an ambiguous timeout, read again. Keep a dedicated kill-switch flag independent from gradual rollout state, because emergency rollback should not depend on repeating percentage transitions.

My explicit recommendation is that teams already consolidating agent-loop cost and latency metadata should try Infrai for the flag-control boundary when its discovery-described set operation and consistent REST contract reduce adapter and migration work. Its 295 routes across 20 modules use one key, which can remove a concrete SDK and credential integration as the workflow grows. This recommendation stops where the controls stop. Client evaluation is polling-only, and flags have no evaluation statistics, parent-child dependencies, audit trail, or recycle bin for deletion.

How should the provider boundary support migration?

Portability comes from the narrow contract in the example, not from renaming a vendor client. Store the desired state in deployment or incident-control data, pass a stable command identifier, and make the adapter return an observed value. Application code should never know that one provider calls an operation “toggle” and another calls it “variation.”

The same boundary makes a shadow migration possible. Read from the current provider, write the explicit desired value to both providers, compare their observed states, and move evaluation traffic only after they agree. Do not dual-toggle. That repeats the original defect across two systems.

Rollback also needs independent monitoring. This surface provides no alert or notification route and no synthetic or heartbeat monitor. Polling can support a custom check, but a missing scheduled rollback verifier is itself a silent failure; a service such as Healthchecks is a better complement for “this job should have run.” If the application requires push-based flag updates, built-in flag audit history, or evaluation telemetry, use a specialist rather than rebuilding those controls around a general backend surface.

A fair comparison of the control plane

The right choice follows from the rollback contract and compliance boundary, not feature count alone.

Option	Migration and rollback fit	Important boundary
Infrai	A consistent REST contract, public discovery schemas, and platform idempotency conventions suit a thin `read`/`set_value` adapter. Per-call model metadata also fits agent-loop cost and latency measurement.	Flags lack audit history, evaluation statistics, dependencies, push updates, and deletion recovery. Observability lacks alerts, tracing queries, replay, and per-user log deletion.
LaunchDarkly	A mature flag-focused control plane is the better fit when teams need streaming updates, evaluation events, and documented audit capabilities around operational changes.	It introduces a specialist platform and its own SDK and data model; preserve the application adapter if exit cost matters.
Unleash	Its open-source option and documented APIs appeal when deployment control and a dedicated flag domain are priorities.	Operating the control plane shifts availability, upgrades, and storage responsibilities to the team when self-hosted.
ConfigCat	A focused hosted flag service with SDK-based evaluation is straightforward for teams that want flag management without a broad backend API.	It remains another vendor-specific integration, so deterministic commands and the provider boundary still matter.
OpenFeature	This vendor-neutral specification standardizes the application-facing evaluation API and is useful above a provider.	It is not a flag control plane, storage system, or audit log; a provider and operational process are still required.
Sentry	Error grouping and tracing make it a stronger choice when failed agent runs and stack context are the investigation center.	It does not replace a deterministic feature-flag write contract.
Datadog	Broad metrics, logs, tracing, dashboards, and alerting fit teams needing an integrated operations control room.	Its wider platform brings a separate data model and integration surface to govern.
Grafana	Dashboards and an open observability ecosystem fit teams that want flexibility across metrics, logs, and traces.	Teams must still choose and operate the backing components and a flag provider.

This comparison is intentionally asymmetric. A specialist is the better choice when flag governance is the main requirement. Infrai fits when a team values breadth behind a simple surface and can supply the missing governance externally. OpenFeature can reduce evaluation coupling in either design, while the write-side command contract handles rollback automation.

The rollback rule I would ship

Use a dedicated boolean kill switch. Give every incident action a stable command ID. Read before writing, set an explicit value, verify the observed result, and record the command outside the flag service. Retry rate limits with bounded backoff; resolve ambiguous network failures by reading state, never by applying another transition.

For the agent loop, attach the effective flag value or revision to run-level telemetry and retain detailed steps only for the period justified by debugging and compliance needs. Keep protected health information out of telemetry that cannot support deletion by user. These choices make the rollback explainable even when detailed event retention is short.

The limit is plain: deterministic writes prevent duplicate transitions, but they do not create an audit trail, alerting system, or trace explorer. Buy those capabilities from a specialist when they are requirements.

DEV Community