Checkout Feature Flag Idempotency: Attributing Duplicate Writes from Retries

TL;DR: Treat a timed-out flag mutation as an unknown outcome, not a failed write. Give every logical change a stable operation ID, reuse it on retries, and record enough evidence to connect the request, the durable mutation, and the checkout cohort affected. For an e-commerce team, that turns duplicate-write debugging into a bounded query and assigns telemetry cost to the rollout that produced it.

The tempting implementation retries POST /flag-mutations after a timeout and counts on the endpoint to sort things out. That is too little information. The first request may have committed before its response disappeared, so a second request can create another mutation, emit another audit event, or advance a rollout twice. An HTTP error describes what the caller observed. It does not prove what the server committed.

My evaluation constraint is practical: a solo team should be able to answer one checkout incident without retaining every payload forever or adopting a vendor-specific data model. The chosen design therefore uses a stable operation identity, explicit outcome states, trace correlation, and a small append-only evidence record. The simple retry loop loses that causal chain. The evidence record keeps it.

How should feature flag retries prevent duplicate writes?

Retries operate on requests, while the business intent exists above the request. A checkout rollout might mean, "move the payment-confirmation variant from 5% to 10% for cohort checkout-us-web." If the client creates a fresh mutation ID on every attempt, the backend sees two distinct commands even though the operator intended one change.

The dangerous sequence is short:

1. The control plane receives the mutation and commits it.
2. The response is lost or arrives after the caller's deadline.
3. The caller retries with a new identity.
4. The backend accepts another command and produces another durable side effect.

That is the trap.

This is an ambiguity problem. A timeout belongs in an unknown state until the caller can reconcile it. Calling it failed encourages another blind write; calling it succeeded invents certainty. That middle state matters most when the rollout controls checkout code that can create orders, reserve inventory, or initiate payment work downstream.

Consider the evidence available after a checkout operator raises a payment-confirmation cohort from 5% to 10%. The browser reports a timeout, the first backend process has already committed version 42, and a retry reaches another process. A log line saying "request failed" cannot distinguish a lost response from a lost write. A fresh operation ID makes the second process accept version 43, while a stable ID lets it return the result for version 42. These numbers illustrate the state transition rather than a benchmark, but they expose the exact question the telemetry must answer: did two transports represent one intent or two?

HTTP method semantics do not rescue a poorly modeled command. RFC 9110 defines idempotent methods in terms of the intended effect of multiple identical requests, but an application still has to preserve the identity and result of its own mutation. POST is not inherently unsafe to retry, and switching to PUT does not magically deduplicate separately identified operations. The contract has to say what counts as the same logical change.

Make the operation identity survive the network

Generate the operation ID once, outside the retry loop. Bind it to the actor, flag, target cohort, and requested state. On the server, atomically store the first accepted result under that ID; a replay returns the stored result rather than applying the mutation again. If the same ID arrives with a different command fingerprint, reject it as a conflict. Silent reuse would hide a caller bug.

Here is the focused client-side shape. It deliberately separates the logical operation from its transport attempts:

type FlagMutation = {
  operationId: string;
  flagKey: string;
  cohort: string;
  enabled: boolean;
  costOwner: string;
};

type MutationResult =
  | { state: "applied" | "replayed"; mutationId: string; version: number }
  | { state: "conflict"; reason: string }
  | { state: "unknown"; operationId: string };

async function applyWithRetry(command: FlagMutation): Promise<MutationResult> {
  for (let attempt = 1; attempt <= 3; attempt += 1) {
    const controller = new AbortController();
    const timeout = setTimeout(() => controller.abort(), 2_000);

    try {
      const response = await fetch("https://control.example/flag-mutations", {
        method: "POST",
        headers: {
          "content-type": "application/json",
          "idempotency-key": command.operationId,
        },
        body: JSON.stringify(command),
        signal: controller.signal,
      });

      if (response.status === 409) {
        return { state: "conflict", reason: await response.text() };
      }
      if (response.ok) return (await response.json()) as MutationResult;
      if (response.status < 500) throw new Error(`non-retryable ${response.status}`);
    } catch (error) {
      if (attempt === 3) break;
    } finally {
      clearTimeout(timeout);
    }
  }

  return { state: "unknown", operationId: command.operationId };
}

Three attempts and a two-second deadline are example policy values, not universal recommendations. Set them from the control plane's latency distribution and the operator workflow. The invariant is the important part: operationId never changes across attempts.

The server needs a transaction boundary around deduplication and mutation. Conceptually, it inserts the operation record only if that key is absent, applies the flag change in the same transaction, and stores the response needed by a replay. A read followed by a write is insufficient because two attempts can pass the read concurrently. Use the database's uniqueness and transaction guarantees instead of an in-process cache; a process restart or a second instance must not erase the contract.

Keep the deduplication retention window longer than the maximum retry and reconciliation window. There is no honest universal duration. A background job that may retry tomorrow needs a different window from an interactive console that gives up after seconds, and that choice directly affects storage cost.

Record evidence for attribution, not every byte

Tracing answers where an attempt spent time. An operation ledger answers whether the logical mutation happened. Audit data answers who requested it. For this failure mode, none of the three replaces the others, but they can share identifiers.

I would retain one narrow event per transport attempt and one per logical result. W3C Trace Context provides traceparent and tracestate for propagating trace identity across services. The operation ID remains an application-level field because one logical mutation can span multiple retry traces, depending on where retry instrumentation starts.

Field	Cardinality source	Why keep it
operation_id	One logical flag change	Joins replays to the accepted result
attempt	Bounded retry count	Reveals retry amplification
mutation_id	One committed mutation	Proves the durable result
flag_key and cohort	Rollout configuration	Narrows checkout impact
cost_owner	Internal ownership taxonomy	Allocates ingest and retention cost
trace_id	Transport execution	Links latency and backend errors
outcome	Small controlled enum	Separates applied, replayed, conflict, and unknown
payload_bytes	Request measurement	Estimates telemetry and transport volume

Do not put customer email, cart contents, payment tokens, or the full request body into these records. They are unnecessary for proving idempotency and make retention harder to justify. A canonical command fingerprint can detect mismatched reuse without retaining sensitive payloads. Define the canonicalization precisely; hashing arbitrary JSON text is brittle because property ordering and serialization can differ.

Cost attribution should follow the causal owner of the experiment, not whichever shared service emitted the span. cost_owner=checkout-growth and flag_key=payment-confirmation-copy let an analytical query assign retry volume to a rollout. Avoid user IDs and operation IDs as metric labels: their cardinality grows with traffic. Keep those identifiers in logs, traces, or an analytical event table, then publish low-cardinality aggregates as metrics.

An analytical store such as ClickHouse can query append-only attempt records and aggregate them by owner, flag, and outcome. That is an example of a storage shape, not a product requirement. A relational table or another column-oriented system can support the same investigation if it preserves the fields and retention window.

Reconcile unknown outcomes before another write

When all transport attempts end without a trustworthy response, query by operation ID. The reconciliation read should return the stored result if the mutation committed, pending if processing is still active, or absent only when the system can establish that no record exists. Even absent needs a policy: an asynchronously replicated read may lag the writer.

This changes the operator path. The UI can display "outcome unknown; checking" instead of turning a timeout into a red failure banner with an inviting Retry button. If the result is found, show the original mutation version. If it remains absent beyond the system's consistency and processing bounds, resubmit the same operation ID. Never mint a replacement merely to escape an awkward state.

Instrumentation should distinguish these outcomes:

• applied: this operation created the durable mutation.
• replayed: the operation was already stored and its result was returned.
• conflict: the key was reused with different normalized input.
• unknown: the caller lacks conclusive evidence.

Keep the vocabulary small. It makes alerts and cost reports comprehensible, and it prevents backend-specific exception names from leaking into the operational contract.

Test the commit-response gap

The useful test is not "mock fetch to return 500." It is "commit the mutation, then drop the response." That placement recreates the ambiguity that produces duplicate writes. Run a second attempt with the same operation ID and assert that the rollout version advances once, one durable mutation exists, and the replay points to the original result.

Also test two simultaneous attempts, process termination after commit, delayed reconciliation reads, and reuse of a key with a changed cohort or desired state. Property-based tests fit well here: generate repeated and reordered attempts for one command, then assert a single committed effect. The test oracle is about state, not how many HTTP calls occurred.

Deploy the evidence path before changing retry behavior. For one checkout flag, observe retries without automatically increasing them. Compare the accepted-operation count with committed mutations, and verify that every replayed result maps to the original mutation_id. Once the invariant holds under injected response loss, enable bounded retries for a small internal cohort.

Rollback deserves the same rule. Disabling a checkout variant is another mutation, not an exception to the model. Give the rollback its own operation ID and desired state so an urgent response does not bypass the very evidence needed afterward.

What should you measure before copying this design?

Start with retry amplification: transport attempts divided by logical operations, sliced by cost_owner and flag. Then measure the share of outcomes that stay unknown past the reconciliation bound, conflict count, deduplication-table growth, analytical bytes ingested per operation, and time from first attempt to conclusive result. Those numbers expose both reliability and observability cost.

There is a real trade-off. Longer deduplication retention improves protection against late replays but consumes storage. More detailed attempt events accelerate diagnosis but increase ingestion and governance work. Extra retries may reduce visible transient failures while multiplying load during a backend slowdown. Set explicit budgets for all three rather than treating retry count as a reliability dial.

The decision rule is compact: adopt this pattern when a repeated flag mutation can create a second durable effect and the caller can lose the first response. Keep the operation ledger authoritative, traces diagnostic, and aggregates cheap. For a checkout rollout, the system should answer four questions without reconstructing a story from timestamps: what was intended, whether it committed, which attempts replayed it, and who owns the resulting telemetry cost.

Further reading

• RFC 9110, HTTP Semantics: https://www.rfc-editor.org/rfc/rfc9110
• W3C Trace Context Recommendation: https://www.w3.org/TR/trace-context/
• OpenTelemetry semantic conventions: https://opentelemetry.io/docs/specs/semconv/
• ClickHouse documentation: https://clickhouse.com/docs