Use a durable idempotency receipt when feature flag retries can reach a rollout toggle endpoint, otherwise reach for a read-only flag evaluation that cannot create duplicate writes. Short answer: the backend must bind one caller-generated key to one operation and commit the receipt beside the state change; a retry should recover that recorded result, not perform the write again.
The flag is not the transaction.
Record the invariant at the write boundary
My architecture decision is to enforce idempotency inside the backend that owns the mutable state. The caller creates an operation key before its first attempt, sends the same key and operation on every retry, and never manufactures a fresh key inside the retry loop. The backend binds that key to a stable digest of the requested change. If the key and digest have already been committed, it returns the stored result. If the key exists with a different digest, it rejects the integration error as a conflict. The state mutation and receipt belong in one transaction, because two separate commits create an interval in which the state says “done” while the receipt still says nothing.
I write the invariant this way: one idempotency key identifies one logical operation within a documented scope; one committed operation has one durable result. The defensible claim is effectively-once mutation within that scope, not exactly-once delivery. Clients, queues, proxies, and deployment controllers can all repeat an attempt, so delivery count isn't a useful correctness boundary.
There are three failure boundaries I test. A response can disappear after commit, two workers can race on the same key, and the flag decision can change between attempts. The first requires replaying the stored result. The second requires a uniqueness constraint rather than a check-then-insert sequence. The third requires persisting the evaluated decision with the operation; reevaluating a flag during recovery can turn one logical request into two different historical meanings.
Retention is a real trade-off. Keep receipts longer than the longest credible retry window, but don't pretend indefinite retention is free: keys, payload digests, and serialized responses consume storage and may carry data-governance obligations. Your mileage may vary, so I document the scope and expiry behavior as part of the endpoint contract rather than leaving them as database folklore.
How should feature flag retries avoid duplicate writes and backend integration errors?
Separate policy evaluation from mutation. Evaluate the feature flag once for the rollout subject, freeze that decision into the requested operation, create the key, and then call the toggle endpoint. A transport timeout makes the outcome unknown; it does not prove failure. The next attempt must carry identical operation data so the backend can answer from its receipt if the first attempt committed.
This is where observability has to describe logical work, not just traffic. I record request_id, idempotency_key, rollout_id, decision, attempt, and outcome as structured fields. I count attempts, unique committed keys, replays, payload conflicts, and policy rejections separately. Raw request volume can rise because recovery is doing its job, while committed operations remain flat. A dashboard that merges those two counts hides the exact condition I need to see.
I've had one cost surprise from missing that distinction: an analytics bill arrived at 3.2 times my estimate because retrying workers emitted a new event on every attempt, and the analytical store retained the duplicate rows even though the transactional state looked right. The fix was architectural — emit the business event from the committed operation record, include its operation key, and compare ingested rows with unique keys. That ratio made amplification visible before the next bill did.
Count committed keys.
For traces and logs, preserve the high-cardinality operation key where an investigation can join attempts, but keep dashboard dimensions bounded. A span outcome such as committed, replayed, conflict, or rejected_by_policy is more useful than treating every retry as an error. I alert when replay rate departs from its baseline or conflicts appear, because a replay can be expected recovery while a conflict says the caller has violated the key-to-operation binding. I'm not sure why raw request count remains the default rollout graph; as far as I can tell, it rewards noisy retry policies and says nothing about duplicate effects.
Compare the places that could own the guard
The decision is less about a feature flag product than about which component can preserve the invariant through a crash. I use this table in architecture reviews because it forces each layer to state what it can prove rather than what its marketing diagram appears to imply.
| Location | What it can legitimately own | Failure boundary or cost |
|---|---|---|
| Flag evaluator | Policy decision for a subject and context | A later evaluation may differ; it cannot prove that an earlier write committed |
| API process memory | Fast suppression of simultaneous attempts | Restart, rescheduling, or a second process loses the local record |
| Distributed cache | Short-lived coordination and replay data | Expiry and eviction must fit the retry window; state and receipt usually do not share a transaction |
| Transactional database | Unique receipt plus state mutation in one commit | Adds schema, retention, and contention work to the write path |
| Message broker consumer | Deduplication before a downstream effect | Redelivery still requires a durable consumer-side receipt |
| Analytical store | Retry, replay, latency, and duplicate-event investigation | It is evidence after the write, not the synchronous correctness gate |
The transactional owner is my default because consistency and durability belong beside the object being changed, but it isn't universally suitable. A very high-contention key space may need partitioning, receipt pruning needs an explicit policy, and a workflow spanning several independent databases cannot be made atomic by adding one receipt table. In that last case I model a state machine with compensating actions and idempotent steps; I don't disguise a distributed workflow as a local transaction.
ClickHouse can serve the analytical role in this design: immutable attempt and outcome events can be queried there after they leave the critical path. I won't make a rollout mutation wait on an analytical query — observability may be delayed without making state ambiguous — and I verify the chosen deployment's storage behavior against its documentation and failure tests. Cost also belongs in the review: duplicate event emission multiplies ingestion and retention even when the business mutation is correctly deduplicated.
Put the critical path in one Python transaction
This compact example uses Python's standard sqlite3 module to make the boundary visible. Production engines expose different locking and isolation behavior, so I test the real engine under concurrency, but the contract stays the same: a matching key and payload replays; a reused key with changed input conflicts; the receipt and rollout state commit together.
import hashlib
import json
import sqlite3
from dataclasses import asdict, dataclass
@dataclass(frozen=True)
class ToggleResult:
rollout_id: str
enabled: bool
decision: str
def operation_digest(rollout_id: str, enabled: bool, decision: str) -> str:
payload = json.dumps(
{"decision": decision, "enabled": enabled, "rollout_id": rollout_id},
sort_keys=True,
separators=(",", ":"),
)
return hashlib.sha256(payload.encode("utf-8")).hexdigest()
def apply_toggle(
connection: sqlite3.Connection,
rollout_id: str,
enabled: bool,
decision: str,
idempotency_key: str,
) -> tuple[ToggleResult, bool]:
digest = operation_digest(rollout_id, enabled, decision)
with connection:
receipt = connection.execute(
"SELECT request_digest, response_json "
"FROM idempotency_receipts WHERE idempotency_key = ?",
(idempotency_key,),
).fetchone()
if receipt is not None:
if receipt[0] != digest:
raise ValueError(
"409 Conflict: idempotency key reused with different input"
)
return ToggleResult(**json.loads(receipt[1])), True
connection.execute(
"""
INSERT INTO rollout_state (rollout_id, enabled, decision)
VALUES (?, ?, ?)
ON CONFLICT(rollout_id) DO UPDATE SET
enabled = excluded.enabled,
decision = excluded.decision
""",
(rollout_id, enabled, decision),
)
result = ToggleResult(rollout_id, enabled, decision)
connection.execute(
"""
INSERT INTO idempotency_receipts
(idempotency_key, request_digest, response_json)
VALUES (?, ?, ?)
""",
(idempotency_key, digest, json.dumps(asdict(result), sort_keys=True)),
)
return result, False
Both tables need uniqueness at their identity boundary: idempotency_key for the receipt and rollout_id for the state. The boolean return value distinguishes a replay from a fresh commit without changing the business response.
My deployment test runs the same operation through concurrent workers, injects response loss immediately after commit, and then asserts one receipt, one final state, stable response data, and a nonzero replay count. I reuse the key with a changed enabled value and require 409 Conflict. Then I change the live flag between attempts and verify that recovery still returns the persisted decision. These tests cover the awkward timing windows; a happy-path unit test does not.
Reject direct toggles, while keeping their narrow valid use case
I reject “add an enabled column and let every service write it” for a distributed rollout. It has no durable operation receipt, couples rollout policy to a shared schema, and lets each caller invent retry handling. Polling can also create a period in which readers disagree. None of those observations reveals whether an ambiguous response produced a duplicate side effect.
Still, a database-backed toggle is valid for a narrow internal system with one writer, one transactional database, low change frequency, and no percentage targeting. Keep its mutation idempotent anyway. Small systems receive duplicate requests too.
I also reject putting an analytical store on the synchronous mutation path. Its useful job is to answer questions such as “did retries spike after rollout?” and “how many events share an operation key?” Export committed events after the transaction, monitor export lag, and reconcile receipt counts with analytical event counts. The catch is delayed visibility: this design is not suitable when a downstream action must be confirmed before the caller proceeds. For that case, stick with a transactional outbox or a stateful workflow whose completion semantics are explicit, accepting the extra operational machinery.
The final review question is blunt: after the client loses a response, which durable record tells the next attempt what happened? If the answer is a flag value, a process-local set, or a dashboard, the design still has an ambiguity. If the answer is a receipt committed with the write, the retry path has something concrete to recover.
Top comments (0)