A retry without an idempotency key is not resilience. It is a second charge, a second ticket, or a second deploy. If an assistant, a worker, or a model-backed endpoint can fire a mutating call twice, you fail closed until the key, the replay window, and the evidence exist.
You do not need a new platform for this. You need a gate that refuses unsigned writes.
Start from the failure, not the model
AI-assisted paths retry more than humans. Timeouts look like failures. Tool calls get re-issued after a reconnect. A coding agent hits apply twice because the first patch looked incomplete.
Your unit tests can stay green while two refunds land. That is not an edge case. That is the default once anything in the loop can auto-retry.
Treat every mutating tool call as a write. Reads can be noisy. Writes need a key.
What “fail-closed” means here
Fail-closed is a refusal rule, not a dashboard color. If the request cannot prove uniqueness, the write does not leave your process.
Copy these criteria as-is:
-
No key, no write. Missing
Idempotency-Keyoridempotency_keyis a 4xx from your gate, not a “best effort” log line. - Key is caller-owned. The model, agent, or retry loop does not invent a new key on each attempt. The key is minted before the first attempt and reused.
- Replay window is explicit. Same key plus same payload inside the window returns the original receipt. Same key plus different payload is a conflict, not a merge.
- Side effects are named. Create, charge, page, deploy, email. If the handler cannot name the mutation, it does not run.
- Evidence is stored before ack. Request hash, key, actor, tool name, and result id land in an append-only receipt. No receipt, no success.
If any row above is “we will add that later,” the path is not production-ready. Later is how duplicates ship.
A gate you can paste into CI
Label the following as a proposal you must adapt. It is a local contract check, not a distributed consensus service.
# idempotency_gate.py
# Proposal: reject mutating tool calls that cannot prove replay safety.
from __future__ import annotations
import hashlib
import json
from dataclasses import dataclass
from typing import Any
MUTATING_TOOLS = {"create_ticket", "refund_charge", "deploy_service", "send_page"}
REQUIRED = ("tool", "idempotency_key", "payload")
@dataclass(frozen=True)
class GateResult:
ok: bool
code: str
detail: str
def canonical_hash(payload: Any) -> str:
blob = json.dumps(payload, sort_keys=True, separators=(",", ":")).encode()
return hashlib.sha256(blob).hexdigest()
def gate_tool_call(call: dict, seen: dict[str, str]) -> GateResult:
for field in REQUIRED:
if not call.get(field):
return GateResult(False, "missing_field", field)
tool = call["tool"]
if tool in MUTATING_TOOLS and call.get("dry_run") is True:
return GateResult(True, "dry_run", tool)
if tool in MUTATING_TOOLS:
key = str(call["idempotency_key"]).strip()
if len(key) < 16:
return GateResult(False, "weak_key", "idempotency_key must be >= 16 chars")
digest = canonical_hash({"tool": tool, "payload": call["payload"]})
prior = seen.get(key)
if prior is None:
seen[key] = digest
return GateResult(True, "accepted", digest)
if prior != digest:
return GateResult(False, "key_payload_conflict", key)
return GateResult(True, "replay", digest)
return GateResult(True, "read_path", tool)
Run it as a test, not as a story.
# test_idempotency_gate.py
from idempotency_gate import gate_tool_call
def test_missing_key_fails_closed():
seen = {}
call = {"tool": "refund_charge", "payload": {"charge_id": "ch_1"}}
result = gate_tool_call(call, seen)
assert result.ok is False
assert result.code == "missing_field"
def test_replay_with_same_payload_is_ok():
seen = {}
call = {
"tool": "refund_charge",
"idempotency_key": "req_7f3c91a0b2dd44e1",
"payload": {"charge_id": "ch_1", "amount": 1400},
}
first = gate_tool_call(call, seen)
second = gate_tool_call(call, seen)
assert first.code == "accepted"
assert second.code == "replay"
def test_same_key_different_body_conflicts():
seen = {}
key = "req_7f3c91a0b2dd44e1"
a = {"tool": "refund_charge", "idempotency_key": key, "payload": {"amount": 1400}}
b = {"tool": "refund_charge", "idempotency_key": key, "payload": {"amount": 2800}}
assert gate_tool_call(a, seen).ok is True
conflict = gate_tool_call(b, seen)
assert conflict.ok is False
assert conflict.code == "key_payload_conflict"
python -m pytest test_idempotency_gate.py -q
Green tests here only prove the gate. They do not prove your payment provider honors the same key. Wire the provider’s idempotency header in a separate contract test, or keep the path closed.
Copy-paste production checklist
Print this. Fill the evidence column. Empty evidence is a no-ship.
| Gate | Evidence you must attach | Fail-closed if missing |
|---|---|---|
| Mutating tools are listed | Checked-in allowlist, not a prompt | Unknown tool name can write |
| Key minted before first attempt | Client/trace code that creates the key once | Retry loop hashes a new UUID each time |
| Key travels on every hop | Header or envelope field in logs | Worker retries with a naked body |
| Payload hash is stable | Canonical JSON (sorted keys, no wall-clock) | Timestamp inside the hash |
| Replay window documented | TTL in hours, plus storage name | “Until restart” or “in memory for now” |
| Conflict is a 409 | Status + metric + example | Silent overwrite |
| Dry-run is default for new tools | Flag in the tool schema | First call in prod is live |
| Timeout ≠ “maybe it wrote” | Distinct codes: timeout, unknown, committed | Timeout auto-retries a write |
| Human abort exists | Kill switch or feature flag | Only the agent can stop itself |
| Receipt is queryable | Request id → key → result id | Success body discarded after 200 |
Do not “approximately” pass a row. Approximately is how double pages reach on-call.
Commands that make the gate visible
You want the refusal in logs, not in a wiki. Start with boring shell checks.
# Proposal: inspect a captured tool-call log for unsigned writes.
# Replace the file with your own capture. Do not paste secrets.
jq -r 'select(.tool_call.mutating==true) | [.ts, .tool, .idempotency_key, .result] | @tsv' \
./captures/toolcalls.jsonl
# Fail CI if any mutating line lacks a key.
if jq -e 'select(.mutating==true and ((.idempotency_key|type)!="string" or (.idempotency_key|length)<16))' \
./captures/toolcalls.jsonl >/dev/null; then
echo "fail-closed: mutating call without a usable idempotency key" >&2
exit 1
fi
If you cannot produce captures/toolcalls.jsonl, you cannot audit retries. Capture is part of the gate, not a nice-to-have.
Where a scratch coding environment fits
You can draft the wrapper, the allowlist, and the tests in a throwaway workspace. That is useful when you do not want the first sketch sitting on a production laptop path.
Disclosure: This article was prepared as part of MonkeyCode's product outreach.
MonkeyCode is an open-source coding assistant with free model access and a free server option. Those two things are enough to generate the gate file and the pytest above, then throw the workspace away. They do not replace the checklist. They do not prove your real payment or ticketing API honors keys. Keep the fail-closed rules on your side of the boundary.
If you try that path, export the files and run pytest on your machine. A generated test that never executed is not evidence.
Decision table for retries
Use this when someone says “just retry it.”
| Observed outcome | Safe next step | Forbidden next step |
|---|---|---|
| HTTP 409 / key conflict | Inspect payload drift; do not retry with a new key | Mint a fresh key to “force it through” |
| HTTP 200 with receipt id | Stop. Store the receipt. | Fire the same body with a new key |
| Timeout, no receipt | Lookup by key. Only retry if lookup is empty and the API documents “timeout means not committed.” | Blind retry because the UI spun |
| 5xx after a write-looking call | Lookup by key first | Exponential backoff on the write itself |
| Agent disconnected mid-tool | Resume with the original key | Let the agent start a sibling call |
| Dry-run succeeded | Flip dry-run off behind the same key only if the API treats dry-run as non-committing | Assume dry-run locked the resource |
Read that last row twice. Dry-run semantics are vendor-specific. If the vendor doc is silent, you fail closed.
How this shows up in agent tool schemas
If a model or coding agent is allowed to call tools, the schema is part of production. A free-form arguments: object is not a contract.
{
"name": "refund_charge",
"mutating": true,
"dry_run_default": true,
"required": ["idempotency_key", "charge_id", "amount_cents"],
"properties": {
"idempotency_key": {"type": "string", "minLength": 16},
"charge_id": {"type": "string", "pattern": "^ch_"},
"amount_cents": {"type": "integer", "minimum": 1}
}
}
Reject extra fields you do not understand. Agents like to add reason, note, and force. force is how keys get bypassed.
Pin the schema in git. Diff it in review. If the agent can widen the schema without a human receipt, you do not have a gate.
Limitations
This checklist does not give you exactly-once delivery across two data centers. It gives you at-most-one intended mutation per key inside one system that honors the key.
It will not save you if:
- the downstream API ignores idempotency headers
- your canonical hash includes clocks, randoms, or unordered maps
- keys are derived from user text the model can rephrase
- retries hop to a second cluster with a cold store
- “success” is a UI checkbox instead of a stored receipt
In-memory seen dicts die with the process. Use them in unit tests. Do not use them as the production store.
Who should not use this approach
Skip this if your path is strictly read-only. Skip it if the binary is a one-shot CLI with no retry and no agent. Skip it if you already have a transactional outbox and the tool layer cannot emit writes except through that outbox.
Do not bolt this onto a system whose source of truth is a spreadsheet. The gate will look formal. The duplicates will still land in the sheet.
Ship rule
If you cannot answer “what happens on the second identical write?” with a receipt id, you do not ship the tool. Fail closed. Then fill the evidence column until a duplicate is boring.
A scratch assistant can help you type the wrapper. Your gate, your store, and your conflict code still have to exist after the chat ends.
Top comments (0)