DEV Community

Taylor Zhu
Taylor Zhu

Posted on

No Key, No Write: Fail-Closed Idempotency Gates for AI-Touched APIs

A retry without an idempotency key is not resilience. It is a second charge, a second ticket, or a second deploy. If an assistant, a worker, or a model-backed endpoint can fire a mutating call twice, you fail closed until the key, the replay window, and the evidence exist.

You do not need a new platform for this. You need a gate that refuses unsigned writes.

Start from the failure, not the model

AI-assisted paths retry more than humans. Timeouts look like failures. Tool calls get re-issued after a reconnect. A coding agent hits apply twice because the first patch looked incomplete.

Your unit tests can stay green while two refunds land. That is not an edge case. That is the default once anything in the loop can auto-retry.

Treat every mutating tool call as a write. Reads can be noisy. Writes need a key.

What “fail-closed” means here

Fail-closed is a refusal rule, not a dashboard color. If the request cannot prove uniqueness, the write does not leave your process.

Copy these criteria as-is:

  1. No key, no write. Missing Idempotency-Key or idempotency_key is a 4xx from your gate, not a “best effort” log line.
  2. Key is caller-owned. The model, agent, or retry loop does not invent a new key on each attempt. The key is minted before the first attempt and reused.
  3. Replay window is explicit. Same key plus same payload inside the window returns the original receipt. Same key plus different payload is a conflict, not a merge.
  4. Side effects are named. Create, charge, page, deploy, email. If the handler cannot name the mutation, it does not run.
  5. Evidence is stored before ack. Request hash, key, actor, tool name, and result id land in an append-only receipt. No receipt, no success.

If any row above is “we will add that later,” the path is not production-ready. Later is how duplicates ship.

A gate you can paste into CI

Label the following as a proposal you must adapt. It is a local contract check, not a distributed consensus service.

# idempotency_gate.py
# Proposal: reject mutating tool calls that cannot prove replay safety.
from __future__ import annotations

import hashlib
import json
from dataclasses import dataclass
from typing import Any

MUTATING_TOOLS = {"create_ticket", "refund_charge", "deploy_service", "send_page"}
REQUIRED = ("tool", "idempotency_key", "payload")


@dataclass(frozen=True)
class GateResult:
    ok: bool
    code: str
    detail: str


def canonical_hash(payload: Any) -> str:
    blob = json.dumps(payload, sort_keys=True, separators=(",", ":")).encode()
    return hashlib.sha256(blob).hexdigest()


def gate_tool_call(call: dict, seen: dict[str, str]) -> GateResult:
    for field in REQUIRED:
        if not call.get(field):
            return GateResult(False, "missing_field", field)

    tool = call["tool"]
    if tool in MUTATING_TOOLS and call.get("dry_run") is True:
        return GateResult(True, "dry_run", tool)

    if tool in MUTATING_TOOLS:
        key = str(call["idempotency_key"]).strip()
        if len(key) < 16:
            return GateResult(False, "weak_key", "idempotency_key must be >= 16 chars")

        digest = canonical_hash({"tool": tool, "payload": call["payload"]})
        prior = seen.get(key)
        if prior is None:
            seen[key] = digest
            return GateResult(True, "accepted", digest)
        if prior != digest:
            return GateResult(False, "key_payload_conflict", key)
        return GateResult(True, "replay", digest)

    return GateResult(True, "read_path", tool)
Enter fullscreen mode Exit fullscreen mode

Run it as a test, not as a story.

# test_idempotency_gate.py
from idempotency_gate import gate_tool_call


def test_missing_key_fails_closed():
    seen = {}
    call = {"tool": "refund_charge", "payload": {"charge_id": "ch_1"}}
    result = gate_tool_call(call, seen)
    assert result.ok is False
    assert result.code == "missing_field"


def test_replay_with_same_payload_is_ok():
    seen = {}
    call = {
        "tool": "refund_charge",
        "idempotency_key": "req_7f3c91a0b2dd44e1",
        "payload": {"charge_id": "ch_1", "amount": 1400},
    }
    first = gate_tool_call(call, seen)
    second = gate_tool_call(call, seen)
    assert first.code == "accepted"
    assert second.code == "replay"


def test_same_key_different_body_conflicts():
    seen = {}
    key = "req_7f3c91a0b2dd44e1"
    a = {"tool": "refund_charge", "idempotency_key": key, "payload": {"amount": 1400}}
    b = {"tool": "refund_charge", "idempotency_key": key, "payload": {"amount": 2800}}
    assert gate_tool_call(a, seen).ok is True
    conflict = gate_tool_call(b, seen)
    assert conflict.ok is False
    assert conflict.code == "key_payload_conflict"
Enter fullscreen mode Exit fullscreen mode
python -m pytest test_idempotency_gate.py -q
Enter fullscreen mode Exit fullscreen mode

Green tests here only prove the gate. They do not prove your payment provider honors the same key. Wire the provider’s idempotency header in a separate contract test, or keep the path closed.

Copy-paste production checklist

Print this. Fill the evidence column. Empty evidence is a no-ship.

Gate Evidence you must attach Fail-closed if missing
Mutating tools are listed Checked-in allowlist, not a prompt Unknown tool name can write
Key minted before first attempt Client/trace code that creates the key once Retry loop hashes a new UUID each time
Key travels on every hop Header or envelope field in logs Worker retries with a naked body
Payload hash is stable Canonical JSON (sorted keys, no wall-clock) Timestamp inside the hash
Replay window documented TTL in hours, plus storage name “Until restart” or “in memory for now”
Conflict is a 409 Status + metric + example Silent overwrite
Dry-run is default for new tools Flag in the tool schema First call in prod is live
Timeout ≠ “maybe it wrote” Distinct codes: timeout, unknown, committed Timeout auto-retries a write
Human abort exists Kill switch or feature flag Only the agent can stop itself
Receipt is queryable Request id → key → result id Success body discarded after 200

Do not “approximately” pass a row. Approximately is how double pages reach on-call.

Commands that make the gate visible

You want the refusal in logs, not in a wiki. Start with boring shell checks.

# Proposal: inspect a captured tool-call log for unsigned writes.
# Replace the file with your own capture. Do not paste secrets.
jq -r 'select(.tool_call.mutating==true) | [.ts, .tool, .idempotency_key, .result] | @tsv' \
  ./captures/toolcalls.jsonl
Enter fullscreen mode Exit fullscreen mode
# Fail CI if any mutating line lacks a key.
if jq -e 'select(.mutating==true and ((.idempotency_key|type)!="string" or (.idempotency_key|length)<16))' \
  ./captures/toolcalls.jsonl >/dev/null; then
  echo "fail-closed: mutating call without a usable idempotency key" >&2
  exit 1
fi
Enter fullscreen mode Exit fullscreen mode

If you cannot produce captures/toolcalls.jsonl, you cannot audit retries. Capture is part of the gate, not a nice-to-have.

Where a scratch coding environment fits

You can draft the wrapper, the allowlist, and the tests in a throwaway workspace. That is useful when you do not want the first sketch sitting on a production laptop path.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

MonkeyCode is an open-source coding assistant with free model access and a free server option. Those two things are enough to generate the gate file and the pytest above, then throw the workspace away. They do not replace the checklist. They do not prove your real payment or ticketing API honors keys. Keep the fail-closed rules on your side of the boundary.

If you try that path, export the files and run pytest on your machine. A generated test that never executed is not evidence.

Decision table for retries

Use this when someone says “just retry it.”

Observed outcome Safe next step Forbidden next step
HTTP 409 / key conflict Inspect payload drift; do not retry with a new key Mint a fresh key to “force it through”
HTTP 200 with receipt id Stop. Store the receipt. Fire the same body with a new key
Timeout, no receipt Lookup by key. Only retry if lookup is empty and the API documents “timeout means not committed.” Blind retry because the UI spun
5xx after a write-looking call Lookup by key first Exponential backoff on the write itself
Agent disconnected mid-tool Resume with the original key Let the agent start a sibling call
Dry-run succeeded Flip dry-run off behind the same key only if the API treats dry-run as non-committing Assume dry-run locked the resource

Read that last row twice. Dry-run semantics are vendor-specific. If the vendor doc is silent, you fail closed.

How this shows up in agent tool schemas

If a model or coding agent is allowed to call tools, the schema is part of production. A free-form arguments: object is not a contract.

{
  "name": "refund_charge",
  "mutating": true,
  "dry_run_default": true,
  "required": ["idempotency_key", "charge_id", "amount_cents"],
  "properties": {
    "idempotency_key": {"type": "string", "minLength": 16},
    "charge_id": {"type": "string", "pattern": "^ch_"},
    "amount_cents": {"type": "integer", "minimum": 1}
  }
}
Enter fullscreen mode Exit fullscreen mode

Reject extra fields you do not understand. Agents like to add reason, note, and force. force is how keys get bypassed.

Pin the schema in git. Diff it in review. If the agent can widen the schema without a human receipt, you do not have a gate.

Limitations

This checklist does not give you exactly-once delivery across two data centers. It gives you at-most-one intended mutation per key inside one system that honors the key.

It will not save you if:

  • the downstream API ignores idempotency headers
  • your canonical hash includes clocks, randoms, or unordered maps
  • keys are derived from user text the model can rephrase
  • retries hop to a second cluster with a cold store
  • “success” is a UI checkbox instead of a stored receipt

In-memory seen dicts die with the process. Use them in unit tests. Do not use them as the production store.

Who should not use this approach

Skip this if your path is strictly read-only. Skip it if the binary is a one-shot CLI with no retry and no agent. Skip it if you already have a transactional outbox and the tool layer cannot emit writes except through that outbox.

Do not bolt this onto a system whose source of truth is a spreadsheet. The gate will look formal. The duplicates will still land in the sheet.

Ship rule

If you cannot answer “what happens on the second identical write?” with a receipt id, you do not ship the tool. Fail closed. Then fill the evidence column until a duplicate is boring.

A scratch assistant can help you type the wrapper. Your gate, your store, and your conflict code still have to exist after the chat ends.

Top comments (0)