Two Writes, One Charge: An Idempotency Decision Tree for Agent Steps

Consider a nightly job that walks an invoice queue and calls a payments tool once per invoice. The run takes eleven minutes. In minute nine the worker is restarted — deploy, OOM kill, a dropped connection, the reason does not matter. The retry loop resumes from the last checkpoint and re-executes the step that was already in the air.

Two writes. One charge. One long morning.

None of that is fixed by a better prompt. It is fixed by deciding, per step, what "already done" means — and writing that decision somewhere a retry can read it. The DEV front page this week is full of debate about what agents will replace. The quieter corollary is that agents now touch payments, mailboxes, and shipping labels, where a duplicate is not a bad answer but a real event.

So the question here is narrow. For one given agent step, do you need an idempotency key, a local dedupe ledger, a compensating action, or nothing at all?

The tree first

Q1. Does the step change state outside the run?
    no  ................................ Leaf A: no key, cache if hot
    yes -> Q2

Q2. Does the downstream accept a caller-supplied dedupe key?
    yes ................................ Leaf B: pass the key
    no  -> Q3

Q3. Is the effect reversible, or safe to repeat verbatim?
    yes ................................ Leaf C: local ledger + tombstone
    no  ................................ Leaf D: record intent, compensate

Two rules sit above the tree. First, an unknown or missing effect declaration is treated as the most dangerous class, never the safest. Second, the decision is made from the step's schema and arguments, not from the model's prose about the step.

Glossary, because the words get overloaded

1. Effect boundary. The edge where a step leaves your process: an HTTP call to a payment API, an SMTP send, an object-store put. Inside the boundary you can retry freely; across it you cannot.
2. Idempotency key. A stable string that identifies this logical attempt, sent to the downstream so a repeat is recognized as a repeat.
3. Key scope. Run-scoped keys hash run_id + step_index + tool. Args-scoped keys also hash the canonicalized arguments. Use args-scoped for anything priced, so a corrected amount produces a new key instead of replaying the old charge.
4. In-flight marker. The row claiming a key before the effect happens. Without it, two workers both see "not done" and both proceed.
5. Dedupe window vs retry horizon. The window is how long the downstream remembers a key. The horizon is how long your retry loop may still fire. The window must cover the horizon, or you carry a local tombstone past the downstream's memory.
6. Compensation. A recorded, replayable inverse action for effects that cannot be deduped at all.

Note what is absent: exactly-once. Across a network boundary you get at-least-once delivery plus idempotent handling, which is effectively once only inside a declared window. Writing that window down is most of the work.

Four leaves, worked

Leaf A — read steps. get_invoice(id) changes nothing. Return execute_no_key. A key here only adds a row per call and a failure mode where a stale cached result outlives the data. If the read is expensive, cache on request content with a TTL, which is a different mechanism with different failure behavior.

Leaf B — keyed writes. charge_card against an API that accepts an Idempotency-Key header. Declare x-effect: keyed_write and key_scope: args, claim the key in the ledger, then pass the 32-hex digest downstream. On a duplicate claim with state done, replay the stored result instead of calling again. If you skip the claim and rely on the header alone, a retry that changes the amount silently becomes a second charge, because the header no longer matches.

Leaf C — blind writes that repeat safely. Sending the same receipt email twice is annoying, not catastrophic, and an object-store put to a deterministic key is naturally idempotent. Declare x-effect: blind_write. Claim locally, execute, and keep a tombstone row so a late retry returns replay_stored_result. The tombstone must outlive the retry horizon — this is the leaf where a too-short TTL quietly reopens the hole.

Leaf D — effects you cannot take back. A shipping label, an SMS to a customer, a physical dispatch. There is no key that undoes a truck. Record the intent with a durable row, execute, and on any ambiguity raise a compensation task plus a human-visible flag rather than retrying. Retrying here is the bug.

A reference implementation

Dependency-free, Python 3.9+, stdlib only. It is written for this article and has not been validated against a production store — swap the SQLite file for your own database before you point it at money.

#!/usr/bin/env python3
"""idempotency.py - decision tree + claim ledger for agent steps.

Reference implementation, not production-hardened.
Run: python3 idempotency.py
"""
from __future__ import annotations

import hashlib
import json
import sqlite3
import time

EFFECT_CLASSES = ("read", "keyed_write", "blind_write", "physical")


def canonical(args: dict) -> str:
    return json.dumps(args, sort_keys=True, separators=(",", ":"))


def effect_class(schema: dict) -> str:
    declared = schema.get("x-effect")
    # Unknown or missing declarations fall back to the most dangerous class.
    return declared if declared in EFFECT_CLASSES else "blind_write"


def step_key(run_id: str, index: int, tool: str, args: dict, schema: dict) -> str:
    parts = [run_id, str(index), tool]
    if schema.get("key_scope") == "args":
        parts.append(canonical(args))
    return hashlib.sha256("|".join(parts).encode("utf-8")).hexdigest()[:32]


class Ledger:
    def __init__(self, path: str = ":memory:") -> None:
        self.db = sqlite3.connect(path)
        self.db.execute(
            """CREATE TABLE IF NOT EXISTS ledger (
                   key TEXT PRIMARY KEY,
                   step TEXT NOT NULL,
                   state TEXT NOT NULL,
                   result TEXT,
                   created_at REAL NOT NULL)"""
        )

    def claim(self, key: str, step: str) -> bool:
        try:
            self.db.execute(
                "INSERT INTO ledger(key, step, state, created_at) VALUES (?,?,?,?)",
                (key, step, "in_flight", time.time()),
            )
            self.db.commit()
            return True
        except sqlite3.IntegrityError:
            return False

    def finish(self, key: str, result: dict) -> None:
        self.db.execute(
            "UPDATE ledger SET state=?, result=? WHERE key=?",
            ("done", canonical(result), key),
        )
        self.db.commit()

    def release(self, key: str) -> None:
        """Free a claim after a pre-effect failure so a retry can take it."""
        self.db.execute("DELETE FROM ledger WHERE key=? AND state='in_flight'", (key,))
        self.db.commit()

    def lookup(self, key: str):
        row = self.db.execute(
            "SELECT state, result FROM ledger WHERE key=?", (key,)
        ).fetchone()
        return None if row is None else {"state": row[0], "result": row[1]}


def plan(step: dict, schema: dict, ledger: Ledger) -> tuple:
    kind = effect_class(schema)
    if kind == "read":
        return ("execute_no_key", None)

    key = step_key(step["run_id"], step["index"], step["tool"], step["args"], schema)

    if not ledger.claim(key, step["tool"]):
        seen = ledger.lookup(key)
        if seen and seen["state"] == "done":
            return ("replay_stored_result", seen["result"])
        return ("wait_or_fail_fast", None)

    if kind == "keyed_write":
        return ("execute_passing_key", key)
    if kind == "blind_write":
        return ("execute_behind_local_dedupe", key)
    return ("execute_with_compensation", key)

The caller's contract is three lines: pass key downstream when the action is execute_passing_key, call ledger.finish(key, result) after a successful effect, and call ledger.release(key) when the call failed before the effect happened. If you cannot tell whether the effect happened, do not release — treat it as done and reconcile.

def demo() -> None:
    ledger = Ledger()
    schema = {"x-effect": "keyed_write", "key_scope": "args"}
    step = {"run_id": "r-1", "index": 3, "tool": "charge_card",
            "args": {"amount": 1200, "currency": "usd"}}

    action, key = plan(step, schema, ledger)
    assert action == "execute_passing_key"
    ledger.finish(key, {"id": "ch_1", "status": "captured"})

    action, replay = plan(step, schema, ledger)
    assert action == "replay_stored_result"

    # A corrected amount must not replay the original charge.
    edited = dict(step, args={"amount": 1100, "currency": "usd"})
    action, key2 = plan(edited, schema, ledger)
    assert action == "execute_passing_key" and key2 != key

    # Unknown effect class is treated as the dangerous one, not the safe one.
    assert effect_class({}) == "blind_write"

    # A read never touches the ledger.
    action, _ = plan(dict(step, tool="get_invoice"), {"x-effect": "read"}, ledger)
    assert action == "execute_no_key"

    print("all checks passed")


if __name__ == "__main__":
    demo()

Run python3 idempotency.py. The output is one line, all checks passed. The point of the demo is not coverage — it is that the three decisions a retry can make (execute, replay, wait) are testable without any network, which is what makes this reviewable in a pull request.

The two windows that actually bite

Window one: key scope drift. The same logical charge submitted with {"amount": 1200} and {"amount": 1200.0} hashes differently if you canonicalize carelessly. sort_keys=True plus separators handles ordering and whitespace; it does not handle numeric type coercion or currency casing. Normalize arguments before hashing, or you will accumulate near-duplicate keys that look fine in logs.

Window two: in-flight leases. A worker that crashes after claiming but before finishing leaves in_flight forever, and every retry gets wait_or_fail_fast. Add a lease: store an expiry, and let a claim on an expired row succeed. In the SQL above that is one more column and a conditional UPDATE; leaving it out is the single most common way this pattern deadlocks a queue.

Where a free tier genuinely fits

Two of the three moving parts here are unglamorous and cheap, which is exactly where a free tier earns its place.

Schema classification is a model task. Proposing x-effect for a few dozen tool schemas is a natural model pass: feed the tool description and parameters, ask for one of the four classes plus a one-line reason. Treat the output as a proposal that a human signs off, and keep the unknown-to-blind_write fallback. This is the MonkeyCode use I would actually defend — free model access (operator-supplied; check the current terms yourself) makes iterating on that classification prompt cost nothing, and the artifact it produces is a reviewed table in your repo, not a runtime dependency.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The ledger worker needs no GPU and no model call. At execution time plan() is deterministic string hashing plus one SQL statement. That makes it a plausible candidate for a free server option, with one hard condition: the storage must survive a restart. An in-memory ledger on a recycled instance is worse than no ledger, because it looks like protection. Verify persistence and uptime against your retry horizon before you host anything with a payment path on it.

Who should not use this

If your agent has no external side effects, or your retry horizon is a single process with no restart, this is scaffolding you do not need. If you already run a workflow engine with durable execution, use its semantics instead of hand-rolling a ledger — two dedupe layers with different windows is its own incident. And if a step is genuinely non-reversible and high-value, do not automate it at all; route it to an approval queue and stop pretending retries are the answer.

Pre-flight checklist

1. Every tool schema declares x-effect and key_scope, or inherits the blind_write fallback on purpose.
2. Canonicalization is tested with numbers, casing, and key ordering.
3. Dedupe window ≥ retry horizon + clock skew + longest in-flight duration.
4. In-flight rows expire; expired claims can be retaken.
5. Leaves C and D have a human-visible flag, not just a log line.
6. The ledger store survives a worker restart — tested, not assumed.

If you want to try the schema-classification pass before committing budget to it, a free model tier is a reasonable place to start; just make sure the ledger underneath it is durable before anything reaches a payment call.