Idempotency keys for AI agents: why hashing the arguments doesn't work

After our last post on why our agents refuse to redo work after a crash, a reader left a comment that took the discussion a level deeper.

The point, roughly: you can't cleanly resume a non-deterministic trajectory, so the only thing that really holds is idempotency at the effect boundary. And then the question that stuck with me: why make the orchestrator decide what's safe to replay, instead of pushing that decision down to the tools themselves?

They're right, and this post is my attempt to answer properly. It covers what we do today, why the obvious version of that idea breaks with LLMs, and the design we're converging on. Fair warning: the last part isn't shipped. It's a proposal, and I'd genuinely like it torn apart.

TL;DR

• Today, ApowerB refuses to auto-replay any run that called any tool, reads included. Safe, blunt, and stuck the moment a run completes halfway.
• The fix is per-tool idempotency keys, so a replay dedupes at the tool. But hashing the arguments doesn't work when an LLM writes them: the replay drafts a slightly different email, and the hash changes.
• The rule that holds: build the key from stable intent, never from generated content.
• We already do this in two places, and enforce "read-only" by construction in a few others. The rest is a design sketch, and an open invitation.

---

Where we are today: default-deny

When a run fails, our runtime records which tools it called. If it called anything, an automatic replay is refused with a 409 that lists the tools (run_main.py). A human can still force it.

We don't classify tools. A GET counts the same as a send_email. That sounds lazy, so let me defend it for a second, because the reader's own examples are the best argument for it:

• a GET against a rate-limited or stateful API is not free to repeat;
• a "pure" lookup that writes an audit row isn't pure;
• a tool from a third-party MCP server can call itself read-only, and you have no way to check.

Inferring purity gets exactly these cases wrong, and they're the ones that hurt. Refusing everything costs a click. Guessing wrong costs a duplicate email to a customer.

Where default-deny breaks

Default-deny is safe. It is also stuck the moment a run finishes halfway.

Picture an email campaign. Our email-campaign skill already puts a human gate in front of the effect: it drafts everything, then never sends without explicit approval. Once approved, it loops over the recipients and sends one email each.

Now the pod dies after email 40 of 100.

• Replay is refused, because tools were called. Correct, and useless: 60 people never get their email.
• Force the replay, and 40 people get it twice.

There is no good answer at the run level. The run is the wrong unit. The email is the unit. That's exactly where the reader's idea comes in: let each effect know whether it already happened.

The obvious version, and why it breaks

The textbook approach: every effectful tool computes an idempotency key from its arguments, say sha256(json(args)), and the runtime skips any call whose key it has already executed.

With a normal program, that works. With an LLM it quietly fails, because the arguments aren't stable either.

First attempt, the model calls:

{"to": "[e-mail]", "subject": "Following up on your request",
 "body": "Hi Anna, following up on the quote you asked for..."}

Replay, same input, same agent:

{"to": "[e-mail]", "subject": "Your quote",
 "body": "Hello Anna, I'm getting back to you about your quote..."}

Different subject, different body, different hash. The ledger has never seen this key. Anna gets two emails, and your idempotency layer reports that everything went perfectly.

The reader's line from the last thread applies one level down: step 3 isn't a stable coordinate. Neither are the arguments.

The rule: stable intent, never generated content

Split every argument into three kinds:

Kind	Examples	In the key?
Identity from the outside world	event id, message id, thread id, record id, normalized recipient address	Yes
Derived deterministically from inputs	hash of an uploaded file's content	Yes
Generated by the model	body, subject, summary, any free text	Never

The key answers "have I already done this thing, for this reason, to this target?", not "have I already sent these exact bytes?". For the campaign: key = (triggering_event, tool, recipient). The replay can rewrite every email from scratch; Anna's key is the same, so she's skipped.

Two places where we already do this

This isn't hypothetical in our codebase. It's just not generalized yet.

1. Webhooks: the key is the provider's identity. When Microsoft Graph re-delivers a notification, a unique index on (subscription_id, resource_id) rejects the duplicate insert, and the message is processed once (outlook.py). Nothing generated goes into that key. It's pure outside-world identity.

2. BI datasets: the key is derived from the content. When an agent imports a file as a dataset, the dataset's ID isn't random. It's a UUIDv5 over who owns it, where it lives, and a SHA-256 of the bytes (bi_datasets.py):

digest = hashlib.sha256(attachment["content"]).hexdigest()
dataset_id = str(uuid.uuid5(
    uuid.NAMESPACE_URL,
    "|".join([owner.lower(), organization_id, project_id,
              attachment["folder"], attachment["name"],
              data.sheet or "", digest]),
))
existing = await store.get(dataset_id)
if existing is not None:
    return {"success": True, "dataset_id": dataset_id, "reused": True}

Run it twice on the same file: one dataset. Change one byte: a new one. Notice what's not in the key: the display name, which is formatted text.

And here's the scar that comes free with it. What if the user deleted that dataset? Soft deletes keep the row, so the deterministic ID is still taken, and the insert collides. Our code catches the IntegrityError and falls back to a fresh random ID. That's the general lesson: an idempotency key outlives the thing it points to. Your ledger needs to know the difference between "already done" and "done, then undone".

And some tools are pure by construction, not by declaration

The other half of the reader's question was classification. Our answer so far: we don't trust a tool that says it's read-only. We trust one that can't write.

• The Oracle tool opens a SET TRANSACTION READ ONLY and rolls back (oracle.py).
• The MongoDB tool refuses aggregation pipelines with a write stage ($out, $merge) (mongodb.py).
• The Redshift tool validates that the SQL is read-only before it runs (redshift.py).

These are the only tools we'd be comfortable marking "safe to replay", and only because the guarantee lives in the tool's code, not in a flag next to it.

The design we're converging on

This part is a sketch, not shipped code. It's the shape we think is right, written down so it can be challenged.

1. Tools declare their effect, and default to the worst case.

@tool(effect="none")            # pure by construction, enforced in the tool
def oracle_run_query(sql: str): ...

@tool(effect="idempotent",
      key=lambda args, ctx: (ctx.trigger_event_id, "send_email",
                             normalize(args["to"])))
def send_email(to: str, subject: str, body: str): ...

@tool                            # no declaration: treated as unsafe
def third_party_mcp_action(**kwargs): ...

2. An effects ledger, with a unique constraint. Before an idempotent tool runs, the runtime looks its key up in a ledger table. Already there: return the stored result, don't execute. Not there: execute, then record. The same unique-index trick as our webhooks, applied to every effect.

3. Undeclared stays default-deny. Anything without a declaration, including every third-party tool, keeps today's behavior: if it ran, the run isn't auto-replayed. Declaring a tool idempotent is how you earn automatic replay, one tool at a time.

4. Pass the key through when the provider supports it. Some APIs accept an idempotency key natively (Stripe's Idempotency-Key header is the classic example). When they do, the key should travel all the way to the provider, so even a crash between the API call and our ledger write dedupes on their side.

With that in place, the campaign that died at email 40 replays cleanly: 40 ledger hits, 60 sends, nobody gets two emails.

The questions I don't have good answers to yet

This is where I'd genuinely like help:

• Which fields go in the key, per tool? For email, (event, recipient) feels right. For "post a summary to Slack", is it (event, channel)? What about an agent that legitimately sends two different emails to the same person for the same event?
• What if the replay takes a different branch? The model may decide to email a different set of people the second time. Per-target keys handle overlap, but what about the people who were emailed in run one and dropped in run two?
• How long does the ledger remember? Forever is wrong (storage, and "the same email next month" is legitimate). Too short reopens the duplicate window.
• The crash between the effect and the ledger write. If the provider has no native idempotency, there's a gap. Is "record intent before, confirm after" plus reconciliation good enough?

Come build it with us

If you've shipped something like this, or you have strong opinions about any of the questions above, we'd love your help. We're opening an RFC issue on tool-level idempotency keys, and every contributor is credited by name.

• Source code: github.com/apowerb/apowerb
• Issues and RFCs: github.com/apowerb/apowerb/issues
• Run it locally (Docker Compose) or on Kubernetes (Helm chart): github.com/apowerb/apowerb-hosting
• Helm chart on Artifact Hub: artifacthub.io/packages/helm/apowerb/apowerb-chart
• Documentation: docs.apowerb.com
• Product page: thaink2.com/apowerb

And thanks to the reader whose comment started all of this. The best part of writing in public is when the comments take the topic somewhere new.

How do you build idempotency keys for your agent's tools? Hash the args, stable fields, something else? Tell me in the comments.

Comments

[Oleksandr]

"The key answers 'have I already done this thing, for this reason, to this target?'" — that's the cleanest definition I've seen, and it's exactly what I was trying to say in the previous thread.

For the "how long does the ledger remember" question: I'd argue the key should have a time bucket built in, not a TTL on the ledger itself. Instead of (event, tool, recipient), use (event, tool, recipient, day). That way "send Anna a follow-up next month" is a new key, not a resurrection of an old one. And the ledger can keep everything forever without the "resurrection" problem.

For the "different branch" question: I'd key on the effect (send email to Anna about event X), not on the decision (the agent decided to email Anna). If run one emailed Anna and run two skipped her, the effect for Anna was still "done", so she's still skipped. The branch changed, but the effect didn't.

Does that match what you're converging on, or am I overcomplicating it?

[Anis Meziani]

Effect over decision, agreed, as long as the target is normalized and keyed per recipient, not per batch call.

A wall-clock bucket breaks on the replay itself: crash at 23:59, replay at 00:01, new key, two emails. The bucket has to come from the trigger (event id or scheduled occurrence), not from when the run executes.

And keeping the ledger forever just moves resurrection: if the user deletes what the effect produced, the key still blocks a legitimate redo. We’d store a state per key (pending, done, revoked) instead of a bare row.

[Oleksandr]

You're right, and I should have caught that. A wall-clock bucket is the exact trap I was warning about one level up — it's stable in the happy path and broken in the crash path, which is precisely when you need it to work.

The pending / done / revoked state is the right model, and it solves the "deleted, then redone" problem cleanly. But it raises a new question: who decides when a done key becomes revoked? Is it automatic (e.g., when the target object is deleted), or does it require a human to mark it?

Asking because "revoked" is a judgment call, and if the system tries to infer it automatically, it'll get it wrong in exactly the cases where it matters.

[Anis Meziani]

Never inferred. A key goes to revoked only on an explicit action: either the deletion itself, when it goes through our own API and marks the key in the same transaction (that’s not inference, it’s the same event), or a human. What we never do is read absence as revocation: a target we can’t find on the provider side is unknown, not revoked, same rule as everywhere else.

Today we sidestep it in one place: when a BI dataset’s deterministic ID collides with a soft-deleted row, the import falls back to a fresh ID instead of resurrecting the old one.

[Oleksandr]

"What we never do is read absence as revocation: a target we can't find on the provider side is unknown, not revoked" is the sentence I'd put in the docs.

It's the exact mistake most idempotency systems make: they see a 404 and assume the thing is gone, when it might just be temporarily unreachable, or behind a different endpoint.

Curious how you handle the "provider returns 500" case. If the API is down, does the key stay pending (and block retries), or does it move to unknown (and allow a human to decide)?

[Adil Harcha]

Great piece! Your distinction between external identity and LLM-generated content for idempotency keys completely hits the nail on the head.

A few thoughts on your open questions:

Multiple emails per recipient/event: Add a logical action tag or step type to the key (e.g., (event_id, recipient, "acknowledgment") vs (event_id, recipient, "summary")) to separate distinct intents.

Branch divergence on replay: If the agent drops a recipient on the second run, they simply aren't emailed again, which is usually fine. For the ones targeted twice, deterministic keys successfully block duplicates. Post-execution reconciliation helps catch any missed critical actions.

Ledger retention: Instead of a blind time-based TTL, tie cleanup to the parent business object's lifecycle (e.g., active campaign or ticket status) combined with a rolling 30-day window.

The gap before ledger write: For APIs lacking native idempotency, an Outbox pattern (PENDING state before the call, confirmed after) paired with an async reconciliation loop is the safest bet.

[Anis Meziani]

Thanks! Fully agree on the outbox and on tying retention to the parent object. One condition on the action tag: it has to come from the tool or workflow definition, never from the model, or it’s generated content again. Let’s fold all four points into the RFC.

[Manoj Kumar Kagitha]

You nailed the hard part: the key answers "have I done this, for this reason, to this target." We run Service Bus duplicate detection on a few flows, and the dedup key always comes from the business identifier, never the payload, for exactly this reason. On the ledger TTL question, scoping retention to the window where replays actually happen feels right. A duplicate from six months ago is usually a different request.

[Anis Meziani]

Thanks for the comment. Yes, Service Bus is a good reference here, since its duplicate detection makes the same call: business ID as the key, and a bounded window rather than forever. One constraint on our side: the window has to cover how long a failed run stays replayable, since a replay can be triggered by a human well after the crash, not just by an automatic retry. How do you handle flows where the business ID alone isn’t enough, like two legitimate messages for the same order?

[Max Kaya]

Two things on the open questions.

On drift between runs: the ledger as described keys on (event, tool, target), which handles overlap but not the drop. I'd freeze the intended target set as a plan row before the first effect — a list of (target, key) written when the plan is approved — and let a replay reconcile against the union of plan and ledger. Then "emailed in run one, dropped in run two" becomes a visible diff (approved but never sent) instead of a silent skip, and the metric you want changes from "did the run finish" to "did every approved recipient get exactly one effect".

On retention: a time TTL feels wrong for this. A provider replay window like Stripe's 24h is a clock, but "the same email next month is legitimate" is a business rule. Since the key already scopes to the event, the event id is the natural expiry — a new event id never collides, so the row can live as long as you keep the business record of the event.

One question: for a tool whose provider has no idempotency support and whose effect is irreversible, do you treat the ledger write as the only barrier, or do you require a provider-side receipt you can look up later before the row counts as done?

[Anis Meziani]

Very good idea. To your question: not the ledger alone. The ledger stops us from duplicating our own effects; it doesn’t prove the effect happened. For an irreversible tool with no provider idempotency, a row would go pending before the call, done only on a provider receipt we can look up later (a message id, a transaction id), and unknown if the process dies in between. Unknown is never retried automatically, it goes to a human. That’s the rule our runs already follow: an unrecorded tool history means “unknown”, never “nothing happened”, and the replay is refused. A provider with no receipt at all never reaches done on its own.

The plan row fits naturally: our email-campaign skill already stops for approval before sending, and the approved list is exactly the set to freeze.

This deserves to be in the RFC. Would you post it there? github.com/apowerb/apowerb/issues

Thanks in advance for your contribution, and for the exchange, it was a really nice one.

[Ayoub Hammoudi]

Stable intent is the right root. One refinement: the key needs the identity of what the intent acts on, not just the intent. "Send the renewal email for invoice 7 to customer 42" replays safely. "Send the renewal email" dedupes two different customers into one. Intent plus the business identifiers it touches, never the generated body.

[Anis Meziani]

Agreed, and that’s why the table puts the record id and the normalized recipient in the key. One catch: those identifiers have to be resolved, not typed by the model. If the model writes “invoice 7” from context and gets it wrong on replay, that’s generated content again and a new key. The tool should look the record up and key on what it actually found.

[Reid Marlow]

A two-phase intent ledger is the only setup that survived our crashes when an external API lacks native idempotency. We insert a pending row with the target coordinates before the socket opens. If the process dies mid-flight, the restart loop checks pending rows against the external service audit log before deciding whether to reissue.

On key composition, adding an explicit action slug to the tuple solves the multi-message problem cleanly: (event_id, target, 'quote') versus (event_id, target, 'receipt'). If the tuple only tracks event and recipient, the agent cannot execute legitimate two-stage notifications for the same trigger.

For branching drift on a restart, treating dropped targets as intentional works better than trying to reconcile against the aborted run. If an agent dies after 40 sends and the second pass decides only 35 of the remaining 60 recipients qualify, the ledger blocks duplicates on the first 40 and lets the new turn handle the rest.

[Anis Meziani]

The pending row before the socket opens is the part most people skip, agreed. One thing that bit people we’ve talked to: audit logs that lag. If the restart checks before the provider has written the send, it sees nothing and reissues. Which service were you checking against, and did you have to wait out a window before trusting a miss?

[ANP2 Network]

The pending-row ground is covered upthread, so here is the case where the provider's key fights yours.

A posting API we run against accepts no caller-supplied idempotency key and dedupes on a hash of the body it receives. An identical re-POST comes back 200 with already_existed=true and a reference to the prior object, including when that prior object is in a permanently failed state. A legitimate retry gets swallowed and reported as success. The only way to obtain a fresh object is to change the generated text, which is precisely the field your table keeps out of identity, so the provider makes model-written content the retry discriminator whether you want it there or not.

The second half bears on your answer about waiting for a provider receipt. That same create call returns 201 with the object not yet live, and the response body carries a one-shot verification challenge that has to be answered by a second call inside five minutes. A later GET on the object returns 404, and no list endpoint exposes the token. Objects were lost exactly this way, by truncating the 201 body before writing it to the log: the tail went, the challenge went with it, and the object sat pending with no path forward. Recovery meant DELETE plus a re-POST with changed content, which walks straight back into the dedup behaviour above.

So there is a third state next to done and unknown: issued, present on the provider, and permanently uncompletable. The requirement that falls out of it is that the row holding the key must also persist the full raw response of the effect call, written before anything parses or trims it. Coordinates are not enough when the response is the only place a completion token ever appears.

Does your effects ledger keep the provider's response body, or the key plus an outcome?

[Ascendra Ventures]

The rule that the key comes from stable intent and never from generated content is the part I would pin above every agent tool. The email campaign that dies at 40 of 100 shows why both obvious answers fail: refuse the replay and 60 people get nothing, force it and 40 get the email twice.

Max Kaya's idea of freezing the intended target set as plan rows before the first effect fits well with what you already have. Once a human approves the campaign, that approval is the stable intent: a fixed list of recipients, each keyed on the event id and the resolved record, with nothing the model wrote. A replay then only has to ask which rows are still pending.

We use the same idea on client builds outside agents too. Write down what done looks like for each target before anything irreversible runs, and recovery becomes a lookup instead of a judgment call. Will the revoked state be visible to the person approving a forced replay?

[Alex Chen]

The tension I keep circling back to is between "same key, same skip" and a deliberately superseded attempt. If the first send failed at the provider and the replay drafts a better email, key (event, tool, recipient) is identical — so the ledger either replays a stored failure or skips a send the agent actively wants to redo. I'd add an intent epoch to the key, bumped only when the agent explicitly supersedes a prior attempt: dedupe then collapses only exact retries, never deliberate revisions. For the crash-between-effect-and-ledger gap, reconciliation reads on startup — a cheap read-only provider query per pending key — have held up better in my experience than any write-time ordering scheme.

[BotSailor]

This was a really interesting read. The point about hashing the arguments falling apart with LLM-generated content makes a lot of sense—what looks like the “same action” to us can easily produce slightly different arguments on a replay.

I especially liked the idea of building idempotency around stable intent and real-world identity rather than the generated text. That feels like a much more reliable foundation for agent systems.

The discussion around pending, done, and revoked was also a great touch. These are the kinds of edge cases that probably don’t show up in demos, but become extremely important once agents are performing real actions. Great practical perspective!