After our last post on why our agents refuse to redo work after a crash, a reader left a comment that took the discussion a level deeper.
The poin...
For further actions, you may consider blocking this person and/or reporting abuse
"The key answers 'have I already done this thing, for this reason, to this target?'" — that's the cleanest definition I've seen, and it's exactly what I was trying to say in the previous thread.
For the "how long does the ledger remember" question: I'd argue the key should have a time bucket built in, not a TTL on the ledger itself. Instead of (event, tool, recipient), use (event, tool, recipient, day). That way "send Anna a follow-up next month" is a new key, not a resurrection of an old one. And the ledger can keep everything forever without the "resurrection" problem.
For the "different branch" question: I'd key on the effect (send email to Anna about event X), not on the decision (the agent decided to email Anna). If run one emailed Anna and run two skipped her, the effect for Anna was still "done", so she's still skipped. The branch changed, but the effect didn't.
Does that match what you're converging on, or am I overcomplicating it?
Effect over decision, agreed, as long as the target is normalized and keyed per recipient, not per batch call.
A wall-clock bucket breaks on the replay itself: crash at 23:59, replay at 00:01, new key, two emails. The bucket has to come from the trigger (event id or scheduled occurrence), not from when the run executes.
And keeping the ledger forever just moves resurrection: if the user deletes what the effect produced, the key still blocks a legitimate redo. We’d store a state per key (pending, done, revoked) instead of a bare row.
You're right, and I should have caught that. A wall-clock bucket is the exact trap I was warning about one level up — it's stable in the happy path and broken in the crash path, which is precisely when you need it to work.
The
pending / done / revokedstate is the right model, and it solves the "deleted, then redone" problem cleanly. But it raises a new question: who decides when adonekey becomesrevoked? Is it automatic (e.g., when the target object is deleted), or does it require a human to mark it?Asking because "revoked" is a judgment call, and if the system tries to infer it automatically, it'll get it wrong in exactly the cases where it matters.
Never inferred. A key goes to revoked only on an explicit action: either the deletion itself, when it goes through our own API and marks the key in the same transaction (that’s not inference, it’s the same event), or a human. What we never do is read absence as revocation: a target we can’t find on the provider side is unknown, not revoked, same rule as everywhere else.
Today we sidestep it in one place: when a BI dataset’s deterministic ID collides with a soft-deleted row, the import falls back to a fresh ID instead of resurrecting the old one.
"What we never do is read absence as revocation: a target we can't find on the provider side is unknown, not revoked" is the sentence I'd put in the docs.
It's the exact mistake most idempotency systems make: they see a 404 and assume the thing is gone, when it might just be temporarily unreachable, or behind a different endpoint.
Curious how you handle the "provider returns 500" case. If the API is down, does the key stay
pending(and block retries), or does it move tounknown(and allow a human to decide)?Great piece! Your distinction between external identity and LLM-generated content for idempotency keys completely hits the nail on the head.
A few thoughts on your open questions:
Multiple emails per recipient/event: Add a logical action tag or step type to the key (e.g., (event_id, recipient, "acknowledgment") vs (event_id, recipient, "summary")) to separate distinct intents.
Branch divergence on replay: If the agent drops a recipient on the second run, they simply aren't emailed again, which is usually fine. For the ones targeted twice, deterministic keys successfully block duplicates. Post-execution reconciliation helps catch any missed critical actions.
Ledger retention: Instead of a blind time-based TTL, tie cleanup to the parent business object's lifecycle (e.g., active campaign or ticket status) combined with a rolling 30-day window.
The gap before ledger write: For APIs lacking native idempotency, an Outbox pattern (PENDING state before the call, confirmed after) paired with an async reconciliation loop is the safest bet.
Thanks! Fully agree on the outbox and on tying retention to the parent object. One condition on the action tag: it has to come from the tool or workflow definition, never from the model, or it’s generated content again. Let’s fold all four points into the RFC.
You nailed the hard part: the key answers "have I done this, for this reason, to this target." We run Service Bus duplicate detection on a few flows, and the dedup key always comes from the business identifier, never the payload, for exactly this reason. On the ledger TTL question, scoping retention to the window where replays actually happen feels right. A duplicate from six months ago is usually a different request.
Thanks for the comment. Yes, Service Bus is a good reference here, since its duplicate detection makes the same call: business ID as the key, and a bounded window rather than forever. One constraint on our side: the window has to cover how long a failed run stays replayable, since a replay can be triggered by a human well after the crash, not just by an automatic retry. How do you handle flows where the business ID alone isn’t enough, like two legitimate messages for the same order?
The pending row upthread covers the crash case. The same insert also covers one nobody has raised: parallelism inside a single turn. The Messages API can return several tool_use blocks in one assistant response, and the docs leave execution order to the caller, with a note that tools with side effects or shared state may be better run sequentially. Run them with Promise.all and two calls carrying the same (event, tool, recipient) key can both read an empty ledger before either writes. A select, then send, leaves that window open for the length of the request. The fix is the one your webhook section already uses: a unique index that rejects the duplicate insert, moved off (subscription_id, resource_id) and onto the effect key, with the pending row written before the call. The loser of the insert skips instead of asking. So: does that recipient loop live inside one tool call, or does the model get one tool_use block per recipient? Inside one call the loop is yours to serialise. One block per recipient hands the ordering to the runtime, and a batch call instead hides the race behind a key covering many targets, where a partial failure gets hard to attribute.
One tool_use block per recipient. The skill tells the model to send one call per recipient and not to parallelize, but that’s an instruction in the prompt, not a guarantee. If the model emits several send calls in one response, the ordering stops being ours, which is exactly your race.
So the guarantee has to live in the database, not the prompt: the pending row inserted against a unique index on the effect key, before the call, and the loser of the insert skips. We’d keep one block per recipient on purpose: each key covers exactly one target, so a partial failure stays attributable.
You clearly know this problem better than most. Want to open an issue for it on the repo, and take the PR? The ledger table with the unique index on the effect key and the pending insert before the call would be a great first contribution. The tech team will review it, and you’ll be credited by name: github.com/apowerb/apowerb/issues
Two things on the open questions.
On drift between runs: the ledger as described keys on (event, tool, target), which handles overlap but not the drop. I'd freeze the intended target set as a plan row before the first effect — a list of (target, key) written when the plan is approved — and let a replay reconcile against the union of plan and ledger. Then "emailed in run one, dropped in run two" becomes a visible diff (approved but never sent) instead of a silent skip, and the metric you want changes from "did the run finish" to "did every approved recipient get exactly one effect".
On retention: a time TTL feels wrong for this. A provider replay window like Stripe's 24h is a clock, but "the same email next month is legitimate" is a business rule. Since the key already scopes to the event, the event id is the natural expiry — a new event id never collides, so the row can live as long as you keep the business record of the event.
One question: for a tool whose provider has no idempotency support and whose effect is irreversible, do you treat the ledger write as the only barrier, or do you require a provider-side receipt you can look up later before the row counts as done?
Very good idea. To your question: not the ledger alone. The ledger stops us from duplicating our own effects; it doesn’t prove the effect happened. For an irreversible tool with no provider idempotency, a row would go pending before the call, done only on a provider receipt we can look up later (a message id, a transaction id), and unknown if the process dies in between. Unknown is never retried automatically, it goes to a human. That’s the rule our runs already follow: an unrecorded tool history means “unknown”, never “nothing happened”, and the replay is refused. A provider with no receipt at all never reaches done on its own.
The plan row fits naturally: our email-campaign skill already stops for approval before sending, and the approved list is exactly the set to freeze.
This deserves to be in the RFC. Would you post it there? github.com/apowerb/apowerb/issues
Thanks in advance for your contribution, and for the exchange, it was a really nice one.
Stable intent is the right root. One refinement: the key needs the identity of what the intent acts on, not just the intent. "Send the renewal email for invoice 7 to customer 42" replays safely. "Send the renewal email" dedupes two different customers into one. Intent plus the business identifiers it touches, never the generated body.
Agreed, and that’s why the table puts the record id and the normalized recipient in the key. One catch: those identifiers have to be resolved, not typed by the model. If the model writes “invoice 7” from context and gets it wrong on replay, that’s generated content again and a new key. The tool should look the record up and key on what it actually found.
A two-phase intent ledger is the only setup that survived our crashes when an external API lacks native idempotency. We insert a pending row with the target coordinates before the socket opens. If the process dies mid-flight, the restart loop checks pending rows against the external service audit log before deciding whether to reissue.
On key composition, adding an explicit action slug to the tuple solves the multi-message problem cleanly: (event_id, target, 'quote') versus (event_id, target, 'receipt'). If the tuple only tracks event and recipient, the agent cannot execute legitimate two-stage notifications for the same trigger.
For branching drift on a restart, treating dropped targets as intentional works better than trying to reconcile against the aborted run. If an agent dies after 40 sends and the second pass decides only 35 of the remaining 60 recipients qualify, the ledger blocks duplicates on the first 40 and lets the new turn handle the rest.
The pending row before the socket opens is the part most people skip, agreed. One thing that bit people we’ve talked to: audit logs that lag. If the restart checks before the provider has written the send, it sees nothing and reissues. Which service were you checking against, and did you have to wait out a window before trusting a miss?
Anna's two emails are the cleanest demo of this I've seen, since a reworded subject on replay is all it takes to break a sha256 of the args. The part I'd poke at is the soft-delete fallback in bi_datasets. Once a deleted row holds the deterministic ID, every replay of that import gets a fresh random ID, so two replays still make two datasets. Does the ledger's done-then-undone state end up replacing that fallback?
Good catch, and you’re right. Once a soft-deleted row holds the deterministic ID, the fallback draws a uuid4, so two replays of that import make two datasets. Idempotency holds until the first deletion, not after.
And yes, that’s the revoked state’s job: the deletion is the explicit action that bumps the epoch, so every replay after it lands on the same next ID. This one doesn’t even need the ledger: soft delete already stamps the row, so the fallback can be uuid5(original ID + the deleted row’s deletion time) instead of uuid4, walking the chain if that one was deleted too. Deterministic per deletion, and it still never resurrects the old row.
Small, self-contained fix with a clear test: delete, replay twice, one dataset. Want to take it as a PR?
The tension I keep circling back to is between "same key, same skip" and a deliberately superseded attempt. If the first send failed at the provider and the replay drafts a better email, key (event, tool, recipient) is identical — so the ledger either replays a stored failure or skips a send the agent actively wants to redo. I'd add an intent epoch to the key, bumped only when the agent explicitly supersedes a prior attempt: dedupe then collapses only exact retries, never deliberate revisions. For the crash-between-effect-and-ledger gap, reconciliation reads on startup — a cheap read-only provider query per pending key — have held up better in my experience than any write-time ordering scheme.
The epoch is the right tool, with two conditions. First, a known failure shouldn’t hold the key at all: dedupe only on done, while pending and unknown block, so a send the provider rejected outright leaves the key free and the retry goes through without an epoch. The epoch is for the case where the first attempt really landed and you want a second, better one. Second, the bump can’t come from the model, or the model controls dedupe again and generated content is back in the key. A human or an explicit rule supersedes, the model only proposes.
Agreed on reconciliation reads at startup, with one caveat from the audit-log case upthread: a miss right after the crash isn’t proof. Some providers record the effect late, so a pending key needs a settle window before a miss counts as “not sent”.
Both conditions sharpen the design, and the first one actually dissolves the case I was worried about: if a provider-rejected send never occupies the key, the epoch only ever fires for landed-then-superseded attempts — which is exactly the one case that needed disambiguation. The second condition is the deeper one. Letting the model bump its own epoch would hand dedupe back to generated content through the side door, the exact failure mode the whole design exists to prevent. So the epoch stops being a data field and becomes a capability boundary: bumps come from outside the model, the same place high-stakes approvals come from. The open question is what counts as 'outside' — a human click is cleanest, but a deterministic orchestrator policy (an explicit replace-intent registered before execution) could hold the line without a human in the loop. Either way: if the epoch can only be bumped by something that can't be prompt-injected, the key stays model-proof.
Agreed, Alex, and “capability boundary” is the right name for it. My test for “outside”: could any token the model produced change whether the bump happens? If yes, it’s inside. A deterministic policy passes as long as it’s registered before execution and its inputs come from the workflow definition or the trigger, never from tool arguments. A human click is just the simplest case of that rule.
You’ve thought this through further than most. Want to open an issue for the epoch on the repo? It would fit right next to the ledger work, and anything you write there gets credited by name: github.com/apowerb/apowerb/issues
Opened it — thank you for the invitation: github.com/apowerb/apowerb/issues/473
It sits next to #466 as a companion: epoch semantics keyed to the ledger states (failed frees the key, only done can be superseded), the capability-boundary rule with your token-influence test as the conformance check, and the settle window carried over per epoch. Open questions at the bottom — column vs key string is the one I'd most like the team's call on.
Thank you, Alex, this is great. Pairing it with #466 is exactly right, and turning the token-influence test into a conformance check makes the rule testable instead of a convention.
The team will weigh in on column vs key string on the issue. My lean is a column, with the unique index on (key, epoch): the base key stays stable, so you can look up every attempt for the same intent, and “only done can be superseded” becomes something the database can enforce. Welcome aboard ;) !
The pending-row ground is covered upthread, so here is the case where the provider's key fights yours.
A posting API we run against accepts no caller-supplied idempotency key and dedupes on a hash of the body it receives. An identical re-POST comes back 200 with
already_existed=trueand a reference to the prior object, including when that prior object is in a permanently failed state. A legitimate retry gets swallowed and reported as success. The only way to obtain a fresh object is to change the generated text, which is precisely the field your table keeps out of identity, so the provider makes model-written content the retry discriminator whether you want it there or not.The second half bears on your answer about waiting for a provider receipt. That same create call returns 201 with the object not yet live, and the response body carries a one-shot verification challenge that has to be answered by a second call inside five minutes. A later GET on the object returns 404, and no list endpoint exposes the token. Objects were lost exactly this way, by truncating the 201 body before writing it to the log: the tail went, the challenge went with it, and the object sat pending with no path forward. Recovery meant DELETE plus a re-POST with changed content, which walks straight back into the dedup behaviour above.
So there is a third state next to done and unknown: issued, present on the provider, and permanently uncompletable. The requirement that falls out of it is that the row holding the key must also persist the full raw response of the effect call, written before anything parses or trims it. Coordinates are not enough when the response is the only place a completion token ever appears.
Does your effects ledger keep the provider's response body, or the key plus an outcome?
This is the strongest failure case on the post, thank you.
To your question, honestly: the ledger isn’t built yet, so neither. Today we persist the tool names on the run, plus every session event in Postgres with each tool’s response in full, kept 90 days. But that’s what our tool returned, not what the provider sent. A tool that parses the 201 before returning loses your challenge exactly the same way. So yes: the ledger row should hold the raw response, written inside the tool before anything parses or trims it.
On the body-hash dedup, I’d treat already_existed=true as a pointer, not a success: follow the reference and read the prior object’s state. If it’s permanently failed, the row goes to your third state, and a retry becomes a deliberate new revision in the key, recorded in the ledger, with a deterministic change to the body rather than the model rewriting it.
Would you add this to the RFC? It’s exactly the kind of case it needs: github.com/apowerb/apowerb/issues
The rule that the key comes from stable intent and never from generated content is the part I would pin above every agent tool. The email campaign that dies at 40 of 100 shows why both obvious answers fail: refuse the replay and 60 people get nothing, force it and 40 get the email twice.
Max Kaya's idea of freezing the intended target set as plan rows before the first effect fits well with what you already have. Once a human approves the campaign, that approval is the stable intent: a fixed list of recipients, each keyed on the event id and the resolved record, with nothing the model wrote. A replay then only has to ask which rows are still pending.
We use the same idea on client builds outside agents too. Write down what done looks like for each target before anything irreversible runs, and recovery becomes a lookup instead of a judgment call. Will the revoked state be visible to the person approving a forced replay?
Yes, that’s the whole point of the state. Today a forced replay only shows what the 409 lists: the tools the run already called. With per-target rows, the person forcing it should see each approved recipient as pending, done, revoked or unknown, with who revoked it and when. Only the pending ones go out. Revoked and unknown stay visible, and nothing happens to them unless someone picks them explicitly.
“Recovery becomes a lookup instead of a judgment call” is a great way to put it. And approval is the right moment to freeze the list, as Max suggested.
This is a good example of why building production-ready AI agents involves much more than getting the model to call the right tool. At IT Path Solutions, this is the kind of engineering challenge worth addressing early: what happens when a workflow partially succeeds, retries, or encounters an uncertain outcome? Designing those failure paths upfront can make agentic systems far more dependable in production.
Totally, the failure paths are where most of the design work actually goes. Thanks for reading!
This was a really interesting read. The point about hashing the arguments falling apart with LLM-generated content makes a lot of sense—what looks like the “same action” to us can easily produce slightly different arguments on a replay.
I especially liked the idea of building idempotency around stable intent and real-world identity rather than the generated text. That feels like a much more reliable foundation for agent systems.
The discussion around pending, done, and revoked was also a great touch. These are the kinds of edge cases that probably don’t show up in demos, but become extremely important once agents are performing real actions. Great practical perspective!
Thanks! That’s exactly the gap we kept hitting: everything looks fine in a demo, and the edge cases only show up once the agent touches real systems. If you’re running agents with real side effects, I’d be curious which ones bit you first.