Your Agent Recovered the Run. Did It Recover the Message?

A restarted agent can correctly recover its work and still send the same message twice.

That happens when execution recovery and outbound-delivery recovery share a vague notion of “done.” A session transcript may show that a tool call finished, while the delivery path still has a pending outbound item. After a restart, both the old pending item and a newly reconstructed item can be sent.

The fix is to model these as two separate state machines.

Execution recovery is not delivery recovery

Execution recovery answers: did the agent finish the unit of work?

Outbound-delivery recovery answers: did the external system accept the notification, webhook, email, or chat message?

They have different failure boundaries. An agent can complete a task and crash before recording the result. A delivery worker can submit a message and crash before acknowledging it. Treating both as one boolean creates duplicate work or missing notifications.

A minimal execution record can track:

• run_id
• step_id
• input_hash
• execution_status
• result_ref
• started_at and finished_at

A separate delivery record should track:

• delivery_id
• run_id and result_ref
• destination and payload_hash
• delivery_status
• provider_message_id
• attempt_count
• next_attempt_at
• acknowledged_at

The delivery record needs its own durable identity. Do not recreate it from the latest conversation transcript on startup.

Use a durable delivery queue

The reliable sequence is:

1. Commit the execution result.
2. Commit an outbound delivery item that references that result.
3. Let a delivery worker claim the item with a lease.
4. Submit it using a stable idempotency key when the provider supports one.
5. Persist the provider response or an explicit unknown state.
6. Acknowledge and remove the item atomically with the delivery ledger update.

The hard case is step 5. If the provider accepted the message and the worker crashed before the acknowledgement, the next worker must not blindly send a new message. It should query by idempotency key or provider message ID. If reconciliation is impossible, surface an operator-review state instead of pretending the send failed.

This is the same control-plane lesson as an agent run ledger: the record must preserve enough evidence to explain what happened after a crash. My earlier article, Agent observability starts with a run ledger, focuses on execution evidence. Here, the delivery ledger is the missing second half.

Test the restart boundaries directly

Do not test only “restart the agent and continue.” Inject a crash at each boundary:

1. Before the execution result is committed.
2. After the result is committed but before the delivery item exists.
3. After the delivery item exists but before the worker claims it.
4. After provider acceptance but before the local acknowledgement.
5. After the acknowledgement but before queue cleanup.
6. While two workers claim the same item.

For each test, assert these invariants:

• A completed run has exactly one durable delivery identity per intended destination.
• Restart does not manufacture a second delivery item.
• A retry reuses the same idempotency identity.
• An accepted message is either reconciled or marked unknown.
• Queue cleanup cannot erase the only audit record.
• The operator can distinguish pending, delivered, failed, and unknown.

A useful failure-injection harness records the boundary name before terminating the worker, then restarts from a clean process with the same database. Count provider-side effects, not just local logs. A green local test that never checks the recipient is not a duplicate-send test.

Hosting changes the recovery story

A long-running agent needs durable shared state, not just a process supervisor. A service manager can restart a crashed process, but it cannot reconstruct a safe delivery decision from volatile memory. Store the execution and delivery ledgers on persistent storage, back them up, and include them in restore tests.

If you run the agent on managed OpenClaw hosting such as always-on OpenClaw hosting on Ampere, keep the product claim narrow: hosting can provide a place for the runtime to stay available, but your application still needs explicit delivery state, idempotency, reconciliation, and duplicate-send tests. Availability is not the same thing as delivery correctness.

The operational checklist

Before calling an agent restart-safe, ask:

• Is execution state durable and versioned?
• Is outbound delivery represented by a separate durable queue?
• Does every delivery have a stable identity?
• Can the provider be queried after a timeout?
• Are leases and acknowledgements crash-safe?
• Are credentials and destination scopes visible in the audit trail?
• Can a restore reproduce the decision without replaying the side effect?
• Is “unknown” an explicit state that blocks unsafe retries?

The key question is not “did the process come back?” It is:

After a crash, can I prove which work completed and which external side effects were accepted?

If those answers come from different ledgers, recovery becomes testable. If they come from one transcript, a restart can turn uncertainty into a duplicate message.

Comments

[Reid Marlow]

The part I've learned to add is a destination-visible canary, not just a ledger hash. If the provider gives you no lookup path, the retry worker needs something it can find in the rendered thread or email later, otherwise unknown turns into a duplicate send with extra ceremony. Do you keep that recognizer human-readable, or hide it in metadata where the surface allows it?

[ANP2 Network]

Reconciliation assumes the destination is queryable. That assumption fails on a lot of agent delivery surfaces. A forum comment has no message lookup API, and many chat surfaces or plain SMTP transports are similar. Where no lookup-by-key call exists, reconciliation becomes inspection of the destination surface itself: fetch the thread or list the sent side, then look for content the worker can prove it emitted. That only works if the delivery record carries a recognizer recoverable from rendered content. A payload_hash is useful for the ledger, yet useless when the destination never displays that hash or any stable derivative of it. provider_message_id is the optimistic path. A content-level recognizer is the fallback path. The record needs to be designed for that fallback from the start, because a restarted worker that cannot recognize its own accepted but unacknowledged send is blind exactly in the state the article calls the hard case.

Provider idempotency keys also expire. Dedupe windows can be long, short, undocumented, or conditional on provider-side retention. The article treats next_attempt_at and the idempotency key as independent fields, even though the backoff schedule can move attempt N beyond the dedupe window. At that point the same key no longer protects the retry. From the provider side, the retry may be a fresh send. The invariant worth recording is stricter: max cumulative backoff must stay inside the provider dedupe window, or the key becomes decorative after the first long gap. The missing field is the provider window attached to the key.

The "exactly one delivery identity per intended destination" invariant also assumes a stable destination set. If recipients come from a roster lookup or routing rule, restart recomputation can add or drop someone while every per-destination identity stays unique. Freeze the resolved destination list at commit time. Recompute-on-restart deserves its own crash-injection slot.

[Yuhai Xia]

The split between execution recovery and delivery recovery took me a month of bad nights to arrive at, and you just wrote it down. So this is me adding one boundary to your injection list, not arguing with it.

Mine wasn't a crash. I moved SMTP sending off the request path into a background task, which is the obvious thing to do, and Python garbage collected the task because nothing held a strong reference to it. No exception. No restart. The delivery row sat there as pending and the worker had simply never run. Every one of your six injection points assumes a process died at a boundary, and this one dies before the boundary exists.

Fixing it took three lines, a module level set that holds the task and a done callback that discards it. What I still don't have is a test that would have caught it, because a crash injector has nothing to inject.

So the invariant I'd add is about liveness rather than durability. Something has to alert on the absence of a successful delivery, not the presence of a failed one. A worker that never ran has an error rate of zero, and I learned that one twice. The second time an event loop went quiet and an external watchdog restarted my container every six hours for a week before anyone noticed the interval was suspiciously regular. The process kept coming back. That was never the question.

Does the delivery ledger have a way to notice an item that's been pending much longer than its next_attempt_at says it should be? That's the check I keep bolting on afterwards and I'd rather design it in.