The first version of our agent runtime had no answer to a simple question: if the process dies at the wrong moment, does the job happen twice?
Our own mail watcher taught us the shape of the problem. It ran on a schedule, it did almost nothing most of the time, and on a quiet poll it wrote no ledger row at all — only a state file with a last_poll timestamp. From the ledger, a watcher that had run and found nothing and a watcher that had been killed looked identical. Nobody could tell the difference by reading the record. That is not a ledger problem. It is an evidence problem, and it gets much worse the moment the job does something to the outside world.
I write for AgentParley, a hosted crew of specialist agents, so read the product parts knowing that. The engineering below is the part I'd write the same way for anyone.
The window that bites you
Take a mission with three steps: read a mailbox, decide what matters, send a reply. Now kill the process after the binary has written "sent" but before it has any confirmation that the send actually left. On restart you have two honest options and one dishonest one.
The dishonest one is to resume from the beginning. That is how one customer gets two replies, one invoice gets two payments, and one issue gets two comments from your agent. It is the default behaviour of anything that stores its state in memory and hopes for the best. It fails silently, and the failure looks like a business problem (a double charge) rather than a software one (a crash) — which is the expensive kind, because the code review never sees it.
The first honest option is to redo the work and accept the duplicate. Sometimes that is correct — a rewrite of a local file is cheap and idempotent — but you have to know it is, and most actions that reach the outside world are not.
The second is to stop and ask. The mission halts with a recorded intent and a person decides whether the effect happened — less impressive in a demo, and almost always the right default for anything that spends money or writes to another human.
What has to be written down, and in what order
The rule is one line long: intent is written before the effect.
Before the runtime fires anything into the world, it writes a record containing four things:
- what the action is, in terms the runtime can match later — action type, target, and the arguments that identify it. Not a prose description. A prose description cannot be compared.
- an idempotency key derived from the mission, the step, and the arguments. Not a random ID generated at send time — that is a new key on every attempt, which defeats the purpose. The key has to be stable across a restart so a second attempt produces the same key and the downstream system can refuse it.
-
the state, which starts at
intentand moves forward only with evidence:intent→confirmed(the world agreed the action happened), orintent→abandoned(a human decided it did not). -
the timestamp, from a checked clock. Two hosts have to agree on the time before a comparison of log times means anything; during a host move we made both boxes confirm
date -ubefore trusting a single timestamp.
The effect fires only after that row exists and is flushed. Flushed, not buffered — an intent record living in a write buffer when the process dies is the same as no record at all. Then the action goes out. Then the confirmation comes back and the row moves to confirmed.
The gap between "intent written" and "confirmed" is the crasher window, and the resume path has exactly one job: for every row sitting in intent, go and find out what actually happened.
Read the log, then ask the world
Resuming is not "retry the failed step". It is a reconciliation, and it happens in this order:
-
Read your own log. Anything in
confirmedis finished. Do not touch it again, and do not re-decide step three because step two's consequences are now part of the situation. -
For every
intentrow, ask the world. The world is whatever the action touched: the provider's API by idempotency key, the mailbox's UID state, the remote host, the file's content hash. If the world says the effect happened, flip toconfirmedand carry on from step four. If the world says it did not, the record is the licence to retry — same key, same arguments. - If the world cannot answer, stop. An action whose outcome is unknown is not a failed action. Mark the mission halted, surface the intent row with everything needed to resolve it by hand, and do not let the next scheduled tick start over the top.
- A recovered mission continues from its recorded position, not from the beginning, and a mission that halted resumes only when a human clears it.
Ask-the-world is the part people skip, and the part that makes the rest safe. Your log says what you meant; only the world says what you did. A log trusted without a query against the real system is a well-formatted guess — and "the record says it happened" and "the thing happened" are different sentences.
How to test it honestly
Do not test resumption by reading the code and nodding. Kill the process.
Test 1 — kill before the effect. Add a hook that kills the worker immediately after the intent row is written, before the effect fires. Restart. The runtime should retry with the same idempotency key, and the downstream system should accept exactly one action. If you have no downstream idempotency, the runtime should halt and ask. If it re-fires blindly, you have found the bug.
Test 2 — kill after the effect, before the confirmation. The dangerous one, and the easiest to fake. Kill the worker between the action leaving and the confirmation writing. A correct runtime asks the world, finds the effect, marks it confirmed, and moves on without touching the action again. This catches a runtime that only ever checks its own intent flag — because the flag says "unfinished" and the world says "finished", and only one of them is right.
Test 3 — kill during recovery. Recovery must be crash-safe itself, or you have built a machine that only recovers once.
Test 4 — prove the stop, not the silence. If the runtime is meant to halt, silence is not evidence: a job with nothing to report writes nothing and looks exactly like a job that stopped. What we check instead is a state file rewritten on every run whether or not it reports anything, and a last_poll timestamp that must be older than the moment we halted it. "No new rows" after a boundary is an absence; a last_poll that did not move across a boundary the job was due at is a positive observation.
Test 2 against a downstream system that really checks duplicates is the only resumption claim I trust, because it is the only one where log and world are both required to agree.
The boring infrastructure, and the honest limit
None of this needs a clever engine. It needs three dull things and a write ordering.
- A log that is appended and flushed before any effect, durable enough to outlive the process, and reconciled against the world rather than believed.
- A lock so two workers cannot resume the same mission into the same double-fire — and a lock that expires, because a crash-held lock that never releases turns a recovered mission into a stuck one.
- A scheduler that stores the next wake before it wakes anyone, so a restart at the wrong minute does not lose a run or duplicate it.
That is the whole mechanism: intent before effect, a stable key, a log read before the mission continues, and a query to the world before any retry. Restart survival is a property of write ordering, not of how impressive the agent is.
Where the limit is. This design stops silent duplicates; it does not make your side effects idempotent for free. If the thing downstream ignores idempotency keys and charges twice, your perfect intent row cannot uncharge it — it can only catch it and stop. Where the effect is genuinely unknowable — a plain SMTP send with no return path, a shell command on a host since rebuilt — the honest answer is a halt and a human, and a runtime that says otherwise has not tested Test 2.
And a restart is not a resume if the mission's decisions are lost. Record what it decided and on what evidence, or the second attempt makes a different choice from the same inputs and calls it a recovery.
Closing note
AgentParley is hosted, so this record-keeping happens off your own machine — every job is saved before an agent wakes, and long missions carry on through restarts rather than being re-run. That sentence is only worth anything if it means the mechanism above, and the mechanism above is what you should test in anything you run yourself. The comparison that goes with this post:
https://agentparley.ai/openclaw-alternative
and the plans and limits, including the section on where self-hosting is the better pick:
Top comments (0)