TL;DR: Every agent framework teaches you two outcomes: success and failure. Production taught me a third, and it is the expensive one. Unknown: the request left, the process died, and nobody knows whether the thing happened. Treat unknown like failure and retry, and you double-post, double-email, and double-charge. Treat it like success and move on, and you silently lose work. The only correct first move is neither. It is reconcile: read the world back, prove what happened, then decide.
I measured the ground my own fleet runs on this week. Over 72 hours, the workers that execute my personal AI's jobs went through 30 generations — measured from run timestamps, so these are lower bounds. The median generation lived about two hours. The shortest lived about 22 minutes. There is no quiet window, no safe hour to sneak a long job through. Twice in late October, a long publishing call was killed mid-flight by exactly this turnover and came back as unknown. The job had not failed in any provable sense. It had simply outlived its executioner.
Retry is not a safety mechanism. In the unknown state, it is the hazard.
The standard advice — add retries, add exponential backoff, retry until it succeeds — quietly assumes failure means "it didn't happen." That assumption is true for a read. It is false for a side effect. When a post, a payment, or an email is in flight and the caller dies, "didn't happen" and "happened, but the receipt died with the process" look identical from the survivor's chair. A blind retry collapses those two worlds into one action, and in one of them that action duplicates the side effect. This is why serious payment systems do not tell you to retry harder. Stripe's own guidance for an outcome that stays unknown is to reconcile first — check what actually happened — not to fire again with a new key, or even the old one, until you know.
The mirror error is just as costly. Mark the job done because it probably worked, and losses go silent: the post that never posted, the backup that never landed, filed under green. Unknown deserves its own branch in every runbook, with its own verb. Not retry. Not continue. Reconcile.
What actually survives a dying process
Four unrelated industrial systems — Temporal, Cloudflare Workflows, DBOS, and LangGraph — converged on the same shape, which is usually a sign you are looking at the shape of the problem, not a vendor feature. State lives somewhere that outlives the process. Work is cut into small steps, and each step's result is written down before the next step starts. Recovery means reading that record and skipping what is already done, not starting over. And there is a catch none of them can engineer away: a step's side effect and its checkpoint are not atomic. A process can die after the world changed but before the record did — which is exactly the unknown state again, which is why the record alone is not enough. The side effect itself has to be safe to re-enter: an idempotency key the receiver honors, or a draft state you can read back, or a receipt you can reconcile against.
There is no exactly-once delivery across systems. What exists is at-least-once delivery plus deduplication — effectively once, if you do the unglamorous work. Anyone who promises you the first phrase is selling you a fiction.
Here is the part I did not expect: my fleet was already running a handmade version of this. Our watermarks are checkpoints. Our ledgers are event histories. Our rule that nobody may report a job complete without reading the target state back is reconciliation, we just had not named it as the mandatory first move after unknown. What was missing was not a framework. It was the discipline written down: every step leaves a line on disk — what was attempted, with what key, and what evidence proves it — and any task expected to run longer than about twenty minutes, or carrying an irreversible side effect, is designed that way from the start. Short, reversible jobs are exempt; durability has a fixed cost, and spending it everywhere is its own kind of waste.
So I no longer ask whether a system is smart, or even whether it is reliable. I ask three questions. Does every step land on disk? Can every side effect be safely re-entered? And when the answer comes back unknown, does it reconcile before it retries? A system that cannot answer those three is not running your work. It is gambling with it, one short-lived process at a time.
— Jiahui Miao, founder of vertciti, building a personal AI operating system for one.
Top comments (1)
Reconcile before retry is the right first move. How do you do the read-back when the side effect went through a UI with no API, so there is no idempotency key and no receipt to query?
From the Auten team (screen control for AI agents), we hit this a lot. What helped: before the click, write down what the screen should show if it worked (the post in the feed, the message in Sent), and check for exactly that before any retry.