FAILED is not UNKNOWN: the retry bug hiding in every AI agent

An agent refunds a customer $500. Stripe processes it. The response never comes back — a proxy timeout, a dropped connection, a container that got OOM-killed mid-call. Your code sees an exception. Your retry decorator does what retry decorators do.

Now the customer has $1,000.

Nothing in that sequence is an LLM problem. The model reasoned correctly, picked the right tool, and passed the right arguments. The bug is in the four lines of infrastructure everybody writes without thinking:

for attempt in range(3):
    try:
        return stripe.Refund.create(payment_intent=pid, amount=amount)
    except Exception:
        time.sleep(2 ** attempt)
raise

That code encodes an assumption that is simply false: that an error means it didn't happen.

Two states is one state too few

Almost every retry system in the wild models outcomes as a boolean. Success, or failure. Returned, or raised.

A call that leaves your process has three possible outcomes:

Outcome	What you know	Safe to retry?
COMMITTED	The remote system acted, and you have proof	No — it's done
FAILED	The remote system did not act, and you have proof	Yes
AMBIGUOUS	You have no idea	No

AMBIGUOUS is not a rare edge case. It is the normal result of a timeout, a connection reset, a 502 from a load balancer, a gateway that gave up before the origin did, or your own process dying between the request and the response. In distributed systems this has a name — the two generals problem — and it has no clean solution. What it has is a discipline: never collapse "unknown" into "failed."

Databases have understood this for forty years. That's what two-phase commit is about. Payment providers have understood it for twenty; that's what an idempotency key is. Agent frameworks are ten months into shipping software that takes consequential action, and most of them still have a max_retries parameter and no concept of an unknown outcome at all.

The difference now is who's driving. A cron job retries in one predictable shape. An LLM retries because it read an error string, decided the action didn't go through, and reasoned its way to trying again — sometimes with slightly different arguments, sometimes three turns later, sometimes from a different worker. It will do this confidently, and it will tell you it succeeded.

The ordering is the whole trick

The instinct is to write a ledger:

result = do_refund(payment_id, amount)
db.mark_done(f"refund:{payment_id}")   # too late
return result

This does nothing. The window you care about is exactly the window where you have no row: the call is in flight, the process dies, and the retry arrives to find an empty table.

You have to claim the effect before the call:

key = f"refund:{payment_id}"

if not store.reserve(key):          # atomic insert, unique constraint
    raise DuplicateEffect(key)      # someone already claimed this

try:
    result = do_refund(payment_id, amount)
except TimeoutError:
    store.mark_ambiguous(key)       # held, NOT released
    raise
except ProviderRejected:
    store.mark_failed(key)          # provably didn't happen — safe to release
    raise
else:
    store.commit(key, result)
    return result

Read the except blocks twice. The entire safety property lives there. A timeout does not release the reservation. That reservation stays held until something outside the agent settles it: a reconciliation call to the provider, or a human. AMBIGUOUS is a state you live in, not a state you clear by guessing.

And the key itself matters. refund:txn_4821 is the identity of a business action. It has to be the same string across a retry, a second worker, a restart, and a fresh conversation with the model. If your key includes a timestamp, a UUID, or a trace ID, you don't have deduplication — you have a log.

The other half: an approval is bound to arguments

The same class of bug shows up in human-in-the-loop flows, and it's uglier because it looks like it's working.

A person approves a $500 refund. The agent gets approved: true back. Two turns later the agent re-plans, decides the amount should be $5,000, and calls the tool. It still holds an approval. The approval is a boolean, and booleans don't remember what they were about.

An approval should be bound to the exact arguments the human read:

approval_hash = sha256(canonical_json({
    "action": "stripe.refund",
    "payment_id": "txn_4821",
    "amount": 500_00,
})).hexdigest()

Change one field and the hash no longer matches, so the approval authorises nothing and the call stops. Same principle for single use: an approval that can be replayed is a permission, and you didn't mean to grant a permission.

While we're here — a tool being present in the agent's tool list is not permission either. "The model can call it" and "this principal may perform it, with these arguments, right now" are different questions, and only one of them is answered by your prompt.

This is not a guardrails problem

Most of the safety tooling in this space watches what the agent says: prompt injection filters, output classifiers, jailbreak detection, PII scrubbing. All useful, all aimed at the model.

None of it helps here. The model was fine. The failure happened in the gap between "the agent decided" and "the real system changed" — one function call wide, no natural language in it at all. That gap needs a different kind of check, one that never sees a prompt and only sees an action with its exact arguments:

• Is this principal entitled to act at all?
• For these exact arguments: allow, require approval, or deny?
• If a human approved something, was it this?
• Could this effect already have happened?
• Did the real system act, and do we know for certain?

Five questions, asked in the last moment before the effect is real.

CTRLRun

I got tired of writing the reservation table by hand on every project, so I built the thing.

CTRLRun is an open-source Python library (Apache-2.0) that sits at that execution boundary. It's a library inside your process, not a service in front of it. It never sees your prompts, your model, or your reasoning traces.

pip install ctrlrun

import ctrlrun

@ctrlrun.protect(
    "stripe.refund",
    effect="refund:{payment_id}",
)
def refund(payment_id, amount):
    ...

What you get around that call:

• allow → the effect is reserved, then your function runs
• approve → ApprovalRequired is raised, bound to these exact arguments
• deny → ActionDenied, and no approval request is even created
• a second attempt at a committed effect → DuplicateEffect, with the original receipt returned
• a lost response → the effect is held AMBIGUOUS, and the blind retry gets AmbiguousEffect
• everything that happened, and the decision behind it, in a receipt

SQLite on one host, Postgres across hosts. State survives a restart, which is the point — a process that died mid-call still leaves a claim behind for the retry to hit. There's an MCP gateway if your agent talks over MCP, and a ctrlrun verify command that checks a set of guarantees in CI.

There's a browser demo at ctrlrun.dev that walks through each of these failures across a bunch of domains — no signup, and the "try it" page runs the actual released wheel in your tab via Pyodide, so the refusals you see are the library's own.

Take the idea even if you skip the library

If you only keep one thing from this: go find the retry logic wrapped around whatever your agents do to production, and check what it does with a timeout. If it retries, you have this bug. It hasn't cost you anything yet because your volume is low and most timeouts really are failures.

Most of them.

---

Source: github.com/CTRLRun/ctrlrun. Issues and disagreement both welcome — particularly if you've hit a failure mode I haven't modelled.

Comments

[Dean Lee]

The distinction between FAILED and AMBIGUOUS is the difference between an outright rejection and an unconfirmed fill in trading systems. When an order message drops without an acknowledgment, treating it as unfilled leaves an open short on the book. Financial clearing learned decades ago that an unconfirmed state must remain reserved until external reconciliation proves the order never crossed.

The reason agent frameworks routinely ship naive retry decorators comes down to how evaluation harnesses are scored. Standard benchmarks reward completion rates within bounded turn counts, so aggressive retry loops look like resilience in synthetic test suites. A pessimistic reservation lock that halts on ambiguity lowers the benchmark score because it requires out-of-band reconciliation before resuming. The incentive in early benchmark design actively rewards duplicate side-effect risk over transaction discipline.