FAILED is not UNKNOWN: the retry bug hiding in every AI agent

Say an agent issues a $500 partial refund on a $2,000 order. Stripe processes it, but the response never makes it back to you. Maybe a proxy timed out, maybe the connection dropped, maybe the container got OOM-killed mid-request. Your code sees an exception, and your retry decorator does what retry decorators do. The customer now has $1,000 back.

The model didn't do anything wrong here. It picked the right tool and passed the right arguments. The bug is in the retry loop almost everyone writes without thinking about it:

for attempt in range(3):
    try:
        return stripe.Refund.create(payment_intent=pid, amount=amount)
    except Exception:
        if attempt == 2:
            raise
        time.sleep(2 ** attempt)

The assumption baked into this is that an exception means the refund didn't happen. That's usually true, which is why this code gets through review. It isn't always true.

A call has three outcomes, and retry code usually handles two

Most retry logic treats the outcome as a boolean: it returned or it raised. But a call that leaves your process can end three ways:

Outcome	What you know	Safe to retry?
Committed	The other side acted, and you have proof	No, it's done
Failed	The other side didn't act, and you have proof	Yes
Ambiguous	You don't know	No

Ambiguous isn't a rare edge case. It's what you get from a timeout, a connection reset, a 502 from a load balancer, a gateway that gave up before the origin finished, or your own process dying between sending the request and reading the response. This is the two generals problem and there's no clean solution to it. What you can do is refuse to treat "I don't know" as "it failed."

None of this is new. Databases dealt with it decades ago with two-phase commit, and payment APIs have idempotency keys for exactly this reason. Stripe has them, and if you pass a stable key Stripe will dedupe the refund for you. But a lot of agent tool wrappers don't pass one, or generate a fresh UUID on every call, which dedupes nothing. Stripe also only keeps keys for 24 hours, and most of the other APIs your agent touches don't support them at all.

What's new is who's doing the retrying. A cron job retries in one predictable way. An LLM reads the error string, decides the action probably didn't go through, and tries again on its own. Sometimes with slightly different arguments, sometimes three turns later, sometimes from a different worker. Then it tells you the refund went through.

Write the record before the call

The obvious fix is to keep a ledger:

result = do_refund(payment_id, amount)
db.mark_done(f"refund:{payment_id}")   # too late
return result

This doesn't help. The case you care about is the one where the call is in flight and the process dies, and in that case this code never writes the row. The retry shows up, finds nothing, and refunds again.

You have to claim the effect before you make the call:

key = f"refund:{payment_id}"

if not store.reserve(key):        # atomic insert on a unique key
    raise DuplicateEffect(key)    # someone already claimed this refund

try:
    result = do_refund(payment_id, amount)
except ProviderRejected:
    store.mark_failed(key)        # provider said no, nothing happened, safe to release
    raise
except Exception:
    store.mark_ambiguous(key)     # timeout, reset, anything else: keep holding it
    raise

store.commit(key, result)
return result

The except blocks are where all the safety is. Only an explicit rejection from the provider releases the key. Anything else keeps it held. If the process dies outright, neither block runs and the row just sits there as reserved, which is also what you want. It stays held until something outside the agent settles it, either a reconciliation check against the provider or a person looking at it. Guessing doesn't get to clear it.

The key matters too. refund:txn_4821 names a business action, so it has to come out the same across a retry, a second worker, a restart, and a brand new conversation with the model. If you put a timestamp, a UUID or a trace ID in it, every attempt gets its own key, nothing ever collides, and what you've built is a log.

Approvals have the same bug

Human-in-the-loop flows have their own version of this, and it's worse because it looks like it's working.

Someone approves a $500 refund and the agent gets approved: true back. Two turns later the agent re-plans, decides the refund should be $1,500, and calls the tool. It still has an approval. The approval was a boolean, and a boolean doesn't know what it was approving.

The fix is to bind the approval to the exact arguments the person saw:

approval_hash = sha256(canonical_json({
    "action": "stripe.refund",
    "payment_id": "txn_4821",
    "amount": 500_00,
})).hexdigest()

If any field changes, the hash doesn't match and the call stops. Approvals should also be single use. One that can be replayed is really a standing permission, and nobody meant to hand that out.

The same goes for the tool list. The model being able to call a tool says nothing about whether this user is allowed to perform that action, with these arguments, right now. Your prompt answers the first question. Nothing answers the second unless you write it.

Prompt guardrails don't cover this

Most agent safety tooling looks at what the model says: prompt injection filters, output classifiers, jailbreak detection, PII scrubbing. That stuff is useful, but it's aimed at the model, and in every example above the model was fine.

The failure happens after the agent decides and before the real system changes. That's one function call with no natural language in it anywhere. Checking it takes something that never looks at a prompt, only sees the action and its arguments, and asks:

• Is this caller allowed to do this at all?
• For these exact arguments, should it go through, wait for approval, or be denied?
• If someone approved something, was it this?
• Could this have already happened?
• Did the other system actually act, and do we know that for sure?

CTRLRun

I got tired of writing the reservation table by hand on every project, so I turned it into a library.

CTRLRun is an open-source Python library (Apache-2.0) that sits at that boundary. It runs inside your process instead of as a service in front of it, and it never sees your prompts, your model or your reasoning traces.

pip install ctrlrun

import ctrlrun

@ctrlrun.protect(
    "stripe.refund",
    effect="refund:{payment_id}",
)
def refund(payment_id, amount):
    ...

With that decorator in place:

• If policy says allow, the effect is reserved and then your function runs.
• If it says approve, you get ApprovalRequired, bound to these exact arguments.
• If it says deny, you get ActionDenied and no approval request is created.
• A second attempt at an effect that already committed raises DuplicateEffect and hands back the original receipt.
• If the response gets lost, the effect is held as ambiguous and a blind retry gets AmbiguousEffect.
• Every decision and its outcome goes into a receipt.

It uses SQLite on a single host and Postgres across hosts. State survives restarts, which is the whole point, since a process that died mid-call still leaves its claim behind for the retry to run into. There's also an MCP gateway if your agent talks over MCP, and a ctrlrun verify command you can run in CI to check a set of guarantees.

If you want to poke at it without installing anything, ctrlrun.dev has a demo that walks through each of these failures in a few different domains. No signup. The "try it" page runs the actual released wheel in your browser through Pyodide, so the refusals you see come from the real library.

Even if you don't use the library

Go look at the retry logic around whatever your agents do in production and check what it does on a timeout. If it retries, you have this bug. It probably hasn't cost you anything yet because your volume is low and most timeouts really are failures.

---

Code is at github.com/CTRLRun/ctrlrun. Issues and pushback are both welcome, especially if you've hit a failure mode I haven't modelled.

Comments

[Richard Smith]

The LLM vs cron retry distinction is the key insight. A cron retries blindly. An LLM re-reasons and can arrive at a different conclusion entirely. AMBIGUOUS is the only safe answer.

[Arpan Ghoshal]

That's what this is about 🔥

[Dean Lee]

The distinction between FAILED and AMBIGUOUS is the difference between an outright rejection and an unconfirmed fill in trading systems. When an order message drops without an acknowledgment, treating it as unfilled leaves an open short on the book. Financial clearing learned decades ago that an unconfirmed state must remain reserved until external reconciliation proves the order never crossed.

The reason agent frameworks routinely ship naive retry decorators comes down to how evaluation harnesses are scored. Standard benchmarks reward completion rates within bounded turn counts, so aggressive retry loops look like resilience in synthetic test suites. A pessimistic reservation lock that halts on ambiguity lowers the benchmark score because it requires out-of-band reconciliation before resuming. The incentive in early benchmark design actively rewards duplicate side-effect risk over transaction discipline.

[Arpan Ghoshal]

Yeah, the trading analogy is a good one. That’s basically the same failure mode.

I also like the benchmark point. I hadn’t framed it that way, but it makes sense that “finish the task” benchmarks can accidentally reward aggressive retries while treating a safe stop for reconciliation as a failure.

That gap between benchmark success and real-world transactional safety is probably worth talking about more.

[kevinbai]

The AMBIGUOUS-is-the-normal-outcome framing is the right mental model, and the LLM twist makes it worse in a way the article only gestures at: the agent doesn't just retry the same call — it re-reasons about the failure and may mutate arguments, so even an idempotency key keyed on (tool, args) won't dedupe attempt 2. Practical addition: bind the idempotency key to the agent's intent (task id + step index) rather than the rendered arguments, and make AMBIGUOUS a first-class tool result the model can see ('unknown — check before acting') instead of an exception it interprets. Exceptions invite re-derivation; results invite verification.

[Arpan Ghoshal]

Yeah, this is the interesting part.

If the agent re-reasons and changes the arguments, deduping only on the rendered call can definitely miss that it’s still trying to complete the same underlying action.

I think I’d separate two things though: the stable identity of the effect, and the exact arguments being executed. The first tells you “this is still the same real-world operation,” while the second is what you bind approval and execution to.

Also agree on AMBIGUOUS being explicit. A generic exception is way too easy for an agent to interpret as “try again.” Making the unknown state something it has to reconcile before acting again is much safer.

Good point.