Picture a moment every person who runs an AI agent knows. A task is halfway done and starting to go wrong. The agent took a weird turn a few steps back and now it is confidently heading somewhere bad.
You have to decide fast on this. Do you step in? And if you do a quick "wait, check your work" nudge will that actually fix it? Or do nothing? Or worse knock a run that was about to recover on its own off the rails?
That question is the whole project. Here is the honest short version of what I found.
Detecting a failure is not fixing it
A lot of recent agent research is about failure attribution — figuring out which step in a long run broke everything. Useful but it stops one step short of what you need when you are on call. Knowing where it broke is not the same as knowing what to do about it.
So I asked a blunter question: given a failure, which fix actually recovers the run and which ones quietly make it worse?
To answer it without fooling myself I rewind each failing run to the exact step where it went wrong, apply one fix, let it play forward and check the real answer against a hard ground truth no LLM grading another LLM. And I always compare against a "do nothing" control, because some runs recover on their own, and I did not want to give my fixes credit for that (or miss a "fix" that's actually worse than leaving the agent alone).
What a capable agent actually gets wrong
First surprise: a decent agent mostly doesn't fail in the dramatic ways people worry about. It rarely loops, rarely forgets to answer, rarely fumbles a tool that throws an error in its face.
It fails in two quieter ways and both are the same underlying mistake: acting on the surface of the situation instead of the real thing underneath.
It makes up an answer it could have looked up. The fact it needs is sitting right there behind a tool call it just never makes, so it fills the gap with something plausible. Reads "manager: #202," never looks up who #202 is, asserts a name anyway.
It trusts a tool that didn't complain. Give it a date like
20260730or a time like1430and it hands that straight to a calculator as if it were a number. The calculator doesn't error — it returns a number and that non-error result becomes a stamp of approval on a calculation that never made sense. The tool "validated" nonsense.
That second one is sneaky because it slips through how we usually categorize errors. Nothing errored. A real tool really ran. Any monitor watching for error signals looks right past it.
The finding I like most: it is about permission, not politeness
Here is the part that changed how I think about fixing agents.
I tested a bunch of repair nudges. Some phrased as commands some as statements. I expected the phrasing to matter. It didn't. What mattered was narrower: does the nudge give the agent permission to go back and do the missing work?
Look at these two. Grammatically almost identical. The only difference is one short clause telling the agent it is allowed to check.
| Repair nudge | Recovery |
|---|---|
| "The answer must be the manager's name." | 0.16 |
| "The answer must be the manager's name — use tools to verify." | 1.00 |
Same sentence, one clause of difference. That is the gap between the agent
shrugging and re-asserting its made-up answer, versus actually going back, making the lookup it skipped, and getting it right. I call it action-licensing: a fix works when it licenses the corrective action not when it is worded a certain way.
One honest note, because it is the spirit of the whole thing: I originally thought a different phrasing style was the cause. More data killed that idea — the "winner" just happened to also be the one granting permission. I retracted the old explanation and kept the real one.
Some failures you can't talk your way out of
Skipped-lookup is recoverable. The "tool validated nonsense" one is not — at least not with words. I threw every text-level fix at it and recovery stayed near the floor.
Why is kind of elegant: the thing that corrupted the run was not the prompt, it was a tool result. The agent is holding a clean, well-formed, completely wrong number that a tool "confirmed." Asking it to re-read the question doesn't remove that false stamp of approval, because the stamp does not live in the prompt. You can't reach it from there.
Useful line to draw: some failures are prompt-shaped and some are not and no amount of prompt-fiddling fixes the ones that are not.
Two more things worth knowing
Catch it early. The deeper into a task the agent gets before you catch the skip, the less any fix buys back. Command-style nudges that recover well early collapse toward zero when you catch the mistake late.
Beware cross-model comparisons. I tried a second, unrelated model. The
make-up-an-answer failure showed up there too, so it's not a one-model quirk. But I couldn't test recovery on it — it wasn't reliable enough at calling tools in the first place, and every fix depends on the agent re-issuing a tool call. So "the fix failed" and "the model can't call tools" become impossible to tell apart.
If you benchmark agent recovery across models, check each one clears a basic tool-reliability bar first, or you'll measure the wrong thing.
What this is, and is not
Plainly, so I don't oversell it: all the recovery numbers come from one primary model (a second confirmed the mechanism but couldn't test recovery). It is a clean, synthetic sandbox — that is what buys the hard ground truth, but it is not a messy real-world workload. And a few of the smaller numbers are directional trends, not precise measurements. This is an early, careful result, not a final word.
If you build agents, three takeaways
- When you nudge a stuck agent, give it permission to redo the work — don't just restate what you want.
- Not every failure is prompt-fixable. If a tool "validated" bad data, talking to the model won't undo it that is a job for tool-side guardrails.
- Catch failures early, and don't benchmark recovery on a model that can't reliably call tools.
Top comments (1)
The control comparison is the part I would keep in every recovery benchmark. I would add one more split: did the repair restore the missing observation, or merely make the next step look plausible? For skipped lookups, log the required evidence and the tool call that produced it; for malformed inputs, validate at the tool boundary and return a typed failure instead of a numerically valid result. That makes “action-licensing” measurable without letting a successful tool invocation count as proof. Have you tried replaying the same repair with the tool output withheld to separate recovery from prompt compliance?