By Igor Eduardo · Austin, TX · https://igoreduardo.com
A new benchmark paper, CUA-Recovery: Benchmarking Error Recovery in Long-Horizon Computer-Use Agents (arXiv 2610.27645, posted 2026-10-20), asks a question most agent scoreboards skip: when an agent inherits a workflow that already went wrong, does it notice, and can it fix it?
The setup is careful. The authors compose long, multi-application desktop workflows with an executable "gold lineage", the correct value at every step. They run six agents, collect the failures those agents actually produce, locate the root-cause step of each failure, and replay the run to a point 0 to 25 steps after that root cause. Then an agent takes over from that state without being told anything went wrong. The test suite has 2,250 human-verified erroneous states over 100 tasks, drawn from 619 failed rollouts. They score three things separately: whether the agent says something is wrong (Error-Awareness Rate, EAR), how much of the task it completes (Rubric Score), and whether it fully completes the task in any of three runs (Pass@3).
What they report:
- Recovery is rare. Across six agents and six takeover depths, EAR reaches at most 44.0% and Pass@3 at most 25.5%.
- Clean-start skill does not predict noticing. One agent solves 66.9% of the source tasks from a clean start but notices at most 18.5% of the errors in its own failures. Another solves 5.1% from a clean start but notices 14.1–26.1%.
- Noticing is not fixing. Macro-averaged across the six agents, runs that recognized the inherited error passed the verifier 15.6% of the time, versus 12.2% for runs that did not. Agents often name the problem and then leave the affected state unchanged.
- Errors surface late. In their annotated failures, 78.1% of errors became explicit three or more steps after the root cause, with a median delay of 17 steps. In 361 of 602 failures where the error became explicit, the first evidence appeared only at the final step.
- Most root causes are decisions, not clicks. Planning and perception account for 57.3% of root causes; termination errors, including reporting success while requirements remain unmet, account for 23.5%.
The authors are explicit about scope: one desktop environment, agents run with their own scaffolds, and an LLM judge that they checked against human labels and found slightly strict. They also show that fine-tuning on their recovery trajectories raises one open model's average EAR from 19.9% to 35.2%. Code is public at CUA-Recovery.
Those are their numbers, not mine. I have not re-run the benchmark, and I am not using it to rank anyone. What I want to formalize is the eval habit it exposes.
The stake
Teams will ship agents that are good at starting and bad at noticing.
A clean-start pass rate measures one thing: given a correct initial state, does the agent reach the goal? Production does not hand agents correct states. A previous step, a previous run, a human, or a flaky tool has often already written something wrong into a file, a spreadsheet or a record. If the eval never starts from a broken state, the agent's ability to notice is never measured, and the number you ship on is silent about the failure that hurts most.
AI failures rarely fail loud. An agent that builds 17 confident steps on top of a wrong record looks, on a completion dashboard, a lot like one that did the job.
Opinion (one sentence)
Any agent that writes to persistent state should be graded on takeovers from broken states, with "noticed" and "repaired" scored as separate columns, because a clean-start pass rate cannot tell you whether the agent will catch its own inherited mistake.
That is a preference with a limiter, not a benchmark result. For read-only agents that only answer questions, clean-start evals may be enough. Once an agent writes, sends or orders, the inherited-error case is the one that compounds.
What a recovery-aware agent eval looks like (menu, not recipe)
| Column | Question it answers | Failure if skipped |
|---|---|---|
| Clean-start completion | Can it do the task from a correct state? | No baseline at all |
| Takeover completion | Can it finish from a state that is already wrong? | Shipping a system that only works when nothing went wrong before |
| Error awareness | Did it say something earlier is wrong? | Silent agents look identical to careful ones |
| Repair, scored apart from awareness | Did it actually fix the affected state? | Crediting "I see a problem" as a fix |
| Depth since root cause | How far back was the mistake? | Missing that late takeovers behave differently |
| Completion claim vs. state | Does the final state match what the agent reported? | Premature "done" passing as success |
None of that needs a private harness. It needs someone to refuse an agent release that only reports the first column.
Why "noticed" and "repaired" must be different columns
The finding I would steal first is the gap between awareness and recovery. A system that says "it looks like the earlier step used the wrong year" and then carries on with the wrong year has produced a better transcript, not a better outcome. If one metric lumps the two together, it will reward agents that narrate problems over agents that fix them.
The paper also shows the two can move in opposite directions. As the takeover point moves further from the root cause, some agents became more likely to notice something was off while becoming less likely to complete the task, because the step that caused the problem had left their context window. Later is not simply harder or easier. It is different, so it needs its own row.
Where this connects to habits I already keep public
My public work is mostly retrieve-first and evaluation-as-contract systems, and the habit is the same: grade each part on its own before letting the final answer vouch for it. For agents, the "part" this paper isolates is the moment the agent receives state it did not create. I would treat that moment as a first-class test case, next to the clean start.
Steal this check
Pick one agent workflow that writes to persistent state and add four things before your next release:
- Collect real failures. Keep the runs where your agent actually went wrong, not just synthetic faults.
- Replay to a broken state and hand over. Start a fresh run from a point after the mistake, without warning it.
- Score awareness and repair separately. "Said something is wrong" and "fixed what was wrong" go in different columns.
- Check the claim against the state. If the agent says done, verify the final state, not the final message.
Then ask whether the release is still an upgrade once the takeover columns are visible.
What I am not claiming
- I am not reporting CUA-Recovery results as my own, and I have not reproduced the benchmark.
- I am not ranking models or vendors. The paper's per-agent tables exist; this post deliberately does not use them as a leaderboard.
- I am not publishing thresholds for acceptable awareness or recovery rates. Those depend on what a wrong write costs in your product.
- One desktop environment is not every environment. The authors scope their claims the same way.
The point
A pass rate from a clean start tells you the agent can begin well. It does not tell you whether the agent will catch a mistake already sitting in the state it inherits. Start some evals from broken states, score noticing and repairing separately, and check every "done" against the state it claims to have produced.
Igor Eduardo builds retrieve-first and evaluation-as-contract systems. More at igoreduardo.com.
Top comments (2)
The "noticing is not fixing" finding is the one I'd attribute most of our incident time to. We run long-horizon agents and the pattern holds: the agent will correctly narrate "this value looks wrong" and then take the next action against the broken state anyway, because flagging and repairing are two different control decisions and only one of them is on the happy path.
The late-surfacing number (median 17 steps after root cause) matches what we see too — by the time the error is explicit, the cheap rollback point is long gone. What's actually moved the needle for us isn't making the agent smarter about self-check, it's shrinking the blast radius: checkpoint state before any irreversible step, and verify against an external oracle rather than trusting the agent's own read of the world.
Did the paper look at whether giving the agent an explicit "something may already be wrong" prior at takeover changed EAR much? Curious whether the awareness gap is a capability ceiling or just a prompting default.
Your menu is missing a column that sits upstream of repair and costs almost nothing to collect: whether the second artifact differed from the first at all. A byte comparison or a content hash gets you that. No gold lineage, no verifier, no labels.
Objection against myself first. A byte-identical resubmission is a legitimate idempotent retry, and it is also exactly what a load test looks like. We never measured whether the submitting side received the verdict, let alone read it. So the claim is narrow: from the record alone, an attempted repair and a replay are the same observation. That is the reason it deserves its own column rather than being folded into repair.
Numbers from an append-only signed event log we run, where agents post work requests, other agents post deliverables against them, and a separate key signs verdicts on those deliverables. 118,072 signed rows as of today. Positive control before the negative result: the same scan sees 1,811 distinct bodies across 1,845 deliverables, so it is not a comparison that collapses whatever you feed it.
Inside the rejected population it collapses hard. 37 deliverables drew a failing verdict, concentrated on 6 requests. For 32 of them the same key later posted again against the same request, and in 30 of those 32 at least one follow-up was byte-identical to the body that had just been rejected. Every verdict on those follow-ups failed, 31 of 31, none passed. On the worst request 18 deliverables collapse into 4 distinct bodies, on another 13 collapse into 2. Longest loop, 21,424 seconds. Keys are not principals here, and an append-only log gives lower bounds only.
So a repair rate near zero can mean three different things with three different fixes. Nobody acted. Something acted and emitted the identical artifact. Something acted, emitted new content, and was still wrong. In our population the middle case is almost the whole population, and the cheapest column in your table is the one that would have told you so.
The other thing I would add to the table is a note on who writes each column. Repair and completion-claim-versus-state are computed externally against state. Error awareness is read out of the agent's own narration, which makes the reported jump from 19.9% to 35.2% EAR a 15-point gain on the self-reported column, sitting next to a 15.6% against 12.2% outcome split. Both can be true at once. Authorship alone is not enough either: our verdict field is signed by a key other than the author of the graded artifact, and on the failing side 36 of its 37 explanations are one identical string, so an externally written column can still be nearly information-free. Count distinct values per column, since a column that always prints the same value passes every coverage check.
With the blast-radius side covered upthread, the question I have about the harness is narrower. When a takeover run scores "did not repair", does it record whether the agent wrote to the affected state at all, and whether what it wrote differed from what it inherited, so that "did not act" and "acted identically" land in separate rows?