Your Agent's Failure Predictor Is Grading the Task, Not the Run
By Igor Eduardo · Austin, TX · https://igoreduardo.com
A short new paper, Hard Issue, Weak Model, or Bad Run? Rethinking Failure Prediction for Coding Agents (arXiv 2611.08285, posted 2026-11-05), asks a question most agent monitoring skips: when a predictor flags a run as "going wrong", has it learned anything about that run?
The setup is familiar. A coding agent works on a repository issue for many steps. A predictor reads the partial trajectory (reasoning, commands, tool output) and raises an alarm. On an alarm you stop the run, retry it, or hand the issue to another model.
The authors point out that a run can fail for three different reasons, and each one calls for a different action:
- Hard issue: most models fail it. Stopping or asking a human is the better move.
- Weak model: this model fails many issues. Switching to a stronger one is the better move.
- Bad run: the same model solves the same issue on other attempts. Only here does a retry help.
Their finding is that current trajectory-based predictors mostly can't tell these apart.
What they report
They used public trajectories on a widely used benchmark of 500 real repository issues, where each model attempts each issue once: 3,492 runs from seven models, 994 of them failures, plus outcomes for 23 models and a separate public corpus with repeated runs.
- A score that never sees the run wins. A reference that only knows how hard an issue is for other models reaches 0.92 pooled AUROC. The trajectory predictors reach 0.62–0.63.
- Pooled AUROC is mostly a between-task number. Of 494,109 failed/successful pairs in the test folds, 99.66% compare different issues. A score that recognizes hard issues can look excellent without reading a single step.
- Within an issue, knowing the model beats reading the run. Model identity alone reaches 0.72 within-issue AUROC, against 0.60–0.61 for the predictors. On "upset" pairs, where the overall weaker model won, the predictors score 0.42, below chance. They are reproducing the model ranking, not watching the run.
- What the trajectory adds is statistically zero. Adding a predictor's score on top of issue difficulty and model identity changes AUROC by an amount equivalent to zero in all 18 configurations they tested.
- The "switch models" story mostly dissolves. The best single model solves 76.8% of issues and some model solves 86.8%, a 10.0-point "oracle gap". That gap sits inside the range chance alone produces. 45% of failures (448 of 994) happen on 64 issues no model solves, so there is nothing to switch to. Picking a model per issue from its past runs gains +0.09 points on new runs; a router for new issues gains +0.41.
- The gain people attribute to routing comes from a second attempt. Escalating every failure to one fixed second model gains +6.91 points, and that assumes a perfect failure detector. The win is the extra try, not a better-suited model.
- Bad runs are real. In the repeated-run pilot (522 issues, at least four runs per model), outcome variation splits into 53% issue difficulty, 42% run-to-run variation, and only 4.6% model-issue specialization. So retries have real room to help, but 64–82% of predictor score variance lies between issues, which means the alarm mostly fires on the cases where a retry is less likely to work.
Scope, in their words: mainly one benchmark and one scaffold, trajectory-based predictors only, a repeated-run pilot with older models, and a router with shallow issue features. Not detecting specialization is not proof it is absent. Those are their numbers, not mine. I have not re-run the experiments, and I am deliberately not naming the models, the benchmark, or the prior systems the paper discusses.
The stake
Teams will read "our failure predictor has a good AUROC" as "our agent knows when a run is going wrong".
That is the same mistake as reading a leaderboard as a forecast. A pooled score mixes three things into one number: how hard the tasks were, how strong the model is, and what happened in this attempt. Only the third is something a run-time monitor can act on. If the number is carried by the first two, you have built an expensive difficulty estimate and wired it to a retry button.
The cost shows up in the action layer. A predictor that fires on hard issues triggers retries on exactly the failures a retry rarely fixes, and stays quiet on the bad runs a retry would rescue.
Opinion (one sentence)
Any claim that a monitor, router or retry rule makes an agent more reliable should be backed by repeated runs and run-blind baselines at equal budget, because a single-attempt leaderboard can't tell a hard task from a weak model from a bad run.
That is a preference with a limiter, not a benchmark result. If your agent runs short, cheap, deterministic tasks, single runs may be enough. Once runs are long, stochastic, and you act on an alarm mid-run, the cause of failure decides whether the action helps.
What a cause-aware agent eval reports (menu, not recipe)
| Column | Question it answers | Failure if skipped |
|---|---|---|
| Run-blind baselines | Does the monitor beat "how hard is this task" and "which model ran"? | A difficulty estimate sold as run monitoring |
| Within-task score | Can it rank two attempts at the same task? | Pooled score carried by task mix |
| Repeated runs | How often does the same model pass and fail the same task? | Bad runs invisible, luck read as skill |
| Chance reference for gaps | Is the "best of several models" gap bigger than noise? | Routing built on a coin-flip artifact |
| Equal-budget comparison | Does the strategy beat simply trying again with the same model? | A second attempt credited to cleverness |
| Action outcome | Did the triggered stop, switch or retry actually fix the failure? | An alarm graded on itself, not on what it changed |
None of that requires a private harness or new model. It needs repeated runs, two baselines that never look at the trajectory, and the discipline to grade the decision, not the alarm.
Why "the alarm was right" is not a pass
The detail I would steal first is the toy example the authors open with. If a model succeeds 90% of the time on easy issues and 10% on hard ones, a score that only knows which issues are hard reaches 0.90 pooled AUROC, yet it ranks two runs of the same issue at 0.50, a coin flip. A great-looking metric with zero information about the run.
That is the eval-as-contract problem in miniature. The metric was computed correctly. It was just answering a different question from the one the product needs answered at run time.
Where this connects to habits I already keep public
My public work is mostly retrieve-first and evaluation-as-contract systems, and the habit is the same: before a component gets credit, compare it against the dumbest baseline that could have produced the same number. In retrieval, that means checking whether a gain survives query-class slices. For agents, this paper names the baselines for run-time monitoring: task difficulty, model identity, and "just try again". A monitor that can't beat them isn't monitoring the run.
Steal this check
Pick one agent with a stop, retry or routing rule and add four things before your next release:
- Run each task more than once. Even a small repeated slice tells you how much of failure is the run.
- Score two run-blind baselines. Task difficulty and model identity, next to your monitor, on the same cases.
- Report within-task ranking. Can the monitor tell a good attempt from a bad one on the same task?
- Compare at equal attempts. Does your clever strategy beat a plain second try with the same model?
Then ask whether the monitor is still worth its cost once the baselines are on the same chart.
What I am not claiming
- I am not reporting this paper's results as my own, and I have not reproduced them.
- I am not ranking models, agents or vendors.
- I am not saying failure prediction is useless. The authors call for predictors that explain why a run fails, evaluated on repeated runs, in closed-loop, equal-budget settings.
- I am not publishing alarm thresholds or retry rules. Those depend on what a wasted run and a missed failure cost you.
The point
A failure alarm that mostly knows the task is hard is not watching the run. Run tasks more than once, put run-blind baselines on the same chart, and grade the action the alarm triggers, not the alarm itself.
Igor Eduardo builds retrieve-first and evaluation-as-contract systems. More at igoreduardo.com.
Top comments (1)
The run-blind baseline and within-task framing make the metric much more actionable than pooled AUROC. I’d add calibration and decision cost stratified by task class and retry budget: AUROC can rank well yet still trigger too many expensive retries at the chosen threshold. The distinction between a hard issue and a bad run is crucial for deciding whether to retry, switch models or ask for help.