DEV Community

Aamer Mihaysi
Aamer Mihaysi

Posted on

The critic belongs in the request path, not the dashboard

Last week I watched an agent burn four steps cleaning up after a step that had already failed. The tool call returned a 200 with an empty list — technically a success, obviously wrong — and nothing in the loop could tell the difference. The episode got scored a minute later, at the end, in a report I opened on Friday.

The failure was visible at step 11. I found out at step 40. That gap is the thing I actually want to shrink, and it's not an eval problem. It's a latency problem.

Which is why AgentRoboCritic caught me the wrong way round. The paper is robotics — frontier agents driving robots directly, no separate policy network in the middle — and its headline is a fine-tuned 30B critic that lifts failure-mode classification from 32.58% to 74.24% over the un-tuned baseline. Everyone I've seen link it reads that as an evaluation result. I read it as a cost-and-latency result. A 30B classifier chewing on short inputs is cheap and fast enough to run on every step, synchronously, inside the loop. That's the difference between a report and a component.

The uncomfortable number is the baseline

32.58%. A frontier model, asked to judge whether a manipulation step did what it was supposed to do, gets it wrong two times out of three.

I don't think that's because the task is deep. I think it's because judging is narrow, repetitive, and boring, and it looks nothing like the reasoning the big models were tuned to be good at. Which has a consequence beyond robotics: whatever judgment your domain needs — did this refund go through correctly, is this SQL semantically the query I asked for, did the scraper get the right page — you are going to have to build it. It won't come bundled with the next model release, and it won't be a prompt.

Small, specialized, yours. That's the shape of it.

What changes when it sits inline

My current supervisor fires on one condition: the same tool call failing three times in a row. It catches loops. That's it. It has no opinion on "succeeded, but on the wrong object," which — in my traces, anyway — is the failure mode that actually costs me money.

A critic that runs inline changes the menu. You can abort the step instead of watching the next four clean up after it. You can retry with a hint. You can escalate to the bigger model for one step instead of the whole episode. You can just stop and page someone. None of that requires a good eval dashboard. All of it requires a judgment that lands in under a second and costs less than the step it's judging.

But — and this is the part the paper doesn't claim, so I'm on my own here — a critic that labels a failure and a critic that handles one are different objects. 74.24% classification accuracy does not tell you the intervention was correct. Being right about "that didn't work" is not the same as being right about "do this instead." I'd want to measure intervention quality separately, and I haven't.

Three things that will bite

Latency in the critical path. Every step now waits on a second model. If your step was 400ms and your critic is 300ms, you've nearly doubled the loop and users feel it. Batch it, run it async against the next step's plan, or accept a slower agent. Pick deliberately.

False positives become a failure mode. An agent that gives up on steps that were fine is worse than one that plows ahead. Hysteresis helps: don't abort on one disagreeing verdict, require two, or require the critic to name what it expected and didn't see. The critic's error rate is now your agent's error rate, and you inherited it silently.

You just shipped a second model. It has its own version, its own drift, and its own way of failing that looks like the agent getting dumber. Version the critic's prompt and weights alongside the agent, and log its verdicts with the same seriousness. Otherwise the first time it degrades you'll spend a day tuning tool descriptions.

The number I'd actually put on the wall

Not classification accuracy. Mean steps between a failure occurring and something noticing it. Mine is currently unbounded — the episode ends, then a report gets written. I'd like it to be one.

That's a deployability metric, which is why I care. A judge you can only afford to run at the end of an episode is a historian. A judge you can afford to run every step is a supervisor, and a supervisor can still be wrong but at least it's in the room.

I haven't built the inline version yet — this is me talking myself into it while the traces are still fresh. If it turns out the 30B critic is too slow or too flaky to sit in the hot path, I'll find out the expensive way, like usual.

Top comments (0)