When an AI agent says it's done and it isn't

Yugesh Jha — Sat, 03 Oct 2026 16:11:19 +0000

The edit is usually fine. The claim about the edit is the problem, and it is a harder one.

You ask for a change across four files. The agent works for a few minutes, edits them, and says the change is complete. It is not. Nothing was run, one caller in a fifth file no longer compiles, and you find out in CI or in review or from somebody else.

The frustrating part is that the edit was usually reasonable. What was wrong was the sentence at the end.

Why it happens

A language model produces the most plausible continuation of what came before. After a sequence of edits, the most plausible continuation is a summary saying the work is finished, because that is how the thousands of examples it learned from ended. Nothing in that process checks.

So "done" is not a claim the model is making about your repository. It is the shape of a closing paragraph. The only thing that turns it into a real claim is a tool that actually ran something, and then reported what came back.

What it costs, and why it is worse than a bad edit

A wrong edit you catch in the diff costs a minute. A wrong claim costs you the assumption you were reviewing under.

This compounds. The reason agents are useful is that you stop reading every line and start reading the summary. Once the summary is unreliable you have to go back to reading every line, and at that point the agent has moved the work rather than done it.

How to tell the difference

The question to ask of any tool is simple: did it run anything, and can you see what came back?

Did it execute your tests, or describe executing them? Those look similar in a summary and are not the same event.
Is the raw output on screen? A tool that ran your suite has output. A tool that did not will paraphrase.
What does it do when it cannot verify? The honest behaviour is to say so. A tool without that check tends to report success anyway.
Did it see the file it broke? An agent reading only your open tabs cannot verify a change it cannot observe.

A test you can run in ten minutes

Take a repository you know well with a test suite that passes. Ask the agent for a change touching at least three files. Then, before you look at the diff, break something it just wrote by hand and ask it to continue.

A tool that runs your suite notices. A tool that does not will keep going and tell you everything is fine. That single exercise tells you more than any comparison page, including ours.

What to do if your current tool does this

Make the tests the specification. An agent iterating against a suite is only as good as the suite. Break the behaviour a test guards and confirm the test goes red, because a test that passes either way actively steers an agent wrong.
Shrink the unit of change. Review fatigue is what turns an unverified claim into a merged defect.
Ask for the commands. If the tool can show what it ran, make it. If it cannot, treat every summary as a draft.

Where AstraCode fits

This is the problem we built AstraCode around. It plans the change, makes it, runs your tests, reads the output, fixes what it broke, and runs them again, with what it ran on screen rather than summarised. When it cannot verify something it says so instead of reporting success. There is a free tier with no card, and the ten-minute exercise above is a fair way to judge it, or any other tool.

Originally published on the AstraCode blog.

Cost per token is the wrong number for coding agents. Here's what we measured instead.

Yugesh Jha — Sat, 03 Oct 2026 14:38:05 +0000

We build AstraCode, an AI code editor whose agent has to prove its own work. Model calls are the largest line on our bill, so in September we stopped guessing and measured: eleven models, the same set of real coding tasks, each run more than once, scored on pass rate and on cost per finished task.

Step count beats token price

The cheapest model per token is not the cheapest model per task. On one task a model took 130 agent steps; another finished in 32. Every step re-sends context, so the step count multiplies everything else.

It is not as simple as "expensive models wander more" either. Some of the priciest models took fewer steps than the cheap one. We wrote that rule down, then had to cross it out.

Takeaway: log steps per task alongside tokens. A model that is 3x cheaper per token and takes 4x the steps is a more expensive model.

Routing by prompt doesn't work

The obvious optimisation is a router: read the request, send easy ones to a small model and hard ones to a big one. We tried three versions of it.

A per-step router came out 0.4% worse than not routing at all.
A classifier that predicted difficulty from the prompt reached a rank correlation of about 0.75 and still lost.
Even an oracle with perfect hindsight picked the small model on every task, so there was no headroom to win.

The reason is variance. The same small model passed one task 67% of the time on the identical prompt. Whether a run succeeds is decided during the attempt, not by the prompt, so no prompt classifier can see it.

Detection does work

What does work is checking the result:

Run the small model.
Check its work. Run the tests, then undo the change and confirm those tests fail without it, so a test that passes either way doesn't count as proof.
Only if a check fails, hand the task to a stronger model.

Pass rate went from 0.64 to about 1.00, at roughly $0.20 per task against $0.03 for the small model alone.

These are our tasks and our harness, so treat them as one data point, not a ranking. Every model we tested is good at something.

What we changed

AstraCode no longer asks you to pick a model. AstraOne starts with the cheaper model and escalates when a check fails. The checks are the same ones you'd want anyway: tests that are shown to fail without the change, a diff you review hunk by hunk, and a checkpoint before every turn.

If you're building on LLMs, two habits are worth more than any model choice:

measure cost per finished task, not per token;
run every task several times before you trust a number.

AstraCode is free to start, no card: https://astracode.io

DEV Community: Yugesh Jha