The transcript is not the artifact

I gave a coding agent a genuinely awkward job and then spent longer checking its work than it spent doing the work. That ratio is the point of this post.

The task was a real one from a real repository: rename a field end to end, across three languages that all disagree about what a string is. PostgreSQL schema, an n8n workflow stored as JSON, and Python. The field also travels under a second name in one service's API, so the rename spanned two tokens, not one.

The agent did it in about twenty minutes. It got everything. And the count was still wrong three times — once by the agent, and twice by me while I was checking the agent.

That is what I want to write down, because the failures are all in the measurement, and measurement is the part everybody skips.

What the agent actually did

Credit where it is due, because the run was better than I expected.

It found the hard case on its own. The n8n workflow file stores its code-node bodies as escaped JSON strings — JavaScript inside a string inside JSON. Instead of reaching for a regex, it said so and changed approach:

"The workflow file stores the code-node bodies as escaped JSON strings, so I'm using a structured JSON edit for that one instead of hand-splicing a giant one-line string."

It then wrote a small script to parse, modify and re-serialise. Afterwards it checked its own work, found the first pass incomplete, and volunteered that without being asked:

"The first JSON edit removed the SQL select cleanly, but the embedded JavaScript string still had the fallback."

And when I asked for a passing type-check, it hit a sandboxed DNS failure, requested network access, was refused — and reported the failure precisely rather than inventing a green check:

"I could not show the requested passing TypeScript type-check because scripts/remotion-service has no local node_modules... I requested networked dependency install... but that approval was rejected, so I stopped there rather than fetching anything."

Then it noticed the empty package store its aborted install had left behind and asked to remove it, "so your worktree only contains the intended code changes."

None of that is junior behaviour. Which is exactly why the counting problem is worth taking seriously: when the narration is this good, you stop auditing.

Lie one: the agent's own diff badge

The agent displays a running total of what it has changed. At the end of the run it read 6 files, +13 −31.

git said 7 files, +16 −34.

It was wrong mid-run too — the badge read "4 files +10 −28" at a moment when the working tree held 6 files, +15 −33.

I do not know why, and it does not much matter. The lesson is structural: a self-reported diff is a claim, not evidence. The agent is describing its intentions; git is describing your disk. Only one of those is the artifact.

Lie two: I quoted a diffstat as if it counted something

This one is mine, and it is the more embarrassing of the two.

My first write-up said the agent had renamed 16 references. It had not. Sixteen was the insertion count from git diff --stat — a line-level number that happens to sit next to a plus sign. It counts lines added. It does not count anything semantic at all.

The actual figure, counted at HEAD across the changed files, was 33: twenty-three occurrences of the original name, plus ten of the second name the same field travels under in the render service.

The wrong number had already propagated into prose, an FAQ, an infographic and a machine-readable file before I caught it.

Never quote a diffstat number as a count of things. Insertions are not references, files changed are not modules touched, and +16 is not "sixteen of anything you care about". If you want to know how many references changed, count references — at a commit, with a grep you wrote for that purpose.

Lie three: I declared zero when it was forty-five

Same run, worse mistake.

I checked for stragglers and reported zero remaining across the repository. Forty-five survived.

The straggler grep was scoped: --include="*.sql|*.json|*.py". Perfectly sensible, since those were the three languages the rename touched. It silently excluded Markdown, and forty-five occurrences were sitting in four frozen planning documents.

Two failures stacked there, and they generalise past this example:

Scoping a search to where you expect the answer will confirm your expectation. Grep unscoped first, then narrow once you have seen the full set. A filter applied before you know the answer is not a filter, it is a hypothesis you have hidden from yourself.

A metric that spans two tokens needs a two-token check. The rename covered both names; my straggler check only looked for one for part of the sweep. If your definition of done spans several strings, every verification step has to span all of them, or "done" means "done for the string I happened to type".

What I do now

The method that survived this run, in the order I run it:

1. Measure with git, never with the badge. git diff --stat for shape, git status to confirm nothing landed outside what you approved.
2. Count semantically, at a commit. Grep for the actual tokens at HEAD, unscoped, and count every token the change spans.
3. Prove syntax with parsers, not eyeballs. On this run that meant JSON.parse on the workflow (28 nodes intact), node --check on the JavaScript extracted from inside that JSON, and py_compile on all four Python files. A diff can look perfect and still not parse.
4. Grep unscoped before you claim zero. Then narrow.

None of this is clever. All of it is the difference between a rename that is finished and a rename that is finished in the places you thought to look.

The finding that actually unsettled me

One more thing happened, and it is the reason I still read the whole diff.

The repository has a project instruction file — the document that tells future agent sessions how this codebase works. It contained an explicit rule: this legacy column is kept for back-compatibility, do not drop it.

The agent dropped it. Then it edited the instruction file to remove that rule, replacing it with a description of the new state.

The charitable reading is probably the correct one. I asked for an end-to-end rename, and leaving documentation asserting a constraint the code no longer honours would be its own kind of wrong. It listed the edit in its summary rather than hiding it.

But sit with the shape of it. The instruction file is the thing that constrains the next session. An agent that revises the rulebook to match what it just did leaves a repository where the rule never appears to have existed — and the next agent, reading a clean file, has no way to know a constraint was ever there. It happened with no approval prompt, because that file lives inside the workspace the agent was given.

I am not arguing it behaved badly. I am arguing that "files inside the workspace can be edited without asking" and "the workspace contains the rules" are two reasonable defaults that combine into something you would not choose deliberately.

The short version

The agent was more careful than I expected and more careful than I was. Every mistake in the paragraphs above that mattered to the published result was mine, and every one of them was a measurement error rather than a code error.

So: let it work, then check the disk rather than the story about the disk. And when you write the result down, count the thing you are claiming to have counted.

If you want the rest of that test — the workspace-escape run, what the approval gates caught, and the parts where the two agents I use genuinely differ — it is in the full Codex vs Claude Code comparison, including the parts where my own evidence is thinner than I would like.