Ahmad Dogar

Posted on Oct 25 Originally published at mahmadmanzoor.com

Make coding agents prove their work

#ai #claude #testing #devops

The most expensive sentence a coding agent writes is "Everything should work now." It is said with confidence, and sometimes it is even true. An agent's summary of its own work is the least reliable part of its output, because it describes what the agent meant to do, not what happened.

So on this site the rule is that every report arrives with evidence I can check without re-running anything: command output, numbers and screenshots. Most of the checks are ordinary tools that existed long before agents. The work is in making the agent run them every time, before it calls itself finished, and in refusing to merge when the evidence is missing.

Cheap and precise checks first, slow and visual ones last.

The order of checks

Cheapest and most precise first. A type error points at a line; a broken screenshot points at nothing in particular. For this site, a Vite, React and TypeScript app with Vercel functions, the sequence is:

npm run typecheck, which is tsc -b across the app, the API and the scripts.
npm run lint with oxlint, fast enough to run after every change.
npm run build, which also runs the prerender step, so a page that crashes during server rendering fails here rather than in production.
npm run budget, the bundle budget described below.
Screenshots of the pages the change touched, at a desktop and a phone width.

The GitHub Actions workflow runs the first four on every push and pull request. Agents run all five locally before reporting, because a red CI run after I have merged is a much slower loop than a red run inside the agent's own worktree.

What a report has to contain

The same format for every agent, every task.

Every agent gets the same report format. The files it changed. The actual output of each command, pasted, not paraphrased. For tests, the counts, because "tests pass" from an agent that skipped the suite looks exactly like the real thing. The budget numbers. Paths to the screenshots.

And a last section, which I read first: what it did not check. An agent that writes "I did not test the signed-in view" has saved me a bug. An agent that leaves the section empty after changing the editor has told me where to look. Asking for it explicitly matters, because agents rarely volunteer the limits of their own testing.

One honest gap: this site has no unit test suite. Its checks are types, lint, the build with its prerender step, the budget and screenshots, and for a mostly static portfolio that has been enough. On projects that do have tests, the counts go in the report, and a change that adds behaviour without adding a test gets sent back with a request for one.

A Stop hook so done means green

Telling an agent to run the checks works most of the time. To make the fast ones unconditional I use a Stop hook: a script Claude Code runs whenever the agent tries to end its turn. If the script exits with code 2, the agent does not stop. It receives the script's stderr as feedback and keeps working.

The agent cannot finish while the fast checks are red.

.claude/settings.json

{
  "hooks": {
    "Stop": [
      {
        "hooks": [
          { "type": "command", "command": "${CLAUDE_PROJECT_DIR}/.claude/hooks/verify.sh" }
        ]
      }
    ]
  }
}

.claude/hooks/verify.sh

#!/usr/bin/env bash
# Runs when the agent tries to finish. Exit 2 sends the errors back and keeps it working.
input=$(cat)
# Already continuing because of this hook? Let it stop instead of looping.
if [ "$(printf '%s' "$input" | jq -r '.stop_hook_active')" = "true" ]; then
  exit 0
fi
cd "$CLAUDE_PROJECT_DIR" || exit 0
if ! out=$(npm run --silent typecheck 2>&1); then
  printf 'Typecheck failed. Fix these before finishing:\n%s\n' "$(printf '%s' "$out" | tail -40)" >&2
  exit 2
fi
if ! out=$(npm run --silent lint 2>&1); then
  printf 'Lint failed. Fix these before finishing:\n%s\n' "$(printf '%s' "$out" | tail -40)" >&2
  exit 2
fi
exit 0

Two choices in that script are worth explaining. It runs only the fast checks; a full build in a Stop hook would make every small answer slow, so the build and the budget stay in the instructions and in CI. And it gives up after one forced retry, via the stop_hook_active flag, instead of trapping the agent on an error it cannot fix. Claude Code has its own cap on consecutive continuations, but I would rather get a clear failure in the report than a long loop.

Bundle budgets in CI

Type checks catch broken code. They do not catch heavy code, and agents add weight casually: a date library for one format call, a whole icon set for three icons. This site has a 3D view built on three.js, which is large, so from the start it had a budget script, scripts/check-budget.mjs, that runs after the build in CI.

The numbers print on every build, so they land in every report.

scripts/check-budget.mjs (excerpt)

const html = readFileSync(join(dist, 'index.html'), 'utf8')
const refs = [...html.matchAll(/(?:src|href)="\/(assets\/[^"]+\.(?:js|css))"/g)].map((m) => m[1])
let initial = 0
for (const ref of new Set(refs)) {
  const buf = readFileSync(join(dist, ref))
  initial += gzipSync(buf).length
  if (/three|fiber|drei/i.test(ref)) errors.push(`3D code is in the initial payload: ${ref}`)
}
console.log(`initial JS+CSS (gzip): ${fmt(initial)} / ${fmt(BUDGET.initialJsGzip)}`)
if (initial > BUDGET.initialJsGzip) errors.push('initial payload over budget')

It gzips every JS and CSS file that index.html loads up front and fails the build above 250 KB. It also fails if three.js appears in the initial chunks at all, by file name or by the string WebGLRenderer, because the 3D view has to stay lazy-loaded. Every image in public/shots must be webp, avif or svg and under 120 KB, which is one reason the diagrams in this post are SVG.

Today the initial payload is 85.1 KB of the 250 KB budget. The useful part is not the headroom. It is that the number is printed on every run, so it appears in every agent report, and a jump of a few dozen KB from one change is visible before merge instead of in a slow page months later.

Show me the screenshot

Two widths catch most layout breaks.

For anything visual, a passing build proves very little. CSS can be valid and still put the search box underneath the navbar. So agents take screenshots with Playwright, which this repo already uses to capture the project images in its gallery, at two sizes: 1440 by 900 for desktop, and 390 wide at 2x for a phone. Between them, those two widths catch most of the layout breaks worth catching.

The script is short enough that agents can run it without thinking about it, and its output is just file paths, which go straight into the report.

shot.mjs

import { mkdirSync } from 'node:fs'
import { chromium } from 'playwright'

const url = process.argv[2] ?? 'http://localhost:4173/blog'
const sizes = {
  desktop: { viewport: { width: 1440, height: 900 }, deviceScaleFactor: 1 },
  phone: { viewport: { width: 390, height: 844 }, deviceScaleFactor: 2 },
}

mkdirSync('shots', { recursive: true })
const browser = await chromium.launch()
for (const [name, options] of Object.entries(sizes)) {
  const page = await browser.newPage(options)
  await page.goto(url, { waitUntil: 'networkidle' })
  await page.screenshot({ path: `shots/${name}.png`, fullPage: true })
  console.log(`shots/${name}.png`)
  await page.close()
}
await browser.close()

Then I open the images myself before merging. Not because the agent cannot look at a PNG (it can, and I ask it to check its own screenshots against the request first), but because the agent decided what "looks right" means when it wrote the code. A look from someone who did not write it is the whole point of review, and it takes seconds per image.

Evidence over trust

None of this assumes the agent is careless. It assumes what we assume of any engineer: "done" should come with something checkable. Agents simply make that cheap to ask for, since five commands and two screenshots cost them a minute. The habit that matters is on my side of the review. I do not merge a change whose report has no evidence in it, however confident the summary sounds. If the evidence is missing, the reply is short: run the checks and send the output. That reply costs me ten seconds and saves the much longer job of finding out myself.

1. Order the checks

Typecheck, lint, build, budget, screenshots. Cheap and precise first.

2. Fix the report format

Files, pasted output, test counts, budget numbers, screenshot paths, and what was not checked.

3. Add a Stop hook for the fast checks

Exit 2 keeps the agent working until typecheck and lint pass.

4. Put size in CI

A gzip budget and a per-image limit, printed on every run.

5. Look at the screenshots

Two widths, opened by someone who did not write the code, before every merge.

Claude Code hooks reference: Stop hooks, stop_hook_active and what exit code 2 does for each hook event.

Originally published on my site. More posts and projects at mahmadmanzoor.com.

DEV Community