DEV Community: Ahmad Dogar

Structured outputs that do not break: schemas, Zod and a repair retry

Ahmad Dogar — Sun, 25 Oct 2026 19:21:21 +0000

A few years ago the hard part of getting JSON from a language model was getting JSON at all. You asked nicely, you got a code fence with a trailing comma, and you wrote a regex to dig the object out. That problem is mostly solved. Current APIs can constrain the output to a JSON schema, so the answer always parses and always has the fields you asked for.

The problem that is left is quieter. When I worked on extracting roof measurements from PDF reports, the worst failures were never broken JSON. They were valid objects with the wrong numbers in them: a roof in squares read as square feet, a total counted twice. The schema was happy every time.

So I think of structured output as layers. Each one catches a different kind of failure, and the last one is a person.

Each layer catches what the one before it cannot.

The examples use the Anthropic TypeScript SDK and Zod. The teaching example is a small extractor that reads the text of a roof report and returns its measurements.

Layer one: let the API enforce the shape

There are two ways to get structured data out of Claude, and I pick between them by asking what the object is for.

Answer or action decides the mechanism.

If the object is the answer, use output_config.format and let the SDK parse it. The model has one job and the response is the data. If the object is the argument to an action, and the model might choose between several actions or none, define a tool with strict: true. The arguments are then guaranteed to match the tool's schema, and you still get to decide whether to run it.

For an extractor the object is the answer. I define the shape in Zod, keeping this schema to plain types only, because it is turned into a JSON schema for the API.

extract/schema.ts

import { z } from "zod";

export const Facet = z.object({
  pitch: z.string().describe('rise over a 12 inch run, written like "6/12"'),
  areaSqFt: z.number(),
});

export const ReportShape = z.object({
  sourceUnit: z.enum(["sq_ft", "squares"]).describe("the unit the report uses for area"),
  totalAreaSqFt: z.number().nullable().describe("converted to square feet; null if the report has no total"),
  facets: z.array(Facet),
  ridgeFt: z.number().nullable(),
  valleyFt: z.number().nullable(),
});
export type Report = z.infer<typeof ReportShape>;

The describe() strings end up in the schema the model sees. They are the cheapest prompt engineering there is: one line, next to the field it explains.

Layer two: shape is not meaning

A schema cannot say "the facet areas add up to the total" or "this number must appear somewhere in the report". Zod can, with superRefine. I build the checks as a function of the source text, so a check can look back at what the model was reading.

Valid JSON, wrong roof.

The check I would keep if I could keep only one is the grounding check. Every number the model claims to have read should be findable in the source, either as written or in the other unit. It catches invented values, and it catches the classic unit slip: the report says 32.4 squares, the model correctly says the unit is squares, then forgets to convert and returns a total of 32.4. Converted back, that is 0.324 squares, which is nowhere in the text, so the check fails.

extract/checks.ts

import { ReportShape } from "./schema";

const numbersIn = (text: string) =>
  new Set((text.replace(/(\d),(\d{3})/g, "$1$2").match(/\d+(?:\.\d+)?/g) ?? []).map(Number));

export function reportChecks(source: string) {
  const seen = numbersIn(source);
  // totals are stored in square feet; a report in squares shows them divided by 100
  const grounded = (n: number, unit: "sq_ft" | "squares") =>
    unit === "squares" ? seen.has(Math.round(n) / 100) : seen.has(n);

  return ReportShape.superRefine((r, ctx) => {
    if (r.totalAreaSqFt !== null && !grounded(r.totalAreaSqFt, r.sourceUnit))
      ctx.addIssue({ code: "custom", path: ["totalAreaSqFt"], message: `${r.totalAreaSqFt} does not appear in the report (unit: ${r.sourceUnit})` });

    if (r.totalAreaSqFt !== null && r.facets.length > 0) {
      const sum = r.facets.reduce((s, f) => s + f.areaSqFt, 0);
      if (Math.abs(sum - r.totalAreaSqFt) > r.totalAreaSqFt * 0.02)
        ctx.addIssue({ code: "custom", path: ["facets"], message: `facet areas add up to ${sum}, but totalAreaSqFt is ${r.totalAreaSqFt}` });
    }

    r.facets.forEach((f, i) => {
      if (!/^\d{1,2}\/12$/.test(f.pitch))
        ctx.addIssue({ code: "custom", path: ["facets", i, "pitch"], message: `pitch "${f.pitch}" is not written as N/12` });
    });
  });
}

Notice the error messages. They are written for the model as much as for me. Each one says which field is wrong, what value was there and what the rule is. That pays off in the next layer.

Before you validate, check why the model stopped

There are three ways to get a response that should not reach your checks at all. A stop_reason of max_tokens means the output was cut off. A refusal means a safety classifier declined the request. And parsed_output can be null. None of these throw an exception. They are successful HTTP responses with nothing useful inside, so the code has to look for them.

Check the stop reason before you read the content.

A truncated extraction is not worth repairing. Either the output budget is too small for the document, which is a bug to fix, or the model went off on a long tangent. Neither gets better by asking again with the same settings.

Layer three: one repair retry, with the errors

When validation fails, retrying the same request blind mostly reproduces the same mistake. Telling the model exactly what failed works much better. The repair request contains the source, the previous answer and the list of failed checks, and asks for the full corrected object.

One retry with specific errors, then stop.

I build each attempt as a fresh single request rather than a growing conversation. The source text is the first block and carries a cache breakpoint, so the second attempt reads it from the prompt cache instead of paying for it again.

extract/extract.ts

import Anthropic from "@anthropic-ai/sdk";
import { zodOutputFormat } from "@anthropic-ai/sdk/helpers/zod";
import { ReportShape, type Report } from "./schema";
import { reportChecks } from "./checks";

const client = new Anthropic();
const SYSTEM = "You extract roof measurements from report text. Use only values that appear in the report. If a value is missing, return null instead of estimating it.";

export type Extraction =
  | { ok: true; data: Report; attempts: number }
  | { ok: false; reason: "refusal" | "truncated" | "empty" | "invalid"; issues: string[]; raw: unknown };

export async function extractReport(source: string): Promise<Extraction> {
  let previous: unknown = null;
  let issues: string[] = [];

  for (let attempt = 1; attempt <= 2; attempt++) {
    const ask = previous === null
      ? "Extract the measurements from the report above."
      : `Your previous answer was:\n${JSON.stringify(previous)}\n\nIt failed these checks:\n${issues.map((i) => `- ${i}`).join("\n")}\n\nRe-read the report and return the full corrected object.`;

    const res = await client.messages.parse({
      model: "claude-sonnet-5-5",
      max_tokens: 16000,
      system: SYSTEM,
      messages: [{
        role: "user",
        content: [
          { type: "text", text: `<report>\n${source}\n</report>`, cache_control: { type: "ephemeral" } },
          { type: "text", text: ask },
        ],
      }],
      output_config: { format: zodOutputFormat(ReportShape) },
    });

    if (res.stop_reason === "refusal") return { ok: false, reason: "refusal", issues: [], raw: null };
    if (res.stop_reason === "max_tokens") return { ok: false, reason: "truncated", issues: [], raw: null };
    if (!res.parsed_output) return { ok: false, reason: "empty", issues: [], raw: null };

    const checked = reportChecks(source).safeParse(res.parsed_output);
    if (checked.success) return { ok: true, data: checked.data, attempts: attempt };

    previous = res.parsed_output;
    issues = checked.error.issues.map((i) => `${i.path.join(".") || "(root)"}: ${i.message}`);
  }
  return { ok: false, reason: "invalid", issues, raw: previous };
}

Why only one retry? Because the second attempt is where the information is. If the model fixes the problem when told exactly what is wrong, the first answer was a slip. If it fails the same check again with the error in front of it, the input is unusual, the report is ambiguous, or the check is wrong. A third attempt rarely changes that, and each one costs a full request. Two failed attempts on the same field is a signal, and the right response to that signal is a person.

I log every repair, including the ones that succeed. A check that triggers a repair on a large share of documents means the prompt or the field description is wrong, and that belongs in the eval set, not in a retry loop.

Layer four: a human, on purpose

The human fallback is not an error path. It is part of the design, and it deserves a real screen. These are the cases I send to review:

the repair retry failed, with the issue list attached
a refusal or a truncated response
values that pass every check but sit outside business limits, like a house with 40,000 square feet of roof
two sources that disagree, such as a total in the summary table that differs from the detail table
any field where a wrong value costs money and the model was not reading it straight off the page

The reviewer sees the source, the output and the reason in one place.

The reviewer sees the relevant part of the source, the extracted fields, and the exact check that failed. They can accept, correct or reject. Every correction is saved with the input, and that pair goes straight into the golden set, which I wrote about in the post on evals. Over time the cases that reach a human become the cases the harness tests most carefully.

The short version

Pick the mechanism by purpose

Use output_config.format when the object is the answer, and a strict tool when the object is the argument of an action.

Keep the API schema plain, put meaning in Zod

Describe fields in the schema and check sums, formats and grounding with superRefine.

Check the stop reason before the content

Truncation, refusal and an empty parse are successful responses that must not reach your checks.

Repair once, with specific errors

Send the source, the previous answer and the failed checks. Cache the source so the retry is cheap.

Send the rest to a person and learn from it

Show the source next to the output, record the correction, and add it to the eval set.

Schemas made JSON boring, which is good. The interesting work moved one layer up, to deciding what "correct" means for your data and writing it down as code the model can read when it gets it wrong.

Originally published on my site. More posts and projects at mahmadmanzoor.com.

Evals before prompts: a small eval harness for an LLM feature

Ahmad Dogar — Sun, 25 Oct 2026 18:49:10 +0000

Every LLM feature I have built started the same way: a prompt that worked on the three examples I had open. On the roof report extraction work, the examples I had open turned out to be the easy ones. Real uploads failed in ways I had not pictured, and the only thing that stopped me from fixing one failure and quietly causing another was turning each failure into a test case.

That is all an eval harness is. A list of inputs you care about, a way to decide whether each output is good, and a number at the end. I now build it before I write the second version of any prompt, because without it every prompt edit is a guess that happens to look right on the case in front of you.

The whole harness is one loop. Change, run, compare, decide.

The harness below is small on purpose: a JSONL file, two kinds of checks, a script and a CI job. The examples use the Anthropic TypeScript SDK and a made-up feature, but nothing about the shape is tied to one provider.

The feature under test

To keep it concrete, picture an inbox helper for a roofing contractor. An email comes in, and the feature returns three things: a category (quote request, storm damage, billing, scheduling or spam), an urgency, and a short draft reply that a person reviews before sending.

Two of those outputs have one right answer. The third is free text where many answers are fine and a few are dangerous. That mix is typical, and it is why the harness needs two kinds of checks.

The golden set

The golden set is a file of cases. Each case has the input, the fields that have one right answer, and a few notes in plain English about what a good reply must and must not do. I add tags so I can slice the score later, and I mark a handful of cases critical: the ones where a wrong answer costs real money or trust.

One line of JSONL per case.

Where do cases come from? Real inputs first, with names and addresses replaced. Then every bug report, word for word. Then the edge cases you can think of: an empty body, a forwarded thread, an email in two languages, a polite message that is actually a complaint. Thirty to fifty cases is enough to start. The set grows every time the feature fails in production, and that habit matters more than the starting size.

I keep the notes short and checkable. "Must not promise a visit date" is a good note. "Should sound professional" is not, because nobody can grade it the same way twice.

evals/golden.jsonl

{"id":"storm-017","tags":["storm","critical"],"input":{"from":"homeowner@example.com","subject":"Shingles in the yard","body":"Wind last night took a bunch of shingles off the back slope. Water is coming through the bedroom ceiling. Can someone come today?"},"expect":{"category":"storm_damage","urgency":"high"},"notes":"Must acknowledge the leak. Must not promise a time or a price. Should ask for a phone number if none is given."}
{"id":"billing-004","tags":["billing"],"input":{"from":"ap@example.org","subject":"Invoice 2291","body":"We were charged twice for the gutter work. Please advise."},"expect":{"category":"billing","urgency":"normal"},"notes":"Must not admit or deny the double charge. Should say the office will check the invoice."}

evals/types.ts

import { z } from "zod";

export const Case = z.object({
  id: z.string(),
  tags: z.array(z.string()),
  input: z.object({ from: z.string(), subject: z.string(), body: z.string() }),
  expect: z.object({
    category: z.enum(["quote_request", "storm_damage", "billing", "scheduling", "spam"]),
    urgency: z.enum(["low", "normal", "high"]),
  }),
  notes: z.string(),
});
export type Case = z.infer<typeof Case>;

Exact checks first

Anything that can be checked with code should be checked with code. It is free, it is instant and it never disagrees with itself. For this feature that means the two enum fields, plus a few rules about the reply that a regex can enforce: no dollar amounts, no links, a length cap.

evals/checks.ts

import type { Case } from "./types";
import type { TriageOutput } from "../src/triage";

export function exactChecks(c: Case, out: TriageOutput) {
  return [
    { name: "category", pass: out.category === c.expect.category },
    { name: "urgency", pass: out.urgency === c.expect.urgency },
    { name: "no_prices", pass: !/[$£€]\s?\d/.test(out.reply) },
    { name: "no_links", pass: !/https?:\/\//i.test(out.reply) },
    { name: "short", pass: out.reply.split(/\s+/).length <= 120 },
  ];
}

Two kinds of checks, run in that order.

Exact checks also run first and short-circuit. If the category is wrong, the case has failed, and there is no reason to pay a second model to read the reply. On a large set that cuts judge calls noticeably, and it keeps the judge focused on the part only a model can grade.

An LLM judge with a rubric it cannot wriggle out of

The draft reply needs judgement, so a model grades it. The mistakes I see most with LLM judges are vague questions ("rate this reply from 1 to 10") and judges that see too little context. A score of 7 tells you nothing, and two runs will give you 6 and 8 for the same text.

What works better for me is a short list of named criteria, each pass or fail, each with a one-sentence reason. The judge sees the original email, the case notes and the draft. I use structured outputs so the verdict always parses, and I use a model at least as strong as the one being graded.

evals/judge.ts

import Anthropic from "@anthropic-ai/sdk";
import { z } from "zod";
import { zodOutputFormat } from "@anthropic-ai/sdk/helpers/zod";
import type { Case } from "./types";

const client = new Anthropic();

const Verdict = z.object({
  criteria: z.array(
    z.object({
      id: z.enum(["answers_the_ask", "no_invented_facts", "follows_notes", "tone"]),
      reason: z.string(),
      pass: z.boolean(),
    }),
  ),
});

const RUBRIC = `You grade a draft reply written for a roofing contractor's inbox.
Grade every criterion separately as pass or fail. There is no partial credit.
- answers_the_ask: the reply responds to what the sender actually asked.
- no_invented_facts: no prices, dates, warranties or names that are not in the email.
- follows_notes: the reply satisfies every point in <case_notes>.
- tone: plain and polite, no exclamation marks, no sales pressure.
Write the reason before deciding pass or fail.`;

export async function judge(c: Case, reply: string) {
  const res = await client.messages.parse({
    model: "claude-opus-5-5",
    max_tokens: 16000,
    system: RUBRIC,
    messages: [
      {
        role: "user",
        content: `<email>\n${c.input.subject}\n\n${c.input.body}\n</email>\n<case_notes>\n${c.notes}\n</case_notes>\n<draft_reply>\n${reply}\n</draft_reply>`,
      },
    ],
    output_config: { format: zodOutputFormat(Verdict) },
  });
  if (!res.parsed_output) throw new Error(`judge gave no verdict: ${res.stop_reason}`);
  return res.parsed_output.criteria;
}

A judge is itself a model output, so I check it before I trust it. I grade twenty cases by hand, run the judge on the same twenty, and read every disagreement. Usually the fix is in the rubric: a criterion that mixed two ideas, or a word like "concise" that I understood one way and the judge another. When the judge and I agree on almost all of them, I stop checking every run and only spot-check failures.

Named pass/fail criteria are easier to agree on than a 1 to 10 score.

Keep the rubric short. Four or five criteria per feature is plenty. If you need more, the feature is probably doing two jobs.

Running it

The run script reads the cases, calls the real feature, runs the checks and writes a result file. Two details matter.

First, outputs vary from run to run. Current Claude models do not accept a temperature setting at all, and even when they did, zero was never fully deterministic. So each case runs three times and gets a pass rate. A case that passes two times out of three is telling you something different from one that never passes, and you want to see both.

Second, concurrency. Firing every case at once hits rate limits; running them one by one is slow. A small worker pool is enough.

evals/run.ts

import { readFileSync, writeFileSync } from "node:fs";
import { Case } from "./types";
import { exactChecks } from "./checks";
import { judge } from "./judge";
import { triage } from "../src/triage";

const RUNS = 3;
const WORKERS = 6;
const cases = readFileSync("evals/golden.jsonl", "utf8")
  .trim().split("\n").map((line) => Case.parse(JSON.parse(line)));

async function runOnce(c: Case) {
  const out = await triage(c.input);
  const exact = exactChecks(c, out);
  if (exact.some((k) => !k.pass)) return { pass: false, failed: exact.filter((k) => !k.pass).map((k) => k.name) };
  const verdict = await judge(c, out.reply);
  const failed = verdict.filter((v) => !v.pass).map((v) => v.id);
  return { pass: failed.length === 0, failed };
}

async function scoreCase(c: Case) {
  const runs = [];
  for (let i = 0; i < RUNS; i++) runs.push(await runOnce(c));
  return { id: c.id, tags: c.tags, passRate: runs.filter((r) => r.pass).length / RUNS, failed: runs.flatMap((r) => r.failed) };
}

const queue = [...cases];
const results: Awaited<ReturnType<typeof scoreCase>>[] = [];
await Promise.all(
  Array.from({ length: WORKERS }, async () => {
    for (let c = queue.shift(); c; c = queue.shift()) results.push(await scoreCase(c));
  }),
);

const score = (100 * results.reduce((s, r) => s + r.passRate, 0)) / results.length;
writeFileSync("evals/results/latest.json", JSON.stringify({ score, cases: results }, null, 2));
console.log(`score ${score.toFixed(1)} over ${results.length} cases`);

In CI on every prompt change

Prompts live in files in the repository, not in a database or a dashboard, so a prompt change is a pull request like any other. The CI job runs only when prompts, the LLM code or the evals change, because a full run costs real tokens.

The gate only runs when something that changes model behaviour changes.

The comparison has two rules. The overall score may not drop by more than two points, which leaves room for run-to-run noise. And every case tagged critical must pass on all three runs. If I could keep only one of the two, I would keep the second: an average can hide one very bad regression behind several small wins.

.github/workflows/evals.yml

name: evals
on:
  pull_request:
    paths: ["prompts/**", "src/llm/**", "evals/**"]
jobs:
  eval:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: 22
      - run: npm ci
      - run: npx tsx evals/run.ts
        env:
          ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
      - run: npx tsx evals/compare.ts

evals/compare.ts

import { readFileSync } from "node:fs";

const base = JSON.parse(readFileSync("evals/baseline.json", "utf8"));
const now = JSON.parse(readFileSync("evals/results/latest.json", "utf8"));

const drop = base.score - now.score;
const critical = now.cases.filter((c: any) => c.tags.includes("critical") && c.passRate < 1);

console.log(`baseline ${base.score.toFixed(1)}, now ${now.score.toFixed(1)}`);
for (const c of critical) console.log(`critical case failed: ${c.id} (${c.failed.join(", ")})`);
if (drop > 2 || critical.length > 0) process.exit(1);

Two practical notes. Pull requests from forks do not get repository secrets on GitHub, so this job will not run for outside contributors without extra setup. And the baseline file is updated in the same pull request that improves the score, so reviewers see the number move next to the prompt diff that moved it.

Tracking the score over time

A single run tells you whether a change is safe. A history tells you whether the feature is getting better. After each run on the main branch I append one line to a JSONL file: date, commit, a hash of the prompt files, the model id, the score and what the run cost.

Illustrative data. The shape is what matters: dips line up with commits.

The model id in that line earns its place on upgrade day. When a new model comes out I change one constant, run the same set, and compare. I do not have to argue about whether the new model "feels better". The cost column matters too: a harness that runs a few hundred model calls per pull request is cheap next to a bad prompt in production, but it is not free, and seeing the number keeps me from adding runs without thinking. For the nightly full run I send the calls through the batch API at half price, which I cover in the post on inference cost.

Where to start

Write ten cases today

Take real inputs, strip personal details, and write down the right answer and one or two notes for each.

Check with code what code can check

Enums, formats, forbidden strings, length. Run these first and stop on failure.

Add a judge with named pass/fail criteria

Four or five criteria, a reason before each verdict, structured output, and a model at least as strong as the one being graded.

Calibrate the judge once

Grade twenty cases yourself and fix the rubric until you and the judge agree.

Gate pull requests and keep a history

Fail on a drop in the average or any failed critical case, and log the score, model and cost on every main-branch run.

None of this is clever. It is the same discipline as unit tests, applied to a part of the system that cannot be unit tested in the usual way. The difference it makes is that a prompt change stops being an opinion and becomes a diff with a number next to it.

Originally published on my site. More posts and projects at mahmadmanzoor.com.

Make coding agents prove their work

Ahmad Dogar — Sun, 25 Oct 2026 18:27:07 +0000

The most expensive sentence a coding agent writes is "Everything should work now." It is said with confidence, and sometimes it is even true. An agent's summary of its own work is the least reliable part of its output, because it describes what the agent meant to do, not what happened.

So on this site the rule is that every report arrives with evidence I can check without re-running anything: command output, numbers and screenshots. Most of the checks are ordinary tools that existed long before agents. The work is in making the agent run them every time, before it calls itself finished, and in refusing to merge when the evidence is missing.

Cheap and precise checks first, slow and visual ones last.

The order of checks

Cheapest and most precise first. A type error points at a line; a broken screenshot points at nothing in particular. For this site, a Vite, React and TypeScript app with Vercel functions, the sequence is:

npm run typecheck, which is tsc -b across the app, the API and the scripts.
npm run lint with oxlint, fast enough to run after every change.
npm run build, which also runs the prerender step, so a page that crashes during server rendering fails here rather than in production.
npm run budget, the bundle budget described below.
Screenshots of the pages the change touched, at a desktop and a phone width.

The GitHub Actions workflow runs the first four on every push and pull request. Agents run all five locally before reporting, because a red CI run after I have merged is a much slower loop than a red run inside the agent's own worktree.

What a report has to contain

The same format for every agent, every task.

Every agent gets the same report format. The files it changed. The actual output of each command, pasted, not paraphrased. For tests, the counts, because "tests pass" from an agent that skipped the suite looks exactly like the real thing. The budget numbers. Paths to the screenshots.

And a last section, which I read first: what it did not check. An agent that writes "I did not test the signed-in view" has saved me a bug. An agent that leaves the section empty after changing the editor has told me where to look. Asking for it explicitly matters, because agents rarely volunteer the limits of their own testing.

One honest gap: this site has no unit test suite. Its checks are types, lint, the build with its prerender step, the budget and screenshots, and for a mostly static portfolio that has been enough. On projects that do have tests, the counts go in the report, and a change that adds behaviour without adding a test gets sent back with a request for one.

A Stop hook so done means green

Telling an agent to run the checks works most of the time. To make the fast ones unconditional I use a Stop hook: a script Claude Code runs whenever the agent tries to end its turn. If the script exits with code 2, the agent does not stop. It receives the script's stderr as feedback and keeps working.

The agent cannot finish while the fast checks are red.

.claude/settings.json

{
  "hooks": {
    "Stop": [
      {
        "hooks": [
          { "type": "command", "command": "${CLAUDE_PROJECT_DIR}/.claude/hooks/verify.sh" }
        ]
      }
    ]
  }
}

.claude/hooks/verify.sh

#!/usr/bin/env bash
# Runs when the agent tries to finish. Exit 2 sends the errors back and keeps it working.
input=$(cat)
# Already continuing because of this hook? Let it stop instead of looping.
if [ "$(printf '%s' "$input" | jq -r '.stop_hook_active')" = "true" ]; then
  exit 0
fi
cd "$CLAUDE_PROJECT_DIR" || exit 0
if ! out=$(npm run --silent typecheck 2>&1); then
  printf 'Typecheck failed. Fix these before finishing:\n%s\n' "$(printf '%s' "$out" | tail -40)" >&2
  exit 2
fi
if ! out=$(npm run --silent lint 2>&1); then
  printf 'Lint failed. Fix these before finishing:\n%s\n' "$(printf '%s' "$out" | tail -40)" >&2
  exit 2
fi
exit 0

Two choices in that script are worth explaining. It runs only the fast checks; a full build in a Stop hook would make every small answer slow, so the build and the budget stay in the instructions and in CI. And it gives up after one forced retry, via the stop_hook_active flag, instead of trapping the agent on an error it cannot fix. Claude Code has its own cap on consecutive continuations, but I would rather get a clear failure in the report than a long loop.

Bundle budgets in CI

Type checks catch broken code. They do not catch heavy code, and agents add weight casually: a date library for one format call, a whole icon set for three icons. This site has a 3D view built on three.js, which is large, so from the start it had a budget script, scripts/check-budget.mjs, that runs after the build in CI.

The numbers print on every build, so they land in every report.

scripts/check-budget.mjs (excerpt)

const html = readFileSync(join(dist, 'index.html'), 'utf8')
const refs = [...html.matchAll(/(?:src|href)="\/(assets\/[^"]+\.(?:js|css))"/g)].map((m) => m[1])
let initial = 0
for (const ref of new Set(refs)) {
  const buf = readFileSync(join(dist, ref))
  initial += gzipSync(buf).length
  if (/three|fiber|drei/i.test(ref)) errors.push(`3D code is in the initial payload: ${ref}`)
}
console.log(`initial JS+CSS (gzip): ${fmt(initial)} / ${fmt(BUDGET.initialJsGzip)}`)
if (initial > BUDGET.initialJsGzip) errors.push('initial payload over budget')

It gzips every JS and CSS file that index.html loads up front and fails the build above 250 KB. It also fails if three.js appears in the initial chunks at all, by file name or by the string WebGLRenderer, because the 3D view has to stay lazy-loaded. Every image in public/shots must be webp, avif or svg and under 120 KB, which is one reason the diagrams in this post are SVG.

Today the initial payload is 85.1 KB of the 250 KB budget. The useful part is not the headroom. It is that the number is printed on every run, so it appears in every agent report, and a jump of a few dozen KB from one change is visible before merge instead of in a slow page months later.

Show me the screenshot

Two widths catch most layout breaks.

For anything visual, a passing build proves very little. CSS can be valid and still put the search box underneath the navbar. So agents take screenshots with Playwright, which this repo already uses to capture the project images in its gallery, at two sizes: 1440 by 900 for desktop, and 390 wide at 2x for a phone. Between them, those two widths catch most of the layout breaks worth catching.

The script is short enough that agents can run it without thinking about it, and its output is just file paths, which go straight into the report.

shot.mjs

import { mkdirSync } from 'node:fs'
import { chromium } from 'playwright'

const url = process.argv[2] ?? 'http://localhost:4173/blog'
const sizes = {
  desktop: { viewport: { width: 1440, height: 900 }, deviceScaleFactor: 1 },
  phone: { viewport: { width: 390, height: 844 }, deviceScaleFactor: 2 },
}

mkdirSync('shots', { recursive: true })
const browser = await chromium.launch()
for (const [name, options] of Object.entries(sizes)) {
  const page = await browser.newPage(options)
  await page.goto(url, { waitUntil: 'networkidle' })
  await page.screenshot({ path: `shots/${name}.png`, fullPage: true })
  console.log(`shots/${name}.png`)
  await page.close()
}
await browser.close()

Then I open the images myself before merging. Not because the agent cannot look at a PNG (it can, and I ask it to check its own screenshots against the request first), but because the agent decided what "looks right" means when it wrote the code. A look from someone who did not write it is the whole point of review, and it takes seconds per image.

Evidence over trust

None of this assumes the agent is careless. It assumes what we assume of any engineer: "done" should come with something checkable. Agents simply make that cheap to ask for, since five commands and two screenshots cost them a minute. The habit that matters is on my side of the review. I do not merge a change whose report has no evidence in it, however confident the summary sounds. If the evidence is missing, the reply is short: run the checks and send the output. That reply costs me ten seconds and saves the much longer job of finding out myself.

1. Order the checks

Typecheck, lint, build, budget, screenshots. Cheap and precise first.

2. Fix the report format

Files, pasted output, test counts, budget numbers, screenshot paths, and what was not checked.

3. Add a Stop hook for the fast checks

Exit 2 keeps the agent working until typecheck and lint pass.

4. Put size in CI

A gzip budget and a per-image limit, printed on every run.

5. Look at the screenshots

Two widths, opened by someone who did not write the code, before every merge.

Claude Code hooks reference: Stop hooks, stop_hook_active and what exit code 2 does for each hook event.

Originally published on my site. More posts and projects at mahmadmanzoor.com.