An agent that writes files into your repo is a compiler with no type checker on its output. Zod gives you one — but it checks the shape of what the model returned. Your build checks what that shape means. Those are different jobs, and everything that lives in the gap between them is where scheduled agents fail at 3am with nobody watching.
We run a nightly pipeline that drafts articles, generates summary blocks, writes MDX to disk, builds a static site, and deploys it. Every artifact that reaches the build step has already passed a z.parse(). Over roughly three months of nightly runs, the build still broke about a dozen times. None of those failures were type errors. Every one was a value that was structurally perfect and semantically wrong.
Passing zod is a claim about shape, not about the world
Here is a schema close to what we started with:
const Article = z.object({
slug: z.string().regex(/^[a-z0-9-]+$/),
title: z.string().min(20).max(120),
category: z.enum(['ai-dev-tools', 'infrastructure', 'meta']),
tools: z.array(z.string()),
publishedAt: z.coerce.date(),
body: z.string().min(2000),
});
It is a reasonable schema. It also accepts, without a single complaint:
- A
slugthat is already a file on disk, so the write silently replaces an article published two weeks earlier. - A
toolsentry naming a product that has no record in the affiliate table, so the footer component renders an empty card and the page ships with a dead link. - A
publishedAtthree days in the future, which is a validDateand an invisible article. - A
bodycontaining the characters{config}inside a sentence.
Each one is a green parse and a red build — or worse, a green build and a broken page.
z.string().url()is the most misleading validator in the library. It asserts that a string parses as a URL and nothing more. A link to a page that 404s, a domain you never registered, and a redirect for an affiliate program that got paused last week all pass identically. If the URL has to exist, the check is a network call — and a network call does not belong inside a schema.
Four failure classes that survive a clean parse
1. Referential drift. The output contains an identifier that must resolve to something else: a slug, a category, a tool id, an image path, a foreign key. Zod confirms it is a string matching a pattern. Nothing confirms the target exists. Our worst instance was a category value that passed z.enum() — the enum was correct, the category page generated, and it generated with zero posts in it, because the enum listed a category we had drained months earlier. A published empty page is a ranking liability that no parser will ever flag.
2. Cross-field contradiction. Each field is individually fine and the combination is nonsense. updatedAt earlier than publishedAt. An audience field of pm on a file the writer put in the dev directory. A readTimeMinutes of 4 on a 3,000-word body. Zod can catch these — but only if you reach for superRefine, and most schemas are written field-by-field, which is exactly the frame in which cross-field bugs are invisible.
3. Collisions and re-runs. A scheduled agent is not a one-shot script. It gets killed mid-run, retried, and run again the next night on overlapping inputs. The same topic produces the same slug twice. Both outputs validate. The second overwrites the first, and your git diff shows a content change rather than an error. Validation has no concept of what already exists; it only sees the object in front of it.
4. Valid string, invalid artifact. This is the one that actually broke our build, twice. A model wrote the phrase pass {config} to the runner into prose. MDX treats {...} as a JavaScript expression, so the compiler tried to resolve an identifier named config and failed the entire build — not just the page. z.string() saw 41 perfectly ordinary characters. Same class of bug: a bare <Tool> in a sentence, a code fence the model opened and never closed, a component referenced in the body but missing from the import block.
The pattern behind all four: your schema validates against the type, and your failures happen at the consumer.
Validate against the consumer, not the type
We now run three layers, in order, all before the artifact is written to its final path.
Layer 1 — shape. Keep your zod schema exactly as it is. It is fast, it is pure, and it catches genuinely malformed output. Just stop treating it as the gate.
Layer 2 — resolution. Every identifier in the output gets looked up against the thing it points at. Tool slugs query the affiliate table. Category values are checked against categories that currently have posts. The target file path is checked for an existing file, and a collision is a hard failure rather than an overwrite. This layer needs I/O, which is precisely why it does not belong inside the schema — schemas should stay synchronous and pure so you can unit-test them without a database.
Layer 3 — dry render. Compile the artifact the way the build compiles it, on the string, before it ever touches disk:
async function renderError(mdx: string): Promise<string | null> {
try {
await compile(mdx, { jsx: true });
return null;
} catch (err) {
return (err as Error).message;
}
}
That is nine lines, and it moved our MDX parse failures from build-time to generation-time. The difference matters more than it sounds: at generation-time the agent is still running, still holds the context, and can retry with the compiler's own error message pasted into the prompt. At build-time it is a broken file, a red deploy, and a human reading a stack trace the next morning.
When layer 2 or 3 fails, do not crash the run and do not silently drop the artifact. Write it to a
quarantine/directory with the failure message in a sibling file, and exit non-zero from the generation step only. Two properties fall out of this: the run stays idempotent, so a retry cannot half-publish, and the failure is inspectable the next morning instead of being a log line that scrolled away.
The rule we ended up with: put the check where the failure actually happens. If a value breaks the renderer, test it with the renderer. If it breaks because a row is missing, query the row. A schema tells you the model returned an object of the right shape. It has never told you the object was correct.
Originally published at pickuma.com. Subscribe to the RSS or follow @pickuma.bsky.social for new reviews.
Top comments (0)