Gabriel Anhaia

Posted on Aug 6

Type Every LLM Response with Zod and 4 Classes of Bug Disappear

#typescript #zod #ai #node

Book: AI That Answers
The series: AI in TypeScript — 5 books, from your first LLM call to agents in production — all five here
My project: Hermes IDE | GitHub — an IDE for developers who ship with Claude Code and other AI coding tools
Me: xgabriel.com | GitHub

A team I talked to had a classification endpoint that had been green
for six weeks. Every request returned 200. Their error rate dashboard
was flat. Then someone ran a report and found that eleven percent of
records had a priority of undefined, and had for the whole six
weeks.

Nothing threw. undefined serialises to nothing in JSON, sorts last,
and renders as an empty cell. The bug was invisible at every layer
that could have caught it, because none of those layers were looking.

Model output fails in a small number of recognisable ways. Here are
four of them, what each one does downstream, and the schema line that
turns each into a caught error at the boundary instead of a data
problem you discover in a quarterly report.

The setup

One schema, used for all four.

import { z } from "zod";

const Ticket = z.object({
  summary: z.string().min(1),
  priority: z.enum(["low", "medium", "high", "urgent"]),
  estimateHours: z.number().nonnegative(),
  tags: z.array(z.string()).min(1).max(5),
});

export type Ticket = z.infer<typeof Ticket>;

And the call, with the boundary in place.

async function classify(body: string): Promise<Ticket> {
  const res = await client.messages.create({
    model: "claude-opus-5",
    max_tokens: 512,
    messages: [{ role: "user", content: prompt(body) }],
  });
  const raw: unknown = JSON.parse(textOf(res.content));
  return Ticket.parse(raw);
}

Now the four failures.

1. The field that quietly is not there

The model returns everything except priority, because the ticket
body was one line and there was nothing to judge.

{
  "summary": "Login button misaligned on Safari",
  "estimateHours": 0.5,
  "tags": ["ui", "safari"]
}

Without a schema this is parsed.priority === undefined, and
undefined is a value TypeScript is happy to carry when the object
came from any. It reaches your database as NULL, your sort as
last, your Slack notification as Priority:.

z.enum makes the field required by construction. Ticket.parse
throws with path: ["priority"], message: "Required".

The instinct at this point is to make it optional so the parse stops
failing. Resist that unless the field is genuinely optional in your
domain. If your UI cannot render a ticket without a priority, then a
ticket without a priority is not a ticket, and the correct behaviour
is to reject it — or to give it a default explicitly:

priority: z.enum(["low", "medium", "high", "urgent"])
  .default("medium"),

That is a decision recorded in code. undefined flowing through is
the same decision made by accident.

2. The enum near-miss

The model returns "High". Or "HIGH". Or "p1", because your
prompt mentioned P1 somewhere and it generalised.

{ "priority": "High", "summary": "...", "estimateHours": 2, "tags": ["auth"] }

This is the one that survives longest in production, because
"High" is a perfectly good string. It passes any check that is
looking for truthiness or type string. It fails only where
something compares it to "high" — a switch that falls through to a
default, a filter that returns nothing, a badge component that
renders grey because no case matched.

z.enum catches it at the boundary with a message that lists the
accepted values. If you would rather absorb casing drift than reject
it, do that explicitly too:

priority: z.preprocess(
  (v) => (typeof v === "string" ? v.toLowerCase() : v),
  z.enum(["low", "medium", "high", "urgent"]),
),

Normalising in a preprocess is fine. Normalising by accident in
four different call sites is how "High" and "high" end up both
present in the same column.

3. The number that is a string

{ "estimateHours": "2.5", "summary": "...", "priority": "medium", "tags": ["api"] }

JavaScript makes this one dangerous rather than annoying, because
"2.5" behaves like a number in some operators and like a string in
others. "2.5" * 2 is 5. "2.5" + 2 is "2.52". A sum across a
sprint that uses + gives you a concatenated string of every
estimate, and a total that is technically defined.

z.number() rejects the string outright. Whether you want that or
coercion depends on the field:

// reject — the model should have sent a number
estimateHours: z.number().nonnegative(),

// or accept and convert, deliberately
estimateHours: z.coerce.number().nonnegative(),

Use coerce where the wire format is legitimately ambiguous, like
dates. Avoid it where a string signals the model misunderstood the
field, because coercion will happily turn "about two" into NaN
and NaN passes z.number(). If you coerce, add .finite().

4. The array that got cut off

The model hit max_tokens mid-array and the JSON is either truncated
or, worse, closed off early with fewer elements than it intended.

{ "summary": "...", "priority": "low", "estimateHours": 1, "tags": [] }

An empty array is the quietest of the four. It is valid JSON, valid
TypeScript, and renders as nothing. Every downstream .map produces
an empty list. No error anywhere.

.min(1) states the domain rule: a classified ticket has at least
one tag. .max(5) catches the opposite failure, where the model
enumerates twenty tags and floods your filter UI.

Truncation deserves a second guard, because a hard cut mid-object
gives you a JSON.parse failure rather than a Zod failure, and the
two want different handling. Check the stop reason:

if (res.stop_reason === "max_tokens") {
  throw new ResponseTruncated(res.usage.output_tokens);
}

That distinguishes "the model was wrong" from "you did not give it
room," which are different bugs with different fixes. The first wants
a better prompt. The second wants a higher max_tokens.

Failing usefully

Four schema lines catch four bug classes. What you do on failure
still matters, and safeParse keeps it in normal control flow:

const result = Ticket.safeParse(raw);
if (!result.success) {
  logger.warn("ticket parse failed", {
    issues: result.error.issues.map((i) => ({
      path: i.path.join("."),
      code: i.code,
    })),
  });
  return { ok: false as const, error: result.error };
}
return { ok: true as const, ticket: result.data };

Logging path and code rather than the whole error gives you
something groupable. After a week you can see that priority
accounts for most failures and fix the prompt where it actually
breaks, instead of guessing.

What this does not fix

A schema checks shape, not truth. priority: "urgent" parses
perfectly whether or not the ticket is urgent. Nothing here catches a
confident wrong answer — that is what evals are for, and they are a
different tool at a different layer.

What the boundary buys you is that every failure it does catch fails
at one line, with a path, at the moment it happens. Not eleven
percent of a column, six weeks later.

If this was useful

AI That Answers works through
the typed boundary in full — schema design, when to coerce and when
to reject, retrying with the validation error, and what structured
output modes do and do not guarantee on top of it.