Trusting an LLM's JSON in production

The demo always works. You prompt the model to "return JSON with these fields," it returns clean JSON, you parse it, everyone claps. Then you put it behind real traffic and real user input, and a week later something downstream throws because a field you assumed was a number came back as the string "two" and nothing validated it on the way in.

This post is about the layer between "the model usually returns the right shape" and "the rest of my system can depend on the shape." That layer is not glamorous and it is most of the work. The examples use a fictional telecom assistant that turns a customer's freeform message into a structured intent, but the pattern is the same whether you're extracting invoices, classifying tickets, or routing a support request.

what actually breaks

When you rely on an LLM to produce structured data, the failures are not the ones you plan for. In roughly the order they cost me time:

1. Malformed output. The model wraps the JSON in a code fence, or adds a friendly sentence before it, or trails off mid-object under load. Rarer now with structured-output modes, still real.
2. Valid JSON, wrong types. A count comes back as "3" instead of 3. A boolean comes back as the string "true". JSON.parse is perfectly happy. Your code is not.
3. Valid types, hallucinated values. The customer said nothing about a router, and the model returns equipment: "router" because the training data made it a plausible guess. This is the dangerous one, because it passes every structural check.
4. Partial correctness under pressure. Give the model a vague message and it fills required fields with confident nonsense rather than admitting uncertainty. Models are trained to answer, not to abstain.

A parse-and-hope pipeline catches exactly one of these. You need a design that assumes all four.

start from a schema, not a prompt

The mistake I made early was describing the output shape in the prompt, in prose, and treating that as the contract. The prompt is a request. The schema is the contract, and it should live in code where you can enforce it.

I define the shape once with Zod and derive everything else from it.

import { z } from "zod";

export const CustomerIntent = z.object({
  action: z.enum(["cancel", "upgrade", "downgrade", "add_line", "question"]),
  // which service the customer is talking about, if any
  service: z.enum(["internet", "mobile", "tv", "bundle"]).nullable(),
  // only set when the model is confident it was stated
  lineCount: z.number().int().min(0).max(20).nullable(),
  // free-text summary for a human to sanity check
  summary: z.string().min(1).max(280),
  // the model's own confidence, which we use for routing, not for trust
  confidence: z.enum(["high", "medium", "low"]),
});

export type CustomerIntent = z.infer<typeof CustomerIntent>;

Two things are deliberate here. Every field that might not be present is .nullable() rather than optional, so the model has to make an explicit choice instead of quietly omitting it. And confidence is a field the model fills in, but as you'll see, I never let it decide anything on its own.

use structured mode, then distrust it anyway

Both the Anthropic and OpenAI SDKs let you constrain output to a schema, either through tool calling or a response-format parameter. Use it. It removes most of failure mode one, the malformed-output case, by making the wire format the model's problem instead of yours.

What it does not remove is failure modes two through four. The provider guarantees you get a syntactically valid object that matches the JSON schema you handed it. It does not guarantee the values mean anything. So the constrained call is step one, and the validation below is step two, and skipping step two because "the API already validates" is how the hallucinated router gets into your billing system.

import Anthropic from "@anthropic-ai/sdk";
import { zodToJsonSchema } from "zod-to-json-schema";

const client = new Anthropic();

async function extractIntentRaw(message: string): Promise<unknown> {
  const res = await client.messages.create({
    model: "claude-sonnet-5",
    max_tokens: 512,
    tools: [
      {
        name: "record_intent",
        description: "Record the structured customer intent.",
        input_schema: zodToJsonSchema(CustomerIntent) as Anthropic.Tool.InputSchema,
      },
    ],
    tool_choice: { type: "tool", name: "record_intent" },
    messages: [{ role: "user", content: message }],
  });

  const block = res.content.find((b) => b.type === "tool_use");
  if (!block || block.type !== "tool_use") {
    throw new Error("model did not call the tool");
  }
  return block.input;
}

Note the return type is unknown, on purpose. Nothing past this function gets to assume the shape until it has been through the validator.

validate at the boundary

This is the whole point of the post. The model's output crosses into your system at exactly one place, and that place validates or rejects. No downstream code parses model output a second time.

type ExtractResult =
  | { ok: true; intent: CustomerIntent }
  | { ok: false; error: string };

function validate(raw: unknown): ExtractResult {
  const parsed = CustomerIntent.safeParse(raw);
  if (!parsed.success) {
    return { ok: false, error: parsed.error.toString() };
  }
  return { ok: true, intent: parsed.data };
}

Now the caller gets a discriminated union. There is no path where you hold a CustomerIntent that hasn't been checked, and the type system enforces that you handle the failure branch. This one pattern, a validated boundary returning a tagged result, prevents more production incidents than any prompt tweak I've ever made.

the retry that actually helps

When validation fails, the naive move is to call the model again with the same input and hope for better luck. Sometimes that works. It works much more often if you tell the model what was wrong.

async function extractIntent(
  message: string,
  attempts = 2,
): Promise<ExtractResult> {
  let lastError = "";

  for (let i = 0; i < attempts; i++) {
    const raw = await extractIntentRaw(
      lastError
        ? `${message}\n\nYour previous response failed validation with: ${lastError}. Return a corrected response.`
        : message,
    );
    const result = validate(raw);
    if (result.ok) return result;
    lastError = result.error;
  }

  return { ok: false, error: lastError };
}

Feeding the validation error back turns the second attempt into a correction rather than a re-roll. In practice most recoverable failures recover on attempt two, and if they don't recover in two, a third rarely helps. Cap it. An unbounded retry loop against a paid API is a way to turn a bad input into a bad invoice.

deciding when to fall back

Here is where confidence earns its place, and where I stop trusting the model. The model's self-reported confidence is an input to my routing logic, never the deciding vote. A low confidence means route to a human. A high confidence on an action with real consequences, like a cancellation, also routes to a human, because the cost of being wrong is asymmetric.

function route(intent: CustomerIntent): "auto" | "human" {
  if (intent.confidence === "low") return "human";
  if (intent.action === "cancel") return "human";
  if (intent.service === null && intent.action !== "question") return "human";
  return "auto";
}

The rule I follow is that the model decides what the customer said, and my code decides what to do about it. Confidence scores, thresholds, and fallbacks are ordinary software written by me, not judgment delegated to the model. That line is what makes the whole thing safe to run unattended.

tool calling is the same problem wearing a hat

Everything above is framed as data extraction, but agentic tool calling is the identical problem. When the model decides to call process_order with some arguments, those arguments are model-generated structured output, and they deserve the same schema, the same validation boundary, and the same "high stakes route to a human" rule before the tool actually runs. The mistake is treating a tool call as an instruction to execute rather than a proposal to validate. Validate the arguments, check the stakes, then execute. A tool handler that runs whatever the model hands it is a demo, not a production integration.

what I log

When one of these pipelines misbehaves in production, you want to reconstruct exactly what happened, and the model's output is not deterministic, so you cannot just re-run it. For every extraction I log the input message, the raw model output before validation, the validation result, the attempt count, and the final route. When something goes wrong, that record tells me whether the model produced bad output, or produced good output that my code then mishandled. Those are different bugs with different fixes, and without the raw-output log you cannot tell them apart.

the short version

Getting an LLM to return the right shape once is a prompt. Getting a system you can depend on is a schema in code, a constrained call, a hard validation boundary that returns a tagged result, an error-aware retry with a cap, and routing logic that you own rather than the model. The model produces a claim. Your code decides whether to believe it. Keep that split clean and the rest is manageable.