The LLM Response Looked Fine. My Parser Disagreed.

The response looked perfectly reasonable in the logs.

It had the fields I asked for. The values made sense. Nothing looked obviously broken.

Then my application rejected it.

That failure taught me to separate two questions I had been treating as one:

1. Does the response look correct to a human?
2. Is the response safe for software to consume?

Those are very different standards.

A person can understand JSON wrapped in a Markdown fence, infer a missing optional field, or ignore a sentence added before the object. A parser cannot—and in production, it should not have to guess.

The response was valid English, not valid data

Imagine asking a model for this:

{
  "category": "billing",
  "priority": "high",
  "summary": "Customer was charged twice"
}

Instead, it returns:

Here is the JSON you requested:

json
{
"category": "billing",
"priority": "high",
"summary": "Customer was charged twice"
}

json

To a human, that is fine.

To JSON.parse(), it is a syntax error.

That is the obvious failure. The more annoying ones are valid JSON with the wrong shape:

{
  "category": "billing",
  "priority": "urgent",
  "summary": ""
}

The JSON parses successfully, but the application still has three problems:

• urgent is not a supported priority
• summary is present but unusable
• the response passed syntax validation without passing business validation

This is why I no longer treat JSON.parse() as validation. It only proves that a string is valid JSON.

The small validation layer I use

For Node.js projects, I usually put a schema between the model response and the rest of the application.

Here is a minimal example using Zod.

Install it:

npm install zod

Create validate-llm-output.js:

import { z } from "zod";

const TicketSchema = z.object({
  category: z.enum([
    "billing",
    "technical",
    "account",
    "other"
  ]),
  priority: z.enum([
    "low",
    "medium",
    "high"
  ]),
  summary: z.string().trim().min(1).max(240)
}).strict();

function removeMarkdownFence(value) {
  const trimmed = value.trim();

  const match = trimmed.match(
    /^```
{% endraw %}
(?:json)?\s*([\s\S]*?)\s*
{% raw %}
```$/i
  );

  return match ? match[1].trim() : trimmed;
}

export function validateTicketOutput(rawOutput) {
  const cleaned = removeMarkdownFence(rawOutput);

  let parsed;

  try {
    parsed = JSON.parse(cleaned);
  } catch (error) {
    return {
      ok: false,
      stage: "json_parse",
      error: error.message,
      rawOutput
    };
  }

  const result = TicketSchema.safeParse(parsed);

  if (!result.success) {
    return {
      ok: false,
      stage: "schema_validation",
      error: result.error.flatten(),
      rawOutput
    };
  }

  return {
    ok: true,
    data: result.data
  };
}

Then test it:

import { validateTicketOutput } from "./validate-llm-output.js";

const response = `\`\`\`json
{
  "category": "billing",
  "priority": "high",
  "summary": "Customer was charged twice"
}
\`\`\``;

console.log(validateTicketOutput(response));

The result is:

{
  ok: true,
  data: {
    category: "billing",
    priority: "high",
    summary: "Customer was charged twice"
  }
}

Now try a response with an unsupported priority:

const response = JSON.stringify({
  category: "billing",
  priority: "urgent",
  summary: "Customer was charged twice"
});

console.log(validateTicketOutput(response));

This time, the validation fails before the value reaches application logic.

I keep parsing and validation separate

It is tempting to put every cleanup rule into one large function. I found it easier to debug when each failure has a distinct stage:

raw model response
        ↓
transport check
        ↓
JSON parsing
        ↓
schema validation
        ↓
business validation
        ↓
side effect

This distinction matters when something breaks.

A truncated response is not the same failure as an unsupported enum. A structurally correct object with a prohibited action is not the same failure as malformed JSON.

If all three become Invalid model response, the log is technically correct and operationally useless.

I record at least:

{
  requestId,
  model,
  stage,
  schemaVersion,
  error,
  responseLength,
  finishReason
}

I do not automatically store the full prompt or response. Whether that is acceptable depends on the data involved. For sensitive inputs, metadata and a redacted failure sample are often safer than dumping raw content into logs.

What I do not silently repair

Some cleanup is deterministic.

Removing one complete Markdown fence is predictable. Trimming whitespace is predictable.

Inventing a missing field is not.

I avoid quietly doing things like:

if (!output.priority) {
  output.priority = "medium";
}

That converts a model failure into application behavior without leaving a clear trace.

The same applies to truncated JSON. If the response ends halfway through an object, I do not ask a regex to reconstruct what the model “probably meant.”

My rule is simple:

• Normalize presentation noise when the transformation is unambiguous
• Reject structural ambiguity
• Never execute a side effect from an invalid response

A failed classification can usually be retried or reviewed.

An incorrect refund, email, database update, or account action is much harder to undo.

Schema validation is not the final gate

A schema can prove that this object has the expected shape:

{
  "action": "refund",
  "amount": 500,
  "reason": "duplicate charge"
}

It cannot prove that the refund is authorized.

That requires business rules:

function validateRefund(ticket, account) {
  if (ticket.action !== "refund") {
    return { ok: false, reason: "unsupported_action" };
  }

  if (ticket.amount > account.maxAutomaticRefund) {
    return { ok: false, reason: "human_approval_required" };
  }

  if (!account.hasDuplicateCharge) {
    return { ok: false, reason: "duplicate_charge_not_verified" };
  }

  return { ok: true };
}

I now treat model output as untrusted input, even when the model was instructed to return a strict format.

The model proposes data.

The application decides whether that data is valid and whether it is allowed to cause an action.

The failure categories worth tracking

Instead of one generic parse-error counter, I track separate categories:

• empty response
• truncated response
• Markdown-wrapped JSON
• invalid JSON
• missing required field
• unsupported enum
• unexpected additional field
• business-rule rejection
• human approval required

This makes provider or model changes easier to evaluate.

If malformed JSON increases after a model switch, that is a format-reliability issue. If business-rule rejections increase, the JSON may be perfect while the decisions are getting worse.

Those problems need different fixes.

What changed in my production code

The biggest change was not a better prompt.

It was moving trust out of the prompt.

I still ask for a precise response format, but I assume the contract can be broken. Every response must pass parsing, schema validation, and business validation before the application uses it.

That makes the failure less mysterious:

The model returned 200.
The text looked fine.
The parser rejected it.
No side effect happened.
The failure was recorded at the correct stage.

That is a much better outcome than allowing a plausible-looking response to quietly become production state.

I work on TokenBay, where I spend a lot of time thinking about model and provider behavior. But this validation layer is provider-agnostic: if an LLM response is going to influence real application logic, I want code—not confidence—to decide whether it is safe.