The JSON Parsed Without Errors. The Data Was Still Wrong.

Every row in that table looked healthy. The JSON had parsed, the worker had logged success, and the database had committed an order_id that matched the source email perfectly. The only oddity was that customer_email was null for eleven orders in a row, and nothing in the pipeline had raised a single alarm. How many rows had been quietly wrong before I even looked?

I was extracting structured order data from raw emails using a free model. The workflow was embarrassingly simple: send a prompt, parse the JSON response, insert the row. Locally, every sample came back in the exact shape I had asked for. In production, the same prompt quietly returned a different shape, and my parser was polite enough to accept it.

The symptom that didn't look like a bug

A null is not an exception. That is the whole problem. My tests were green, the logs said processed: ok, and the database happily stored NULL in a column that was never meant to hold it. How many other fields had my parser forgiven without a single log line?

If you have ever debugged a system where the failure mode is a missing value instead of a crash, you know how easy it is to blame the data instead of the code. My first suspect was the model. Maybe it had returned null because the email was ambiguous, or because the prompt was weak, or because free models are simply less reliable. That hypothesis felt reasonable, and it was completely wrong.

The first suspect was wrong

Here is the parser I had written, in its full, terrible glory:

const email = result.customer_email ?? result.email ?? null;

That line was the real bug. It was designed to be forgiving, and it forgave every deviation by silently converting it into null. The model was not returning null at all — it was returning perfectly valid JSON with a slightly different shape, and my parser was translating that difference into a missing value. Was the model really that unreliable, or was my code hiding the truth?

So I went back to the raw outputs and looked at what the model had actually produced.

What the model actually returned

Same prompt, three different responses, all of them valid JSON:

{ "order_id": "A-1042", "customer_email": "[e-mail]", "line_items": [] }

{ "order_id": "A-1043", "email": "[e-mail]", "line_items": [] }

{ "order_id": "A-1044", "customer": { "email": "[e-mail]" }, "line_items": [] }

Three shapes, zero parse errors, and only one of them matched my schema. The model had drifted between customer_email, email, and a nested customer.email object, and my ?? chain had turned every mismatch into a null. The fix was not a better prompt — it was a contract at the boundary.

The technique that caught it: shape diffing

I stopped debugging the single failure and started measuring the distribution. The trick is to run the same prompt across many samples and compare the flattened key sets of every response. Any key that appears in one response and not another is a drift signal.

function flattenKeys(value, prefix = "") {
  if (Array.isArray(value)) {
    return value.length ? flattenKeys(value[0], `${prefix}[]`) : [`${prefix}[]`];
  }
  if (value && typeof value === "object") {
    return Object.entries(value).flatMap(([k, v]) =>
      flattenKeys(v, prefix ? `${prefix}.${k}` : k)
    );
  }
  return [prefix];
}

Then group the outputs by their flattened shape:

const shapes = new Map();
for (const raw of rawOutputs) {
  const keys = flattenKeys(raw).sort().join(",");
  shapes.set(keys, (shapes.get(keys) ?? 0) + 1);
}
console.table([...shapes.entries()]);

Run that on twenty samples and the mystery becomes a distribution. In my case, the customer_email shape appeared in about 60% of responses, the email shape in 30%, and the nested object in the remaining 10%. The drift was not a rare edge case — it was the norm, and my parser had been hiding it the entire time.

The workflow, once the harness existed, was simple:

• Collect twenty to fifty raw model outputs from production logs or a replay script.
• Flatten each response into a sorted key list.
• Group by key list and count how often each shape appears.
• Investigate any shape that shows up more than once.

The fix: validate at the boundary, not in the parser

The lenient parser was the wrong layer to be forgiving. Strictness belongs at the boundary, where the raw model output enters the system. I replaced the ?? chain with a schema validation that rejects anything that does not match the contract:

import { z } from "zod";

const orderSchema = z.object({
  order_id: z.string(),
  customer_email: z.string().email(),
  line_items: z.array(z.object({
    sku: z.string(),
    quantity: z.number().int().positive(),
  })),
});

const parsed = orderSchema.safeParse(result);
if (!parsed.success) {
  await saveRejectedOutput(rawOutput);
  throw new Error(`shape mismatch: ${parsed.error.issues.length} issues`);
}

The rejection bucket

Now a drift becomes a loud, visible rejection instead of a silent null. The raw output is saved to a rejection bucket so I can inspect exactly what the model meant, and the error surfaces in monitoring instead of hiding in a database column. It took about an hour to implement, and it has caught more shape drift in a week than my old parser caught in a month.

Where MonkeyCode fit into this

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

I ran the shape-diff harness on MonkeyCode's free model access, mostly because the whole point of the exercise was to sample dozens of responses, and free access made that easy to justify. The free server is where the bug actually bit: the production path had no validation, so the drift landed silently in the database. The lesson, though, was about the boundary — the same silent null would have happened with any model, free or paid.

Limitations and who should skip this

Strict schema validation catches shape drift, but it does not catch semantic drift. A model can return customer_email: "not-an-email" and Zod will reject it, but it can also return a confident, well-formed, completely wrong total — and no schema will save you from that. Validation is a safety net, not a proof of correctness.

If you are doing one-off extraction where a null is acceptable, this whole apparatus is overkill. If your model supports constrained decoding or a native JSON mode, use that first and treat validation as the backup. And if your pipeline can tolerate missing fields, by all means keep the lenient parser — just be aware that you are choosing silent degradation on purpose.

The takeaway

The JSON parsed. The data was still wrong. The fix was not a better prompt, and it was not a better model — it was a contract at the boundary and a harness that measures how often the model keeps it. If you have a story about a silent null that slipped past green tests, I would genuinely like to hear how you caught it.