The JSON Was Valid. The Customer ID Was Invented.

Last week's report showed eleven transactions attached to a customer ID that did not exist in our customers table. No exception was thrown, no webhook failed, and every log line said processed=true. The ingestion job had done exactly what it was told: it parsed the model's JSON, validated the shape, and committed the record.

I run this pipeline on MonkeyCode's free server, and the extraction step calls a free model through their gateway. The model's job is to pull transaction details out of support emails and return them as JSON. Disclosure: This article was prepared as part of MonkeyCode's product outreach. The failure I'm about to walk through is not a product bug — it's a design gap in my own pipeline, and it would have happened with any model that returns JSON.

The interesting part is that the debugging took me three wrong hypotheses before I looked at the raw response. Let me show you the path, because the reusable lesson is not "models hallucinate" — it's that my validation layer was checking the wrong things.

The symptom

The report joined transactions to customers with a LEFT JOIN and flagged every row where the customer didn't resolve. That's how I found the orphans:

SELECT t.transaction_id, t.customer_id, c.name
FROM transactions t
LEFT JOIN customers c ON c.id = t.customer_id
WHERE c.id IS NULL;

Eleven rows came back, and they all had one thing in common: a customer_id like CUST-48291 that matched the expected format perfectly. The regex ^CUST-\d{4,6}$ passed, the JSON was valid, and the record was written without a single error. So where did the wrong ID come from?

The wrong suspects

My first instinct was the report query itself, so I re-ran it against a known-good transaction and it worked. My second suspect was a race between the ingestion job and the customers sync, but the timestamps ruled that out — the orphan IDs were written hours after the last customer import. My third suspect was truncation, so I added a response-length check, and every response was complete.

Then I finally did what I should have done first: I printed the raw model output for one of the bad transactions.

{
  "transaction_id": "txn_8f3a",
  "customer_id": "CUST-48291",
  "amount": 129.00,
  "currency": "USD",
  "timestamp": "2026-08-12T14:22:10Z"
}

The JSON is valid. The types are right. The format is right. The customer ID is invented.

The root cause

The email that produced this transaction never mentioned a customer ID, and my prompt told the model to "infer the customer if it seems obvious." The model inferred confidently and wrong, which is the worst kind of model failure because it looks exactly like success.

The pipeline stayed silent for three reasons, and each one is a design decision I made:

1. The parser checked structure and format, but not semantics — any string matching CUST-\d{4,6} was accepted.
2. The message was acknowledged immediately after parsing, so the event was gone before any referential check could run.
3. The transactions table had no foreign key on customer_id, so the database happily accepted a ghost.

The fix: three validation layers

I added three layers, in the order that catches failures cheapest first:

1. Schema validation with jsonschema — types, required fields, and format patterns.
2. Referential check — the customer_id must exist in the customers table before the record is committed.
3. Retry with feedback — if the referential check fails, send the validation error back to the model once and ask it to correct the output or return null.

Here is the core function, which is small enough to copy into any ingestion pipeline:

import json
from jsonschema import validate, ValidationError, FormatChecker

TRANSACTION_SCHEMA = {
    "type": "object",
    "required": ["transaction_id", "customer_id", "amount", "currency", "timestamp"],
    "properties": {
        "transaction_id": {"type": "string", "pattern": "^txn_[a-z0-9]+$"},
        "customer_id": {"type": ["string", "null"], "pattern": "^CUST-[0-9]{4,6}$"},
        "amount": {"type": "number", "minimum": 0},
        "currency": {"type": "string", "minLength": 3, "maxLength": 3},
        "timestamp": {"type": "string", "format": "date-time"},
    },
}

def resolve_customer(raw: str, known_ids: set[str]) -> dict:
    try:
        data = json.loads(raw)
    except json.JSONDecodeError as exc:
        return {"ok": False, "stage": "parse", "error": str(exc)}

    try:
        validate(instance=data, schema=TRANSACTION_SCHEMA, format_checker=FormatChecker())
    except ValidationError as exc:
        return {"ok": False, "stage": "schema", "error": exc.message}

    if data["customer_id"] and data["customer_id"] not in known_ids:
        return {
            "ok": False,
            "stage": "reference",
            "error": f"unknown customer {data['customer_id']}",
        }

    return {"ok": True, "record": data}

The retry-with-feedback step is where the model gets a second chance, and it is the part that surprised me most:

def ingest_with_feedback(raw: str, known_ids: set[str], model_call) -> dict:
    first = resolve_customer(raw, known_ids)
    if first["ok"]:
        return first

    if first["stage"] in ("schema", "reference"):
        feedback = (
            "The previous JSON failed validation: "
            f"{first['error']}. "
            "Return the same object with customer_id set to null "
            "if you cannot find a real customer ID in the email."
        )
        second_raw = model_call(raw, feedback)
        second = resolve_customer(second_raw, known_ids)
        if second["ok"]:
            return second

    return {"ok": False, "stage": first["stage"], "dead_letter": raw}

The model_call argument is whatever your gateway client exposes; the important part is that the feedback string goes into the same prompt. When the second attempt fails, the raw response goes to a dead-letter queue on durable storage, and the original message is never acknowledged. That last part matters more than the retry itself, because an acked message is a lost event.

The decision table

Stage	Failure example	Action
Parse	invalid or truncated JSON	retry once, then dead-letter
Schema	missing field or wrong type	retry once with feedback, then dead-letter
Reference	format-valid but unknown customer_id	retry once with feedback, then dead-letter
Commit	database constraint violation	do not ack; park for manual review

The table is the artifact I actually reuse, because it forces me to decide where each failure lands before it happens — steal it and adapt the stages to your own pipeline. The week after the fix, the same invented ID appeared, the retry corrected it to null, and the record was parked with the raw email attached. The report now shows a visible gap instead of a silent ghost.

Limitations and who should not use this

Retry-with-feedback costs an extra model call, and on a free tier that means budgeting for rate limits — my earlier incident with a retry storm taught me to cap the loop at one retry, not three. If your model is non-deterministic and the feedback loop does not converge, you need a human review queue, not more retries. If customers are created in the same batch as transactions, the referential check has to run after the batch completes, not per record. And if your use case tolerates a wrong ID, this whole layer is overkill — add the cheapest check that catches the failure you actually saw.

The reusable lesson

Format-valid is not the same as correct, and the gap between those two is where silent corruption lives. The debugging moves that saved me were reproducing with a known-good record, printing the raw response before blaming the parser, and adding the semantic check at the same layer where the data is written. Next time your model output "looks valid," ask what it would take for it to be wrong in a way your checks cannot see — then write that check. How many other pipelines are acking messages before the data is actually correct?