This is part of my build-in-public series on running LLM workflows in n8n in production. Everything here is from real templates I sell and support — the mistakes are mine, the fixes are tested.
It parsed. It was still wrong.
A workflow I ran for weeks had a step that asked an LLM to classify incoming messages and return JSON: a category, a priority, and a short summary. The JSON node never complained. The workflow never errored. And once or twice a week, a message landed in the wrong queue, or with a priority that made no sense, or with a summary describing a different message entirely.
Nothing was broken in the way workflows usually break. The model returned valid JSON every time. It just did not always return the JSON I asked for: a category outside the allowed list, a priority of 9 on a 1–5 scale, a summary that was actually a copy of the input. My workflow trusted the shape and ignored the meaning, so bad data flowed downstream wearing a valid-JSON badge.
Parsing is not validation. If your workflow treats "it parsed" as "it is correct", you have no validation at all — you have a parser with good manners.
Rule 1: validate the schema, then validate the meaning
Schema validation catches the wrong type: a string where a number should be, a missing field, an extra field you did not expect. That is the easy half, and it is worth doing — a strict schema rejects the obvious garbage before it reaches anything important.
But the failures that cost me time were all schema-valid. So the second half checks meaning:
- Category must be one of the allowed values, not just a string.
- Priority must be in range, not just a number.
- Dates must be real dates, not just date-shaped strings.
- IDs the model cites must actually exist in the system it is describing, not just look like IDs.
I write these as a checklist per field, not as a vague "looks right" step. If a field cannot be checked against a list, a range, or a real record, that is a design smell: either the field should not be trusted downstream, or the workflow needs a source of truth to check it against.
Rule 2: fail closed, into a review queue — not into a retry
When validation fails, the tempting move is to retry the model call. Sometimes that works, and that is exactly why it is dangerous: a retry that usually fixes the output teaches the workflow that bad output is a transient problem. But a model that invents a category will often invent it again, slightly differently, and each retry is another chance for a plausible-looking wrong answer to slip through.
My rule now: a validation failure goes to a review queue, not back to the model. The queue entry keeps three things: the raw model output, the validation errors, and the original input. A human — or a stricter, cheaper deterministic check — decides what the right value was. Only the fields that failed get fixed; the rest of the output is kept, so review is a correction, not a rewrite.
There is one exception, and it is narrow: a pure format failure, like trailing text after the JSON or a markdown fence around it, gets one cleanup pass with a deterministic parser, not another model call. If cleanup still fails, it goes to review like everything else.
Rule 3: keep the raw output, always
The most useful debugging artifact in these workflows is the raw model output, before any parsing, cleanup, or transformation. I log it with the execution ID, even on success. On success it costs almost nothing. On failure it is the difference between "the model said something wrong" and "here is exactly what it said, and here is which check caught it".
This also protects against a subtle trap: if you only log the parsed result, a validation bug looks like a model bug. I have chased a "model keeps returning bad categories" problem that turned out to be my own allow-list missing a value the model was correctly using. The raw output showed it in one look; the parsed log had already thrown the evidence away.
Rule 4: regression fixtures from real failures
Every validation failure that reaches review becomes a fixture: the raw output, the expected verdict, and the reason. The fixture set runs against the validation step whenever I change it — a new allowed category, a tightened range, a new field.
This matters because validation logic drifts. Someone adds a category to the prompt and forgets the allow-list. Someone widens a range "temporarily". The fixtures catch the drift before production does, and they do it without calling the model at all: the checks are deterministic, so they are fast and free to run.
The fixture set started with five real failures. It now catches roughly one regression a month, usually mine, usually on a Friday. That is a good trade.
The checklist I actually use
- Strict schema first: types, required fields, no unexpected fields.
- Meaning checks second: allow-lists, ranges, real dates, IDs that exist.
- Validation failure goes to a review queue with raw output, errors, and input — not to an automatic retry.
- One deterministic cleanup pass for format-only failures; anything else goes to review.
- Every real failure becomes a regression fixture; fixtures run on every validation change.
- Log raw output on every run, success or failure.
It is not glamorous work. No part of it makes the model smarter. But it is the difference between a workflow that processes LLM output and a workflow that trusts it — and in production, that difference shows up as messages in the wrong queue, quietly, for weeks, while every log says success.
I'm building AI Automation Lab — production-ready n8n workflow templates. The free AI Hook Generator shows how I structure real workflows; my open-source templates live at github.com/zhp910318/n8n-ai-automation-lab.
Top comments (0)