Valid JSON, Wrong Answer: Semantic Checks for LLM Output in n8n

This is part of my build-in-public series on running LLM workflows in n8n in production. Everything here is from real templates I sell and support — the mistakes are mine, the fixes are tested.

The schema passed. The answer was still wrong

A support-triage workflow of mine had every guard I thought it needed. The LLM returned strict JSON. A schema node validated the shape: category was a string, priority was one of low, medium, high, amount was a number. Every field present, every type correct, zero validation errors for weeks.

Then a customer wrote in about a double charge, and the workflow filed it as category: "feature_request", priority: "low". The JSON was perfect. The meaning was nonsense. Nobody noticed for days, because the schema node was green and green means done — the same blind spot I wrote about with alerts that never fire. A schema checks that the answer has the right shape. It says nothing about whether the answer is true.

This is the failure mode that survives every shape-level fix: the model is confident, fluent, and wrong in a way that still parses. If your only check is "did it return valid JSON," you are testing the envelope, not the letter.

Check meaning with rules the model cannot argue with

The fix is a second layer of checks that do not come from the model at all. They come from the input itself, from deterministic rules, and from the relationships between fields.

First, cross-field consistency. If category is refund but the message contains no money, no order number, and no word like "charged" or "refund," that is a contradiction, not a judgment call. If priority is high but amount is zero and no urgency words appear, something is off. These checks are cheap — a keyword list, a regex, a comparison between two fields — and they catch exactly the class of error a schema cannot see: two individually valid values that cannot both be true.

Second, plausibility ranges from the business, not the prompt. An extracted amount of $10,000 on a $29 product is not a rounding error; it is a misread decimal or a hallucinated zero. A date in the past for a "schedule for tomorrow" request is a misread. Hard-code the ranges your business actually has — product prices, valid regions, working hours — and reject anything outside them. The model does not know your price list. Your workflow should.

Third, re-derive, don't re-ask. When a check fails, the instinct is to retry the same prompt and hope for a better answer. Usually you get the same confident wrong answer back, at full token cost — the retry trap I wrote about with fallback chains. Instead, recompute the disputed field deterministically where you can: extract the amount with a regex from the original text, look up the category from a keyword map, compare. If the deterministic answer and the model answer disagree, you have learned something real: this input is ambiguous. Ambiguous inputs do not get retried. They go to review, with both answers attached, so a human decides once instead of a model guessing three times.

What "review" needs to carry

A semantic-check failure that lands in review without context is just another queue nobody watches. Each flagged item carries the original input excerpt, the model's answer, the rule that failed, and the deterministic re-derivation. That is enough for a human to decide in under a minute, and it is also a dataset: every resolved disagreement is a case you can add to a test set.

That test set is the real payoff. Once a month, I run new prompt or model changes against twenty known tricky inputs — sarcasm, mixed requests ("refund this and also upgrade that"), numbers written as words, dates without years — and count semantic mismatches, not schema errors. Schema errors are already zero; they tell me nothing. Semantic mismatches are the metric that actually moves. If a model change drops mismatches from five to two, keep it. If it raises them, roll back, even if the new model is faster and cheaper. Speed on wrong answers is not an improvement.

The rule I now build by: shape validation proves the model followed instructions. Semantic checks prove it understood the input. You need both, and only one of them can be written as a schema. The other has to be written as rules about your business — which is, inconveniently, the part no prompt can do for you.

---

I'm building AI Automation Lab — production-ready n8n workflow templates. The free AI Hook Generator shows how I structure real workflows; my open-source templates live at github.com/zhp910318/n8n-ai-automation-lab.