AI is unreliable by design. It hallucinates, drifts off-prompt, and—when you ask for structured output—often returns JSON that doesn’t parse or doesn’t match your schema. In production, one bad response can break a flow, confuse a user, or trigger a support ticket. So if you’re building with LLMs, validation isn’t optional; it’s part of the architecture.
This post is a practical validation strategy: why it matters, what kinds of validation you need, where to put it, and how to keep it fast and maintainable.
Why validation matters
Production failures are expensive.
A parsing error in a critical path can mean a failed checkout, a broken dashboard, or a silent wrong answer. Downtime and rollbacks cost time and trust.
User trust is fragile.
One visible bug—“Sorry, something went wrong”—undermines confidence. Users don’t care that the model returned a trailing comma; they care that your app didn’t handle it.
Examples of what goes wrong:
-
Parse error: The LLM returns
{"name": "John",}. Your code doesJSON.parse(llmOutput)and crashes. -
Wrong type: The schema says
ageis a number; the LLM returns"age": "twenty-five". Your validator fails or your downstream logic breaks. -
Missing field: You require
email. The model omits it. You either fail the request or pass incomplete data into your system.
Validation catches these before they reach business logic. The goal is to either get valid, schema-conforming data or fail in a controlled, debuggable way.
Types of validation
Not all validation is the same. It helps to separate three layers.
1. Syntax validation — Is it valid JSON?
Can you parse it? No trailing commas, no unquoted keys, no single quotes. If this fails, you don’t have a data structure at all.
2. Schema validation — Does it match the shape?
Given a JSON Schema (or similar), does the parsed object have the right properties and types? Are required fields present? This is where you enforce the contract between the LLM and your app.
3. Semantic validation — Does it make sense?
Domain rules: “email must look like an email,” “date must be in the future,” “status must be one of these enums.” This is usually custom logic in your code, after syntax and schema are satisfied.
For most LLM pipelines, syntax + schema are the foundation. Get those right first; add semantic checks where needed.
Where to validate: the middleware layer
The right place for syntax and schema validation is between the LLM and your application—a thin middleware layer that:
- Receives the raw LLM output (string).
- Optionally extracts JSON from prose or markdown.
- Repairs syntax if possible (trailing commas, quotes, etc.).
- Validates (or enforces) against your JSON Schema.
- Returns either valid data or a clear error.
Your app then consumes only validated, typed data. You don’t scatter try/catch and regex across the codebase; you have one place that either succeeds or fails with a structured error.
Performance matters.
This layer runs on every LLM response. If it adds 100ms, users notice. Aim for sub-10ms so that validation is effectively free compared to the LLM call. That usually means a dedicated service (or a very fast library) rather than a heavyweight script.
Error handling.
Decide your policy: retry (e.g. once with a “fix your JSON” prompt), repair (if the service can fix syntax and enforce schema), or fail fast with a clear error to the user or to your monitoring. A repair step often gives the best balance: fewer failed requests, fewer retries, happier users.
Tools and approaches
DIY: regex and custom parsers.
You can strip trailing commas, normalize quotes, and even extract JSON from markdown with regex and a bit of parsing. It works for simple cases but gets brittle: edge cases, nested structures, and escape sequences. Maintenance cost is high.
Libraries: JSON Schema validators.
Use a solid validator (e.g. Ajv, jsonschema) to check parsed JSON against a schema. Great for “is this valid?” They don’t repair; they don’t extract. So you still need to fix syntax and strip prose yourself.
API services: validation + repair + extraction.
Offload the whole pipeline to a service that accepts raw LLM output and returns valid, schema-conforming data (or a clear error). You keep your code simple and get repair, extraction, and enforcement in one place. Latency stays low if the API is built for speed (e.g. Rust, edge deployment).
JSON Guardian is built for this flow. You send the raw string; you get back:
- Validate — Check against JSON Schema (Draft 7).
- Repair — Fix trailing commas, single quotes, unquoted keys, and similar.
-
Enforce — Repair first, then enforce schema with type coercion (e.g.
"twenty-five"→25). - Extract — Pull JSON out of markdown or prose.
- Partial — Complete partial/streaming JSON for real-time UI.
- Batch — Run any of the above on multiple items in one request.
So you can do: extract → repair → enforce in sequence, or call a single endpoint that fits your step. Free tier: 10,000 requests/month at jsonguardian.com; also on RapidAPI.
Summary
| Concern | What to do |
|---|---|
| Why | Avoid production failures and protect user trust; one validation layer catches parse and schema errors. |
| What | Syntax validation (valid JSON), schema validation (shape + types), then semantic rules if needed. |
| Where | Between LLM and app—a thin middleware that extracts, repairs, validates/enforces. |
| How | Keep it fast (sub-10ms); choose retry, repair, or fail-fast; prefer a dedicated service over DIY for repair + schema. |
Validation is non-negotiable for production AI. Start with syntax and schema; add extraction and streaming support as your use cases grow. Your future self (and your users) will thank you.
Try JSON Guardian: jsonguardian.com · RapidAPI · Free tier: 10k requests/month
Top comments (0)