Taming Unstructured LLM Output with a Validate-and-Retry Agent

How a 400-line React app guarantees schema-perfect JSON from any LLM — no backend, no JSON-mode, just the human trick agents use: check your own work and retry.

1. The Problem

LLMs are wonderful language models and terrible format enforcers. Ask one for a JSON object and you'll get markdown fences, prose, trailing commas, null where a number belongs, and an enum value that was never in the list.

For a chatbot that's cosmetic. For a data pipeline, a form builder, or an API integration, one malformed field is a hard failure.

The usual fixes:

• JSON mode / response_format — works, but locks you to specific providers (OpenAI's json_schema, Anthropic's tool-forcing) and still doesn't guarantee every field matches your business rules.
• "Lazy JSON" — JSON.parse(response.match(/\{(.*)\}/s)) — noticeably slower (LLMs generate JSON faster when they don't pad it) and famously brittle.
• Schemas on one side, parsing on the other — a backend that hugs the provider and never tells the model what it did wrong.

None of them communicate. Which brings us to the approach this project is built around.

---

2. The Idea: Don't Trust — Verify

Production agent harnesses (and good junior engineers) share one habit: check your own work, and if it's wrong, get the error and fix it — don't recompute blindly.

The Structured Output Validator turns that habit into a loop:

┌─────────────┐      ┌──────────────┐      ┌─────────────┐
│  LLM Call   │ ───▶ │  Parse JSON  │ ───▶ │  Validate   │
└─────────────┘      └──────────────┘      └─────────────┘
                                                │  │
                 invalid / parse error         │  │  valid  ──▶ ✅ done
                                                ▼  │
                                        ┌───────────▼───┐
                                        │    Retry     │ ──▶ back to LLM Call
                                        └───────────────┘

Rather than asking the model to "do better" on the second try, the agent sends the exact validation errors from attempt N back as a new user turn — the same way you'd hand a failing PR back to a teammate with the linter output attached.

---

3. Architecture — No Backend Required

The entire loop runs in the browser. There is no server, no proxy, no database. The only network I/O is a direct fetch() from the page to the provider of your choice.

┌─ App.jsx (React, client state) ─────────────────────────────┐
│  provider · apiKey · model · baseUrl · schema · prompt      │
│  maxRetries · attempts[] · stage · finalStatus              │
│                                                             │
│   run()  ──▶  ┌──────────────────────────────────────────┐  │
│               │  for i in 0..maxRetries:                 │  │
│               │    if retry: append correction turn      │  │
│               │    raw    = callLLM(config, messages)    │  │
│               │    parsed = extractJSON(raw)             │  │
│               │    errors = validateJSON(parsed, schema) │  │
│               │    errors==[] → VALID, stop              │  │
│               │    else      → INVALID, retry            │  │
│               └──────────────────────────────────────────┘  │
│                        ▲            │                     │
│                        │ fetch()    │ fetch()              │
└───────────────┬────────────────┼────┴───────────┬─────────┘
                │                │                │
   ┌────────────▼───┐           ◀                    ┌────────────▼────────┐
   │  src/lib.js    │   ............ HTTPS ......... │    LLM provider    │
   │  pure, NO React │   OpenAI / Anthropic /       │  Ollama / LM Studio │
   │  - SCHEMAS      │   custom OpenAI-compatible   │  (local, no key)    │
   │  - validateJSON │                               └─────────────────────┘
   │  - extractJSON  │
   │  - callLLM      │
   └─────────────────┘

The design deliberately separates concerns:

• src/lib.js — pure, framework-free functions. Testable with Node in five seconds, no DOM, no React import.
• src/App.jsx — React components + the stateful agent loop + the dark-terminal UI.

This split is the single most useful engineering decision here: the core algorithm can be unit-tested, reused in a CLI, or ported to TypeScript without touching the UI.

---

4. Core Logic, Walked Through

4.1 Schema as declarative data

A schema is just an object of fields, each with a rule set:

weather: {
  label: "Weather Report",
  fields: {
    city:           { type: "string", minLength: 2 },
    temperature_c:  { type: "number", min: -90, max: 60 },
    condition:      { type: "enum", values: ["sunny","cloudy","rainy","snowy","windy"] },
    humidity_pct:   { type: "number", min: 0, max: 100 },
  },
  example_prompt: "Generate a current weather report for Tokyo.",
}

Because the schema is data, the same structure drives four different things: the sidebar preview, the system-prompt generator, the validator, and the example prompts.

4.2 The validator — validateJSON(data, schema)

Returns an array of human-readable error strings. Empty array means valid.

export function validateJSON(data, schema) {
  const errors = [];
  for (const [field, rule] of Object.entries(schema.fields)) {
    const val = data[field];
    if (val === undefined || val === null) {
      errors.push(`Missing required field: "${field}"`);
      continue;
    }
    if (rule.type === "string") {
      if (typeof val !== "string")
        errors.push(`"${field}" must be a string, got ${typeof val}`);
      else if (rule.minLength && val.length < rule.minLength)
        errors.push(`"${field}" must be at least ${rule.minLength} characters...`);
      else if (rule.pattern && !val.includes(rule.pattern))
        errors.push(`"${field}" must contain "${rule.pattern}"...`);
    } else if (rule.type === "number") {
      if (typeof val !== "number")
        errors.push(`"${field}" must be a number, got ${typeof val}`);
      else if (rule.min !== undefined && val < rule.min)
        errors.push(`"${field}" must be ≥ ${rule.min} (got ${val})`);
      else if (rule.max !== undefined && val > rule.max)
        errors.push(`"${field}" must be ≤ ${rule.max} (got ${val})`);
    }
    // ... boolean and enum cases
  }
  return errors;
}

The practical payoff is in the error strings — they're not just booleans, they're feedback the model can act on. That's the whole trick: the message you show the human is (nearly) the message you feed back to the agent.

4.3 The extractor — extractJSON(text)

The model won't always play nice, so parsing is tolerant — three strategies in order of trust:

export function extractJSON(text) {
  // 1. fenced json block
  const fenced = text.match(/```
{% endraw %}
(?:json)?\s*([\s\S]*?)
{% raw %}
```/);
  if (fenced) { try { return JSON.parse(fenced[1].trim()); } catch {} }
  // 2. the entire raw response
  try { return JSON.parse(text.trim()); } catch {}
  // 3. the first { ... } substring
  const braceMatch = text.match(/\{[\s\S]*\}/);
  if (braceMatch) { try { return JSON.parse(braceMatch[0]); } catch {} }
  return null;
}

null → the agent records a PARSE ERROR and retries with a stricter nudge.

4.4 The retry loop — the heart of the app

for (let i = 0; i < maxRetries; i++) {
  if (i > 0) {
    const retryMsg = buildRetryMessage(lastErrors);   // lists the exact failures
    messages.push({ role: "assistant", content: "" }); // model's (broken) reply slot
    messages.push({ role: "user", content: retryMsg }); // correction as a user turn
  }
  const raw = await callLLM(config, messages);
  const parsed = extractJSON(raw);
  if (!parsed) { /* record PARSE_ERROR, continue */ }
  const errors = validateJSON(parsed, schema);
  if (errors.length === 0) { /* record VALID, break */ }
  else { /* record INVALID + errors, continue */ }
}

Key details that matter:

1. Multi-turn, not re-prompt. The correction is appended as a new user message after the model's own failed answer. Providers like Anthropic treat the assistant turn as the model's previous output; squeezing both into one request preserves conversation context the model uses to self-correct.
2. The error list is the feedback. No vague "please fix it" — the model is told "age" must be ≥ 0 (got -5).
3. Early stop. The instant validation passes, the loop breaks. Latency is recorded per attempt so you can see the cost of each correction round.
4. Everything is visible. Each attempt is a collapsible card showing parsed JSON, the error list, and the exact retry message sent — so a failed run teaches you why it failed.

---

5. Provider Abstraction — One fetch(), Five Providers

callLLM branches on provider:

if (provider === "anthropic") {
  url = "https://api.anthropic.com/v1/messages";
  headers = { "Content-Type": "application/json", "x-api-key": apiKey,
              "anthropic-version": "2023-06-01" };
  body = JSON.stringify({
    model, max_tokens: 1024,
    system: systemPrompt,          // split out of the messages array
    messages: userMessages,
  });
} else {
  // OpenAI-compatible: OpenAI, Ollama, LM Studio, custom
  url = `${base}/v1/chat/completions`;
  headers = { "Content-Type": "application/json",
              ...(apiKey ? { Authorization: `Bearer ${apiKey}` } : {}) };
  body = JSON.stringify({ model, messages, temperature: 0.7 });
}

The two gotchas worth internalizing:

• Anthropic keeps system separate. It won't accept a role: "system" message in the messages array — you must pull it out into the system field and pass only user/assistant turns. This is also why max_tokens is required (Anthropic rejects requests without it).
• Local servers get no auth header. Ollama / LM Studio speak the OpenAI-compatible format but don't want Authorization, so the header is added conditionally.

---

6. The UI — A Dashboard for an Agent's Thought Process

The design goal was transparency: treat every attempt as a post-mortem artifact, not a black box.

Left sidebar (~340px): provider config, API key, model, schema selector with a live field-preview, prompt textarea (auto-filled per schema), a 1–5 max-attempts slider, RUN, RESET.

Right main panel:

• Pipeline diagram — four nodes wired with arrows. The active stage pulses amber (#f0c040), completed stages lock in teal (#0d7377). When the loop finishes you see the whole path light up — a literal visualization of the algorithm from §2.
• Result banner — green ✓ Valid output in N attempts or pink-red ✗ Failed after N attempts.
• Attempt log — one collapsible card per round: badge (VALID / INVALID / PARSE ERROR / RUNNING), latency in ms, pretty-printed parsed JSON on the left, validation verdict on the right, and the exact correction prompt beneath when a retry was triggered.

Dark terminal theme:

Token	Hex
Background	#0a0e14
Text	#c5cdd9
Accent (buttons/active)	#0d7377
Success	#7fffd4
Error	#ff6b8a
Warning / running	#f0c040
Borders	#1e2530

Sticking to a small, explicit palette keeps the "terminal" feel cohesive and makes state changes (running → valid → failed) instantly legible at a glance.

---

7. What Actually Runs (the state machine)

The app is really a small state machine exposed through React state:

idle → calling → parsing → validating → (retrying | done_valid | done_fail)

stage drives the pipeline diagram; attempts[] renders the log; finalStatus renders the banner. abortRef gives RESET a way to interrupt an in-flight loop between attempts. Because the loop is await-serialized over fetch(), there's no concurrency to race — old-but-simple beats clever-but-racy for a tool like this.

---

8. Tailwind-Free, Zero-Dependency UI

The app leans on inline style objects plus a small <style> block for keyframes and scrollbar theming. No CSS framework, no component library, no state library beyond React itself. The only external runtime dependency is React. That's a deliberate constraint: the whole point is that a single readable App.jsx tells the full UI story.

---

9. Lessons & Trade-offs (the honest section)

• extractJSON strategy #3 is a double-edged sword. Grabbing {...} greedily can over-capture if the model emits two JSON objects or a JSON-with-prose sandwich. For this use case (single object out), it's the right call; for arrays or multi-object output you'd want delimiter-based extraction.
• Retries cost LLM tokens. Each correction round re-sends the whole conversation. An agent that fails 5 times burns ~5× the prompt tokens. That's why the loop runs client-side with 1–5 attempts — you trade a few cents of tokens for a deterministic guarantee.
• temperature isn't the problem. The model usually knows the rules; it drifts on format. Sending it its own failing output + the diff is what fixes it — temperature tuning barely moves the needle.
• Validation schema ≠ LLM schema. One natural next step is feeding the full JSON Schema to providers that support response_format and keeping validateJSON as the client-side safety net. Belt and suspenders.

---

10. Where To Go From Here

Contributor-sized ideas, roughly in value order:

1. OpenAI strict json_schema / Anthropic tool-forcing to raise first-attempt success from "usually" to "almost always," with validateJSON as a fallback.
2. Richer rule set — arrays, nested objects, regex, integer-only, uniqueness.
3. Usage tracking — read usage from replies; show cumulative tokens/cost.
4. Streaming — validate as chunks arrive via ReadableStream.
5. Mock mode — deterministic fixtures to demo/test without an API key.
6. TypeScript port — SchemaField unions make SCHEMAS fully type-checkable.
7. Vitest tests — validateJSON / extractJSON are pure; this is a 10-minute job.
8. Run-report export — download the whole attempt log as JSON/Markdown.

The lesson this project turns into an app: format problems don't need better prompting — they need verification. By making validation explicit, visible, and feed-back-able, a generic LLM becomes a dependable structured-output machine, completely client-side, provider-agnostic, and free to run locally.

If you want to poke at it: clone the repo, point it at Ollama or LM Studio, and watch the agent catch its own mistakes on screen.

Code & more: https://www.dailybuild.xyz/project/220-structured-output-validator