LLM Returns Invalid JSON Once in a While: Structured Outputs vs Validate-and-Retry

TL;DR: If json.loads() on model output fails only occasionally, there are three distinct causes: truncation, stray prose or code fences around the JSON, and schema drift (valid JSON, wrong shape). Provider-side structured outputs (constrained decoding) eliminate the second and third for supported schemas; a validate-and-retry loop with a typed model covers everything else, including the first. In production you usually want both.

What does the failure actually look like?

The symptom that sends people to a search engine is usually one of these:

json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
json.decoder.JSONDecodeError: Unterminated string starting at: line 1 column 140 (char 139)
pydantic_core._pydantic_core.ValidationError: 1 validation error for Ticket
priority
  Input should be less than or equal to 4 [type=less_than_equal, ...]

They look alike but have different fixes:

Symptom	Real cause	Right fix
Expecting value: line 1 column 1	Model wrapped the JSON in a code fence or added "Here is the JSON:"	Constrained decoding, or extract the outermost braces
Unterminated string / Expecting ',' delimiter near the end	Output hit max_tokens mid-object (truncation)	Raise the limit, shrink the schema, check the stop reason
Parses fine, then your code crashes on a missing key or odd enum value	Schema drift: valid JSON, wrong shape	Typed validation, or a strict schema on the provider side

The first thing to do before changing any prompt: log the raw response and the stop/finish reason for every failure. Most "the model is flaky" reports I have seen turned out to be truncation, which no prompt wording can fix.

Takeaway: classify the failure from the raw text and the stop reason before you touch the prompt.

Step 1: Validate with a typed model instead of json.loads

Parsing and validating in one step gives you an error message precise enough to feed back to the model. This uses Pydantic v2 and works with any provider, because call_model is just a function you supply.

from typing import Callable, Literal
from pydantic import BaseModel, Field


class Ticket(BaseModel):
    category: Literal["bug", "billing", "feature_request"]
    priority: int = Field(ge=1, le=4)
    summary: str = Field(min_length=1, max_length=200)


def extract_json_object(raw: str) -> str:
    """Strip code fences / chatter by taking the outermost braces."""
    start, end = raw.find("{"), raw.rfind("}")
    if start == -1 or end <= start:
        raise ValueError("no JSON object found in response")
    return raw[start : end + 1]


def parse_ticket(raw: str) -> Ticket:
    return Ticket.model_validate_json(extract_json_object(raw))

The brace extraction is a blunt tool: it will not rescue truncated output, and it can mis-slice if the model emits two objects. That is fine, because the retry loop below catches those cases.

Step 2: Retry with the error, not with the same prompt

Resending the identical request is a coin flip. Sending the validation error back gives the model something concrete to correct. In Pydantic v2, ValidationError subclasses ValueError, so one except covers both parse and schema failures.

Message = dict[str, str]


def extract_ticket(
    call_model: Callable[[list[Message]], str],
    prompt: str,
    max_attempts: int = 3,
) -> Ticket:
    messages: list[Message] = [{"role": "user", "content": prompt}]
    last_error: ValueError | None = None

    for _ in range(max_attempts):
        raw = call_model(messages)
        try:
            return parse_ticket(raw)
        except ValueError as exc:
            last_error = exc
            messages += [
                {"role": "assistant", "content": raw},
                {
                    "role": "user",
                    "content": (
                        "That response failed validation:\n"
                        f"{exc}\n"
                        "Reply with only a corrected JSON object."
                    ),
                },
            ]

    raise RuntimeError(f"no valid output after {max_attempts} attempts") from last_error

Two details that matter. Cap attempts at 2–3: each retry costs a full round trip and re-sends the growing conversation, so an uncapped loop turns one bad input into a bill spike. And if the failure was truncation, retrying with the same max_tokens will truncate again, so check the stop reason and raise the limit instead of looping.

Takeaway: a retry that includes the exact validation error converges; a retry that repeats the prompt just re-rolls the dice.

When do provider-side structured outputs replace the loop?

Most major providers now offer some form of schema-constrained output: OpenAI's Structured Outputs with a strict JSON schema, tool/function calling with an input schema on Anthropic's API, and grammar- or schema-constrained decoding in local runtimes such as llama.cpp and in libraries like Outlines. The model's token choices are restricted so the output cannot violate the schema's syntax. As of late 2026 the details differ per provider, so check current docs for supported schema features.

What this buys you: no code fences, no prose preamble, no invalid enum values, no missing required keys. If you want this without writing the plumbing yourself, OpenAI's Structured Outputs is the option that guarantees schema adherence on a strict schema, at the cost of being tied to that API.

What it does not buy you:

• Truncation is still possible. If generation stops at the token limit, the object is still cut off. Strict mode does not make that safe.
• Only a schema subset is supported. Constraints like minLength, maximum, or deeply recursive structures are often unsupported or ignored, so your Field(ge=1, le=4) check may still need to run in your own code.
• Syntax is not semantics. A schema-valid priority: 4 can still be the wrong priority. Constrained decoding makes the shape certain, not the content correct.
• First-request latency. Some providers compile the schema on first use and cache it, so a rarely used schema can be slower on its first call.

Takeaway: constrained decoding guarantees shape, not truncation-safety and not correctness, so keep your own validation behind it.

Which approach should you use?

Situation	Use
Hosted model, simple flat schema	Provider structured outputs plus Pydantic validation
Provider lacks strict mode, or schema uses unsupported constraints	Validate-and-retry loop
Local model (llama.cpp, Ollama-style runtimes)	Grammar/schema-constrained decoding if your runtime supports it, retry loop as backstop
Long outputs (big arrays, many records)	Split into smaller calls; truncation risk grows with output length

FAQ

Why does the LLM return invalid JSON only sometimes?
Sampling is probabilistic, so a model that follows your format 98% of the time will still occasionally add a code fence, stop mid-object at the token limit, or pick an off-schema value. Intermittent failures are normal without constrained decoding.

Does asking for JSON in the prompt guarantee valid JSON?
No. A prompt instruction raises the odds but guarantees nothing; only schema-constrained decoding restricts the output tokens, and even then truncation can break the object.

How many retries should I allow for LLM JSON parsing?
Two or three. If the third attempt still fails, the cause is usually truncation or an ambiguous schema, and more retries will not help.

Bottom line

If you call a hosted model with a simple schema, turn on the provider's strict structured output and keep a Pydantic model behind it for the constraints the schema cannot express. If you are on a provider or local model without it, the validate-and-retry loop above is enough for most small-team workloads. Either way, log the raw response and stop reason so you can tell truncation from drift. Cap retries, because the loop's failure mode is cost, not correctness.

Related reading

• GitHub Actions vs CircleCI vs Buildkite: The Real Cost of "Free" CI Minutes
• npm ci Says package.json and package-lock.json Are Not in Sync: Causes, Fixes, and How pnpm and Yarn Compare
• Zapier vs Make vs n8n: When Paying Per Task Stops Making Sense

Comments

[Launch Gate]

The taxonomy table is doing more work than it looks: the error's position is the classifier, not just the symptom. Expecting value: line 1 column 1 means the input was wrong from byte zero; Unterminated string landing right around your max_tokens boundary means the stream was cut. That rule generalizes past LLM output. In CSV parsing the same shape shows up: a decode error at byte 0 is an encoding or BOM problem, a parse error on the final line of an otherwise clean file is a truncated transfer, and a file that parses but lands every row in a single column is schema drift in a different costume (the delimiter was sniffed wrong). Position first, fix second.

One refinement for the retry loop: Pydantic's ValidationError carries loc, and JSONDecodeError carries pos, so the retry prompt can point instead of paste. Echoing the couple hundred characters around the failure offset ("the error is here: ...") focuses the model on the actual break and keeps the retry cheap when the original output was long. Resending the full raw output works, but on multi-thousand-token responses the repair prompt gets expensive fast, and the pointer version converges in about the same number of rounds.

Also worth stating out loud: validate-and-retry catches shape, not truth. A schema-valid ticket with a hallucinated priority passes every check in the loop. If anything downstream acts on the values, a sanity bound on the values themselves is a separate layer from the parse.