When an LLM response fails validation, feed the error back into the retry

If you ask an LLM for structured output and validate it against a schema, you already know the failure
mode: most of the time it is fine, and every so often it hands back something that does not parse or
misses a required field. The usual reflex is to wrap the call in a retry and move on.

The problem is that a plain retry is the same prompt, the same temperature, roughly the same odds. You
are paying for another round and hoping the dice land differently.

There is a better move, and it is almost free to add: when validation fails, put the validation error
and the model's own bad output back into the next prompt, and ask it to fix that specific thing.

Here is the core of the loop I used for this on a RAG platform I built (trimmed to the essential path):

while attempts < max_attempts:
    try:
        msgs = [messages] if isinstance(messages, str) else messages
        if error_message:
            msgs = [*msgs, error_message]        # last attempt's error rides along
        response = await make_completion_request(..., messages=msgs)
        if validator:
            validator(response)                  # raises ValidationError on bad output
        return response
    except ValidationError as e:
        attempts += 1
        error_message = f"""
        The last response from the API failed validation due to the following error:
            <error>{format_error_for_llm(e)}</error>
        Your task is to fix the error and return the corrected response data:
            <data>{serialize(response).decode()}</data>
        """
        response = None

Two details do the work:

1. The error is described for the model, not for a log. format_error_for_llm turns the raw validation exception into a plain instruction ("field X must be an integer, you sent a string"). The model is good at patching a concrete, named mistake; it is bad at guessing why an opaque retry keeps failing.
2. You hand back its own previous output as the thing to correct. It is not regenerating from scratch, it is editing. That keeps the parts that were already right and usually fixes the one field that was wrong on the first pass.

The tradeoffs, because there always are some:

• It costs an extra call on a bad response, and the follow-up prompt is longer (it carries the error plus the prior payload). On a schema that fails often, that adds up. Cap the attempts.
• It only works when the bad response is parseable enough to serialize back into the prompt. Truly empty or truncated output has nothing to correct, so you still need a normal retry underneath.
• Prompt semantics do not always transfer if you also fail over between providers mid-loop. If you do that, do not count a provider swap as a real attempt.

That is the whole idea. It is not a framework, it is about ten lines around a call you already have. If
you are generating structured output at any volume, it turns a chunk of your "model was flaky" retries
into first-try-after-feedback successes.

Comments

[Tae Kim]

The error-as-input pattern is the right move and I've run a near-identical loop in production — the key insight is that you're giving the model a concrete, named mistake to fix rather than asking it to regenerate from scratch, which is a very different (and much cheaper) task. One thing I'd add: when the error message itself is ambiguous, wrapping format_error_for_llm to output the specific field path alongside the constraint tends to cut second-failure rates significantly, because the model is much better at targeting a path like data.items[0].price than a schema-level description.

[Vasyl]

Ran nearly the same loop in C# and the thing that surprised me was what the failure logs taught me after a few weeks: one enum field caused the majority of retries. Rewriting that field's description in the schema cut the retry rate more than anything I did to the loop itself. So I'd add: track first-try-pass rate per field over time. The feedback loop fixes individual responses, but the aggregate stats tell you where the prompt or schema is actually broken. Do you keep those failures around, or treat them as fire-and-forget?

[mote]

The 'editing not regenerating' framing is the key insight here — keeping what was already correct is obvious in hindsight but easy to miss when you're in a retry loop.

One thing worth separating: format/parse failures (the model can self-correct) vs semantic failures (the model keeps hallucinating the same wrong field). For the second kind, this retry loop might just converge slowly on the same wrong answer with different wording.

Have you thought about tagging failures by type and routing them differently? Like — parse errors get the self-correct loop, but missing required fields that look like hallucinated values could route to a retrieval step instead, or at least trigger a different system prompt hint.

The error-as-context trick is something we've been exploring in a multimodal storage layer for agents too — storing the validation failure path alongside the corrected response gives the next context window a trace of what went wrong before. Useful signal for long-running agent loops.