DEV Community

Arthur Doss
Arthur Doss

Posted on

Implementing Structured Output Retries Without Infinite Agent Loops

Structured outputs are one of the most useful capabilities in modern LLM applications.

Instead of returning unstructured text, a model can generate JSON objects that applications use for classification, data extraction, tool execution, and workflow decisions.

But structured output does not automatically mean reliable output.
A model may return valid JSON with missing fields, unsupported values, incorrect data types, or arguments that violate business rules.

The usual solution is to retry the model with an explanation of what went wrong.

That approach works until the retry logic becomes another source of failure.

An agent may repeatedly generate invalid responses, retry with nearly identical instructions, and continue consuming tokens without making progress.

Reliable structured output handling requires more than schema validation. It needs bounded retries, explicit failure states, and a clear distinction between recoverable formatting errors and application-level failures.

In this guide, we'll build a lightweight structured output retry mechanism using Python and Pydantic. We'll also add regression tests, retry budgets, failure classification, and safeguards against infinite agent loops.

1. Why Structured Output Retries Become Dangerous

Consider an AI agent that extracts invoice information from documents.

The application expects:

{
  "invoice_id": "INV-1042",
  "amount": 1250.50,
  "currency": "USD",
  "approved": false
}
Enter fullscreen mode Exit fullscreen mode

The response must satisfy several requirements:

  • invoice_id must be a non-empty string.
  • amount must be greater than zero.
  • currency must use a supported currency code.
  • approved must be a Boolean.

Now imagine the model returns:

{
  "invoice_id": "INV-1042",
  "amount": "unknown",
  "currency": "USD",
  "approved": "maybe"
}
Enter fullscreen mode Exit fullscreen mode

The JSON syntax is valid, but the data violates the expected schema.

A developer might implement:

while True:
    response = call_model()

    try:
        return validate(response)
    except Exception:
        continue
Enter fullscreen mode Exit fullscreen mode

This code has several problems.

First, the loop has no termination condition.

Second, it retries on every exception, including programming errors that another model call cannot fix.

Third, it provides no feedback explaining why validation failed.
Fourth, it may repeatedly generate the same invalid response.

In a production agent, this can increase cost, latency, and infrastructure usage while preventing the workflow from completing.

The solution is to make retries an explicit part of the application's control flow.

2. Separate JSON Parsing From Schema Validation

Structured output failures occur at different levels.

JSON syntax errors

The response cannot be parsed as JSON.

Example:

{
  "invoice_id": "INV-1042",
  "amount": 1250.50,
}
Enter fullscreen mode Exit fullscreen mode

The trailing comma makes this invalid JSON.

Schema validation errors

The JSON parses successfully but does not match the expected structure.

Example:

{
  "invoice_id": "INV-1042",
  "amount": "unknown"
}

Enter fullscreen mode Exit fullscreen mode

Required fields are missing or contain invalid values.

Business-rule violations

The output matches the schema but violates an application-specific requirement.

For example, an invoice may have a valid positive amount but exceed the amount the user is authorized to approve.

Tool execution failures

The structured output may be correct, but the external API rejects the operation or becomes unavailable.

These failures should not share the same retry policy. A formatting problem may justify asking the model to regenerate its response.

An authorization failure should generally stop execution. A network timeout may require an infrastructure-level retry rather than another model generation.

The first design rule is to classify failures before deciding whether another LLM call is appropriate.

3. Project Structure

We'll build a small example with the following files:

structured-output-retries/
|
|-- models.py
|-- retry_handler.py
|-- test_retry_handler.py
|-- requirements.txt
|
-- .github/
-- workflows/
-- tests.yml`

The implementation uses Python 3.11 or later.

Install the dependencies:

shell
python -m pip install "pydantic>=2,<3" pytest

Create requirements.txt:

pydantic>=2,<3
pytest>=8,<10

The retry handler will use a callable model interface rather than depending on a specific LLM provider.

That makes the implementation easier to test.

4. Define a Strict Structured Output Schema

Create models.py:

`python
from typing import Literal

from pydantic import (
BaseModel,
ConfigDict,
Field
)

class InvoiceExtraction(BaseModel):
model_config = ConfigDict(
extra="forbid",
strict=True
)

invoice_id: str = Field(
    min_length=1
)

amount: float = Field(
    gt=0
)

currency: Literal[
    "USD",
    "EUR",
    "GBP"
]

approved: bool
Enter fullscreen mode Exit fullscreen mode

`
This schema enforces several constraints. The extra="forbid" configuration rejects unexpected fields. Strict validation prevents implicit type coercion where supported by the field type.

The amount must be positive. The currency must match one of the allowed values. The approved field must be a Boolean.

Why strict validation matters

Without strict validation, some frameworks may automatically convert values. For example, a string containing a number might be accepted as a numeric field.

That behavior can be convenient for ordinary form processing. For AI-generated outputs, however, it may hide inconsistent model behavior.
Strict validation makes failures more visible and easier to evaluate.

Schema validation is not authorization

The approved field is included to demonstrate Boolean validation.

It must not be treated as proof that a real invoice has been authorized. A production payment or approval system should obtain authorization from trusted application state and enforce it independently of the model-generated output.

5. Define Explicit Retry Limits

The retry mechanism needs a clear execution budget.

We'll use three controls:

  • Maximum model attempts.
  • Maximum repeated identical invalid responses.
  • A wall-clock deadline checked between attempts.

These controls address different failure patterns.

The attempt limit prevents unlimited model calls.

The repeated-response limit detects situations where the model produces the same invalid result without improvement.

The deadline prevents the retry handler from continuing after the allotted execution window.

A deadline check between calls cannot interrupt a model request that is already running. The model client must enforce its own request timeout.

Define retry-specific exceptions

Create retry_handler.py:

`python
import json
import time
from collections.abc import Callable
from dataclasses import dataclass

from pydantic import ValidationError

from models import InvoiceExtraction

class StructuredOutputError(Exception):
pass

class RetryBudgetExceeded(
StructuredOutputError
):
pass

class RepeatedInvalidOutput(
StructuredOutputError
):
pass

class RetryDeadlineExceeded(
StructuredOutputError
):
pass

@dataclass(frozen=True)
class RetryPolicy:
max_attempts: int = 3
max_identical_failures: int = 2
timeout_seconds: float = 30.0

def __post_init__(self):
    if self.max_attempts < 1:
        raise ValueError(
            "max_attempts must be positive"
        )

    if self.max_identical_failures < 1:
        raise ValueError(
            "max_identical_failures must be positive"
        )

    if self.timeout_seconds <= 0:
        raise ValueError(
            "timeout_seconds must be positive"
        )
Enter fullscreen mode Exit fullscreen mode

`
The policy is immutable. This helps prevent retry limits from being modified accidentally during execution.

6. Implement Structured Output Validation

Next, create a function that parses and validates the model response.

Add this to retry_handler.py:

`python
def parse_structured_output(
raw_response: str
) -> InvoiceExtraction:

try:
    payload = json.loads(raw_response)

except json.JSONDecodeError as exc:
    raise StructuredOutputError(
        f"Invalid JSON: {exc.msg}"
    ) from exc

try:
    return InvoiceExtraction.model_validate(
        payload
    )

except ValidationError as exc:
    errors = exc.errors(
        include_url=False,
        include_input=False
    )

    raise StructuredOutputError(
        f"Schema validation failed: {errors}"
    ) from exc
Enter fullscreen mode Exit fullscreen mode

`
This function handles two specific failure categories. If JSON parsing fails, it raises a structured output error.

If Pydantic validation fails, it raises a structured output error containing validation details.

Other unexpected exceptions are not converted into retryable errors.
That distinction is important. A programming error should not trigger repeated model calls.

7. Implement the Bounded Retry Handler

Now we can build the retry loop. The handler accepts a callable that generates a model response.

The callable receives optional validation feedback from the previous attempt.

Add this to retry_handler.py:

`python
def generate_with_retries(
model_call: Callable[
[str | None],
str
],
policy: RetryPolicy
) -> InvoiceExtraction:

started_at = time.monotonic()

previous_invalid_output = None
identical_failures = 0
validation_feedback = None

for attempt in range(
    1,
    policy.max_attempts + 1
):
    elapsed = (
        time.monotonic() - started_at
    )

    if elapsed >= policy.timeout_seconds:
        raise RetryDeadlineExceeded(
            "Retry deadline exceeded"
        )

    raw_response = model_call(
        validation_feedback
    )

    try:
        return parse_structured_output(
            raw_response
        )

    except StructuredOutputError as exc:
        if raw_response == previous_invalid_output:
            identical_failures += 1
        else:
            identical_failures = 1

        previous_invalid_output = raw_response

        if (
            identical_failures
            >= policy.max_identical_failures
        ):
            raise RepeatedInvalidOutput(
                "Model repeatedly returned "
                "the same invalid output"
            ) from exc

        validation_feedback = str(exc)

raise RetryBudgetExceeded(
    "Maximum structured output "
    "attempts exceeded"
)
Enter fullscreen mode Exit fullscreen mode

`
This implementation guarantees that the handler initiates no more than the configured number of model calls.

It also detects repeated identical invalid responses.

The retry loop terminates when:

  • The model produces valid structured output.
  • The attempt budget is exhausted.
  • The repeated-invalid-output limit is reached.
  • The deadline is exceeded before an attempt.

Why the loop is safer

There is no while True. The number of attempts is explicit.
Only structured output validation errors are retried. Unexpected model-client exceptions propagate to the caller.

The handler does not execute any external business action. It only produces validated data. That separation reduces the risk of repeating side effects during output repair.

8. Provide Validation Feedback to the Model

Retries are more useful when the model knows what failed.

Instead of repeating the original prompt unchanged, provide concise validation feedback.

For example:

`plaintext
Your previous response failed validation.

Errors:

  • amount must be a positive number
  • approved must be a Boolean
  • currency must be USD, EUR, or GBP

Return only a valid JSON object matching the schema.
`
The retry handler already passes validation feedback to the model callable.

A provider-specific adapter can use that feedback to construct the next request.

Conceptually:

`python
def model_call(
validation_feedback: str | None
) -> str:

messages = [
    {
        "role": "system",
        "content": (
            "Extract invoice information "
            "as valid JSON."
        )
    }
]

if validation_feedback:
    messages.append({
        "role": "user",
        "content": (
            "The previous output failed "
            "validation:\n"
            + validation_feedback
        )
    })

return call_llm_provider(messages)
Enter fullscreen mode Exit fullscreen mode

`
This is integration pseudocode.

The call_llm_provider function must be implemented using the selected model provider's SDK.

Avoid feeding unlimited error history back to the model

Some retry implementations append every failed response and validation error to the conversation.

Over time, the prompt becomes larger and more confusing.

A better approach is to keep only the information needed for the next repair attempt.

For example:

  • The original task.
  • The required schema.
  • The latest validation errors.
  • Relevant source data.

This keeps the repair process focused. Validation feedback should also be sanitized and bounded before being included in a prompt.

9. Test Successful Recovery After an Invalid Response

We can test the retry handler without calling a real LLM.

Create test_retry_handler.py:

`python
from retry_handler import (
RetryPolicy,
generate_with_retries,
RetryBudgetExceeded,
RepeatedInvalidOutput
)

def test_recovers_after_invalid_json():
responses = iter([
'{"invoice_id": "INV-1",',
(
'{"invoice_id": "INV-1", '
'"amount": 100.0, '
'"currency": "USD", '
'"approved": false}'
)
])

feedback_received = []

def fake_model(feedback):
    feedback_received.append(
        feedback
    )

    return next(responses)

result = generate_with_retries(
    fake_model,
    RetryPolicy(
        max_attempts=3,
        max_identical_failures=2
    )
)

assert result.invoice_id == "INV-1"
assert result.amount == 100.0
assert result.approved is False

assert feedback_received[0] is None
assert feedback_received[1] is not None
Enter fullscreen mode Exit fullscreen mode

`
The first response contains invalid JSON. The second response is valid. The test verifies that the retry handler recovers successfully and provides feedback after the first failure.

10. Test Retry Budget Exhaustion

A reliable retry mechanism must stop when the model continues producing invalid output.

Add this test:

`python
def test_retry_budget_exceeded():
responses = iter([
'{"amount": "invalid"}',
'{"currency": "unknown"}',
'{"approved": "maybe"}'
])

calls = 0

def fake_model(feedback):
    nonlocal calls
    calls += 1

    return next(responses)

try:
    generate_with_retries(
        fake_model,
        RetryPolicy(
            max_attempts=3,
            max_identical_failures=3
        )
    )

except RetryBudgetExceeded:
    pass

else:
    raise AssertionError(
        "Expected retry budget failure"
    )

assert calls == 3
Enter fullscreen mode Exit fullscreen mode

`
The model produces three different invalid responses. The handler stops after the configured maximum of three attempts. This prevents an infinite loop even when the model never returns valid data.

11. Detect Repeated Invalid Responses

Some models repeatedly produce the same malformed structure.

For example:

json
{
"invoice_id": "INV-1042",
"amount": "unknown"
}

If the model returns this exact response multiple times, another retry may not be useful.

Add the following test:

`python
def test_repeated_invalid_output():
calls = 0

def fake_model(feedback):
    nonlocal calls
    calls += 1

    return '{"amount": "unknown"}'

try:
    generate_with_retries(
        fake_model,
        RetryPolicy(
            max_attempts=5,
            max_identical_failures=2
        )
    )

except RepeatedInvalidOutput:
    pass

else:
    raise AssertionError(
        "Expected repeated output failure"
    )

assert calls == 2
Enter fullscreen mode Exit fullscreen mode

`
The handler stops after two identical invalid responses. This is a simple stagnation detector.

It does not detect semantically equivalent responses with different formatting. More advanced implementations may normalize JSON or compare validation-error patterns to identify repeated failures.

12. Separate Structured Output Retries From Tool Retries

One of the most important architectural decisions is separating model-output repair from external tool execution.

Consider this workflow:

plaintext
User Request
|
v
LLM Generates Tool Arguments
|
v
Schema Validation
|
v
Authorization Check
|
v
Tool Execution
|
v
Result

Structured output retries should occur before tool execution. If the model generates invalid arguments, the application can ask for corrected arguments.

Once a tool has executed, retrying the entire model workflow may repeat the external action.

Example: payment creation

Suppose an agent generates valid payment arguments and calls a payment API.

The API processes the payment, but the connection times out before the application receives confirmation.

The agent should not simply regenerate the arguments and submit another payment.

Instead, the execution layer should reconcile the operation using a stable operation identifier or provider-supported idempotency mechanism.

The retry policies are different:

  • Structured output retry: Regenerate invalid model output within a bounded attempt budget.
  • Network retry: Retry transient transport failures according to service-specific rules.
  • Business operation retry: Use idempotency and reconciliation to prevent duplicate effects.
  • Authorization failure: Stop execution rather than ask the model to try again.

Combining all four into one generic retry loop is a common source of agent reliability problems.

13. Add an Agent-Level Execution Budget

Even when every structured output call has a retry limit, an agent can still loop indefinitely at the workflow level.

For example:

plaintext
Generate Action
|
v
Validate Action
|
v
Execute Tool
|
v
Observe Result
|
v
Generate Next Action
|
+------> Repeat

Each individual model call may have a maximum of three attempts. But if the agent can generate unlimited actions, the overall workflow still has no bound.

Use separate budgets

A production agent should consider limits for:

  • Total model calls.
  • Structured output repair attempts.
  • Tool executions.
  • Workflow steps.
  • Total elapsed time.
  • Token consumption.
  • Estimated execution cost.

These limits should be enforced by the orchestration layer.

A simple agent-level budget might look like:

`python
from dataclasses import dataclass

@dataclass
class AgentBudget:
max_steps: int = 10
max_model_calls: int = 20
max_tool_calls: int = 8

steps: int = 0
model_calls: int = 0
tool_calls: int = 0

def consume_step(self):
    if self.steps >= self.max_steps:
        raise RuntimeError(
            "Agent step budget exceeded"
        )

    self.steps += 1

def consume_model_call(self):
    if self.model_calls >= self.max_model_calls:
        raise RuntimeError(
            "Model call budget exceeded"
        )

    self.model_calls += 1

def consume_tool_call(self):
    if self.tool_calls >= self.max_tool_calls:
        raise RuntimeError(
            "Tool call budget exceeded"
        )

    self.tool_calls += 1
Enter fullscreen mode Exit fullscreen mode

`
This is a minimal in-memory budget. A production implementation may need durable counters, concurrency controls, and budget persistence across workflow resumes.

The key point is that output repair and overall agent execution must have independent limits.

14. Handle Deadline and Timeout Behavior Correctly

A retry deadline is useful, but it must be implemented carefully. The earlier retry handler checks elapsed time before each model call.
That prevents new attempts after the deadline has passed.

However, it does not interrupt a model call that takes longer than the configured timeout.

For example, if the deadline is 30 seconds and a model request takes 60 seconds, the handler cannot stop the request merely by checking the clock before the call.

Enforce timeouts at the model client

The provider adapter should configure request-level timeouts.
For asynchronous applications, cancellation and deadline propagation may also be necessary.

A robust design distinguishes:

  • Request timeout: Maximum time allowed for an individual model API request.
  • Retry deadline: Maximum time allowed for the structured output repair operation.
  • Workflow deadline: Maximum time allowed for the complete agent task.

These limits should be coordinated so nested operations cannot exceed the overall execution budget.

15. Use Native Structured Output Features When Available

Many model providers offer structured output features that constrain responses to a specified schema.

These features can reduce malformed JSON and schema violations. However, they do not eliminate every reliability concern.

A model may still produce values that are syntactically valid but semantically incorrect.

For example, a model could return:

`json
{
"invoice_id": "INV-1042",
"amount": 1250.50,
"currency": "USD",
"approved": true
}

`
The structure is valid. But the approved value may not reflect an actual authorization decision.

Native structured output should therefore be combined with application-level validation.

A practical validation sequence

  1. Request structured output using the provider's supported schema mechanism.
  2. Parse the response using a trusted parser.
  3. Validate it against the application schema.
  4. Apply business rules and authorization checks.
  5. Retry only recoverable generation failures.
  6. Execute external tools only after validation succeeds.

This approach uses provider capabilities without making them the sole reliability boundary.

16. Monitor Retry Behavior in Production

Structured output retries should be observable. Otherwise, an application may appear functional while silently making multiple model calls for every successful request.

Useful metrics include:

  • First-attempt validation success rate: Percentage of requests that produce valid output without repair.
  • Average attempts per successful response: Indicates how frequently the application needs retries.
  • Retry budget exhaustion rate: Percentage of requests that fail after reaching the maximum attempt count.
  • Repeated-invalid-output rate: Frequency of stagnation failures.
  • Validation error categories: Missing fields, incorrect types, unsupported enum values, and malformed JSON.
  • Retry latency: Additional response time caused by repair attempts.
  • Retry token consumption: Extra model usage associated with unsuccessful generations.
  • Workflow budget failures: Agent tasks terminated because they exceeded execution limits.

These metrics help teams identify whether failures originate in prompts, schemas, model behavior, or orchestration logic.

17. Common Retry Implementation Mistakes

Several patterns can make structured output handling less reliable.

Retrying every exception

Programming errors, authorization failures, and external API failures should not automatically trigger another model generation. Catch only expected, recoverable validation failures.

Using unlimited loops

Every retry operation needs a finite attempt budget.

Repeating the same prompt without feedback

When appropriate, provide concise validation errors so the model can correct its output.

Appending unlimited failure history

Long retry conversations increase token usage and may introduce conflicting instructions. Keep repair context focused.

Retrying after side effects

Do not repeat external operations simply because the agent failed to receive confirmation. Use idempotency and reconciliation.

Treating schema validity as business correctness

A valid JSON object can still contain incorrect or unauthorized values. Apply independent business rules.

Ignoring overall agent budgets

Bounded structured output retries do not prevent infinite workflow loops. Enforce limits at both levels.

Assuming deadlines interrupt active requests

Timeouts must be enforced by the underlying client or execution environment.

18. Production Readiness Checklist

Before deploying structured output retries in an AI agent, verify the following:

  • Output schemas reject unexpected fields where appropriate.
  • JSON parsing and schema validation are handled separately.
  • Only recoverable validation failures trigger regeneration.
  • Retry attempts have an explicit maximum.
  • Repeated invalid responses are detected.
  • Validation feedback is concise and sanitized.
  • Model request timeouts are configured.
  • Retry deadlines are enforced between attempts.
  • Agent-level execution budgets are independent of repair budgets.
  • Tool execution occurs only after required validation.
  • Authorization is enforced outside the model.
  • Side-effecting operations use idempotency or reconciliation.
  • Retry metrics and failure categories are observable.
  • Regression tests cover successful repair and budget exhaustion.
  • Schema changes are evaluated before deployment.
  • The application returns an explicit failure when recovery is unsuccessful.

Conclusion

Structured output retries are useful because language models do not always produce data that satisfies application requirements.

But retrying without clear limits can turn a simple validation problem into an expensive or potentially endless agent workflow.

A reliable implementation separates JSON parsing, schema validation, business rules, and external tool execution.

It retries only recoverable generation failures, provides targeted validation feedback, and stops when the configured budget is exhausted.
At the agent level, additional limits prevent repeated tool calls and uncontrolled workflow execution.

Native structured output features can reduce formatting failures, but they do not replace application validation or authorization.

The goal is not to make an AI agent retry until it succeeds. The goal is to make it succeed within defined limits or fail in a controlled, observable way.

Top comments (0)