What to Log When Your AI Agent Retries: A Production Tracing Checklist

Your agent tells the customer the refund went through. The dashboard shows a green 200 and a normal latency. The next morning, finance finds two refunds for the same order.

Nothing crashed, nothing alerted, and nobody can say what happened, because the only record is a log line that says "tool call succeeded". One of the most-commented DEV posts of the past couple of weeks made the point well: your agent's retry bug isn't an AI problem, it's a distributed-systems problem with an LLM on top. I'd add one thing: you can't fix that kind of bug until you can see it, and most agent setups can't.

This article is about what to record so a retry, a duplicate action or a runaway loop shows up in your traces instead of in a customer complaint.

Why retries hide in agents

Retries happen in at least three places in a typical agent, and each one is invisible by default.

1. The SDK retries for you. The OpenAI Python SDK README, for example, says certain errors are retried 2 times by default with a short exponential backoff, including connection errors, 408, 409, 429 and 5xx responses. That's sensible behavior, but it means one call in your code can be up to three requests on the wire, and your log line only sees the final result.

2. Your own code retries. A wrapper with for attempt in range(3) around a flaky tool, a queue that redelivers a job after a timeout, a framework that re-runs a failed step.

3. The model retries. This is the one specific to agents. If a tool returns an error, or a result the model doesn't like, the model can simply call the tool again, sometimes with slightly different arguments. From the outside it looks like normal agent work.

Stack those up and a single user request can trigger the same side effect several times, with every individual step reporting success. A status code and a latency number can't tell you that. A trace can, if it records the right things.

Log the structure, not just the events

The fix starts with switching from flat log lines to traces: one trace per user request, with a span for every model call and every tool call nested inside it. When the duplicate refund happens, you want to open one trace and see two issue_refund tool spans under the same request, and which model turn produced each one.

You don't have to invent the names. OpenTelemetry has semantic conventions for generative AI (they recently moved to their own repository, and their status is still Development, so expect changes). The parts that matter here:

• A model call is an inference span named {gen_ai.operation.name} {gen_ai.request.model}, for example chat plus the model name.
• A tool call is an execute_tool {gen_ai.tool.name} span. The conventions explicitly encourage developers to instrument tools their own code invokes by hand when automatic instrumentation doesn't cover them, which is most custom tools.
• An agent run is an invoke_agent {gen_ai.agent.name} span.
• gen_ai.conversation.id ties together the messages of one conversation or session.

Using the shared names means your traces will read the same in whatever backend you pick, and you can switch backends later.

What to record on every span

Here's the minimum I'd want on each span to debug a retry problem. Attribute names come from the GenAI and HTTP conventions; the two marked "custom" are my own, since the conventions don't define them.

On model calls:

• gen_ai.request.model and gen_ai.response.model, so you know which model actually answered.
• gen_ai.response.id, the provider's identifier for the completion.
• gen_ai.usage.input_tokens and gen_ai.usage.output_tokens. A retry storm shows up as a token spike long before it shows up on an invoice.
• gen_ai.response.finish_reasons, so you can tell a model that finished from one that was cut off.
• The provider's request ID. Anthropic's error docs, for instance, say every API response includes a unique request-id header, and the same ID appears as request_id in error bodies. Store it, because it's what support will ask for.

On tool calls:

• gen_ai.tool.name and gen_ai.tool.call.id, the identifier the model supplied for that call. Two different call IDs hitting the same tool with the same arguments is the model retrying.
• error.type when the call failed, using the error code the provider or library returned.
• app.retry.attempt (custom): 0 for the first try, 1, 2... for your own retries.
• app.idempotency_key (custom): the key you sent to the downstream API for that action, if it supports one.

On HTTP calls underneath: the HTTP conventions already define http.request.resend_count, set on each resent request to the ordinal number of the attempt, for any reason: redirects, auth challenges, 503s, network errors. If your HTTP client instrumentation sets it, SDK-level retries stop being invisible.

Pair tracing with idempotency

Tracing tells you a duplicate happened. Idempotency stops it from costing you money. Payment APIs already work this way. Stripe's docs describe using an idempotency key when creating or updating an object so that, after a connection error, you can safely repeat the request without creating a second object or performing the update twice.

The agent version: generate the key from what the action means (for example the order ID plus the action name), not from the attempt, so every retry of the same intent sends the same key. Then record that key on the span. When you see two issue_refund spans with the same app.idempotency_key, you know the retry happened and the downstream API deduplicated it. Two spans with different keys for the same order is the bug.

Here's a sketch of a manually instrumented tool call using the OpenTelemetry Python API. It's an example of the shape, not code from a real system, and issue_refund stands in for your own function:

from opentelemetry import trace
from opentelemetry.trace import Status, StatusCode

tracer = trace.get_tracer("support-agent")

def run_refund_tool(call_id: str, order_id: str, amount_cents: int, attempt: int = 0):
    idem_key = f"refund:{order_id}"  # same intent -> same key on every retry
    with tracer.start_as_current_span("execute_tool issue_refund") as span:
        span.set_attribute("gen_ai.operation.name", "execute_tool")
        span.set_attribute("gen_ai.tool.name", "issue_refund")
        span.set_attribute("gen_ai.tool.call.id", call_id)
        span.set_attribute("app.retry.attempt", attempt)        # custom
        span.set_attribute("app.idempotency_key", idem_key)     # custom
        try:
            return issue_refund(order_id, amount_cents, idempotency_key=idem_key)
        except Exception as exc:
            span.set_attribute("error.type", type(exc).__name__)
            span.record_exception(exc)
            span.set_status(Status(StatusCode.ERROR))
            raise

Be careful with prompts and outputs

It's tempting to log every prompt and every answer. The GenAI conventions advise against doing that by default: instructions, user messages and model outputs are considered sensitive and often large, so instrumentations SHOULD NOT capture them unless you opt in. The opt-in is the OTEL_INSTRUMENTATION_GENAI_CAPTURE_CONTENT setting, whose default is NO_CONTENT, and the same gate covers tool call arguments and results.

The conventions suggest recording content on spans mainly where volume is manageable and privacy rules allow it, such as pre-production, and storing content externally with a reference on the span in production, so it can have separate access controls. For retry debugging you usually don't need the full prompt anyway. Call IDs, attempt numbers, idempotency keys and token counts tell you most of the story.

A production logging checklist

Before your agent touches anything that costs money or can't be undone:

1. One trace per user request, with model, tool and agent spans nested inside it.
2. Every tool span has the tool name, the model's tool call ID and an attempt number.
3. Every side-effecting tool sends an idempotency key based on intent, and the key is on the span.
4. Model spans carry token counts, finish reasons and the provider's request ID.
5. HTTP client spans record resend counts, so SDK retries are visible.
6. You know your SDK's default retry policy, and you've decided whether to keep it.
7. Prompt and output capture is a deliberate choice, off by default in production.
8. You have one saved query: "same tool, same intent, more than one call in a trace".

The last item is the one that turns traces into an alert instead of an archive.

Where to go next

Retries are one slice of a bigger problem: an LLM system can return a fast, well-formed 200 that is completely wrong. LLM Observability & Production Monitoring: The Complete Guide covers the rest in 40 pages, 3 parts and 15 chapters: tracing LLM apps and multi-step agents, the metrics that matter (including quality signals), reference-free quality checks and LLM-as-judge on live traffic, cost and latency in the tail, user feedback, alerting without noise, drift detection, debugging incidents you can't reproduce, and turning production failures into permanent test cases. It's framework- and vendor-agnostic, with a worked example and a 30-day path from a black box to an observable system.

This article was written with the assistance of AI and reviewed by the author.