Short answer: capture the error once, attach it to the agent turn that caused it, and record the retry as a separate event. For a logistics agent, the useful unit is not a red error count. It is a small record that connects request outcome, latency, retry count, token usage, and shipment workflow. That lets a team resolve a failing carrier lookup without guessing which retry or downstream call consumed the budget.
The least complex design is a standard trace around the agent loop, a structured error event on the failed span, and metrics aggregated from those events. Keep the error payload free of shipment addresses, customer text, and credentials. Cost attribution needs identifiers and measurements, not a copy of every request.
It is a poor fit for teams that need full session replay or a long-lived archive of raw payloads. The limitation is deliberate: those needs call for a separate, access-controlled evidence store; putting everything into the error stream makes retention, privacy, and query cost harder to control.
What should an API error explain?
An error record should answer four questions: which workflow was running, which dependency failed, how long the attempt took, and what happened next. In a route-planning assistant, that might mean a carrier-rate request returned a timeout while the agent was choosing between two delivery plans. A later retry must retain the same workflow identifier but get its own attempt identifier.
This distinction matters because retries can hide the original failure. If the application increments only a generic api_errors_total counter, a dashboard can show that errors rose while giving no way to separate a slow carrier from an invalid tool argument. A span gives the event a place in the request timeline; a metric gives the team a stable aggregate. OpenTelemetry describes metrics as a signal for representing measurements over time, which is the right role for counts and latency distributions rather than raw diagnostic context.
I keep the attribution fields deliberately boring: workflow_id, agent_turn, tool_name, dependency, attempt, status_code, latency_ms, input_tokens, and output_tokens. The boring fields are the ones an eval harness can assert on.
The retry budget is finite.
Keep it bounded.
A small Python capture path
The example below uses a generic exporter interface. It is intentionally not tied to a hosted error product. The application can send the records to an OpenTelemetry-compatible collector, a self-hosted service, or a test sink while the agent is being evaluated.
from dataclasses import dataclass, asdict
from time import perf_counter
from typing import Any, Callable
import json
@dataclass
class ApiAttempt:
workflow_id: str
agent_turn: int
tool_name: str
dependency: str
attempt: int
status_code: int | None
latency_ms: int
input_tokens: int
output_tokens: int
error_type: str | None = None
def capture_attempt(
call: Callable[[], dict[str, Any]],
*,
workflow_id: str,
agent_turn: int,
tool_name: str,
dependency: str,
attempt: int,
input_tokens: int,
output_tokens: int,
emit: Callable[[dict[str, Any]], None],
) -> dict[str, Any]:
started = perf_counter()
status_code = None
error_type = None
try:
response = call()
status_code = int(response["status_code"])
if status_code >= 400:
raise RuntimeError(f"dependency returned {status_code}")
return response
except Exception as exc:
error_type = type(exc).__name__
raise
finally:
elapsed = round((perf_counter() - started) * 1000)
emit(asdict(ApiAttempt(
workflow_id=workflow_id,
agent_turn=agent_turn,
tool_name=tool_name,
dependency=dependency,
attempt=attempt,
status_code=status_code,
latency_ms=elapsed,
input_tokens=input_tokens,
output_tokens=output_tokens,
error_type=error_type,
)))
def emit_json(record: dict[str, Any]) -> None:
print(json.dumps(record, separators=(",", ":")))
# Each retry calls capture_attempt with a new attempt number.
# The surrounding agent loop decides whether the error is retryable.
There are two useful choices in this code. First, the finally block records latency for both success and failure, so a timeout does not disappear from the latency distribution. Second, the function re-raises the exception. Capture code should not silently turn a failed carrier call into a successful agent answer. Resolution belongs to the policy around the call: retry a transient timeout within a bounded budget, ask for a missing field again, or return a clearly limited answer.
The sample does not emit the response body. That is a cost and privacy decision. A redacted error class and status code usually support triage; raw payloads can contain addresses, account references, or prompts. If the payload is needed temporarily, sample it behind an access-controlled diagnostic path and set an expiration policy.
How do retries become a cost problem?
A retry is a new attempt, but it is not a new business operation. Grouping by workflow_id lets a cost report show the full operation. Grouping by attempt lets an engineer find the expensive branch. Those views answer different questions and should not be collapsed into one counter.
For each agent turn, calculate cost from the measurements available at that point. A simple internal model can be expressed as: input tokens multiplied by the input rate, plus output tokens multiplied by the output rate, plus metered dependency charges. The rates belong in configuration outside the event payload; the event should preserve usage and a rate-card revision so a later report can be reproduced. Prices change, and they should not be baked into a code example as if they were permanent facts.
The first failure mode I guard against is retrying every exception. A 401, a malformed tool argument, and a 504 do not deserve the same policy. Retrying the first two can increase latency and token spend without increasing the probability of a correct answer. The second failure mode is putting a full prompt in every error event. That makes diagnosis expensive and creates a new data-leak surface.
A practical decision rule is small: retry only errors classified as transient, cap attempts, cap elapsed time, and stop when the marginal value of another model turn is lower than the operational cost. That trade-off should be visible in the record, because a successful answer can still be an operational failure if it required four expensive model turns and delayed a dispatch decision. Keep the original exception type and the final disposition. “Resolved after retry” and “returned a partial route after timeout” are different outcomes for evaluation.
Resolving the signal in production
Start with a trace view for one workflow. Find the failed dependency span, inspect its attempt number and latency, then compare the same fields across a day of traffic. If failures cluster by dependency and status code, the integration boundary needs attention. If they cluster by tool name and malformed arguments, the agent contract or validation layer needs attention. If latency is normal but token usage rises, inspect the retry and context-growth policy.
Metrics make that pattern durable. Track request count, error count, retry count, and latency buckets with low-cardinality dimensions such as dependency, operation, and status class. Do not use shipment ID, customer email, or an unbounded exception message as a metric label. Those values make storage and querying less predictable while adding little value to an aggregate.
Tests should exercise the same record shape as production. A fake dependency can return a timeout on attempt one and a valid response on attempt two. The assertion should verify that there are two attempt records, one workflow ID, one retry increment, and a final disposition. An eval case can then attach a cost ceiling and a latency ceiling to the expected agent behavior. This catches regressions that a green end-to-end answer alone misses.
Before shipping, I check the workflow ID survives every tool call, the attempt number starts at one, failures retain latency, and the exporter is non-blocking or bounded. I also check that a collector outage cannot prevent the agent from returning its business response. Observability is part of the control loop, but it is not allowed to become the dependency that takes the control loop down.
That is enough instrumentation to resolve most API errors without turning the article into a product setup guide: a trace for sequence, metrics for trends, structured events for diagnosis, and explicit retry policy for cost attribution.
Further reading
- OpenTelemetry, “Metrics signal concepts”: https://opentelemetry.io/docs/concepts/signals/metrics/
Top comments (0)