Hassann

Posted on Aug 26 Originally published at apidog.com

AI Agent Tool Call Tracing: What to Log on Every Request

Observability for AI Agents: Trace Every Tool Call

A user reports that the agent “did something weird” yesterday afternoon. You open the logs and find this:

INFO  agent run started
INFO  calling tool: updateOrder
INFO  tool returned 200
INFO  agent run completed

The agent called updateOrder, but you do not know which arguments it used, which order it modified, why it chose that tool, or what the API returned. The run succeeded by every measure you recorded, yet you cannot reconstruct a single decision.

Agent failures often make sense only in hindsight. That makes the log part of the product.

This guide explains what to record for every tool call, how to correlate a model decision with the HTTP request it produced, what to redact, and how to turn traces into tests. API observability covers the service layer; this article focuses on the agent layer above it.

Apidog is useful once you have a trace: replaying the same request against the same endpoint is often the fastest way to understand a bad call.

Three layers, one trace

An agent produces events at three levels:

Reasoning layer — What the model saw, which tools were available, which tool it selected, and which arguments it generated.
Tool layer — Argument validation, policy enforcement, HTTP mapping, and result handling.
HTTP layer — Method, URL, headers, body, status, and latency.

Debugging usually crosses these layers:

“The agent sent the wrong customer ID” is a reasoning problem visible at the HTTP layer.
“The API returned a 200 with an empty body” is an HTTP problem that may appear as strange reasoning later.

Correlating events by timestamp fails as soon as runs overlap. Use:

One trace_id per agent run
One span_id per tool call
The IDs on every record at every layer

OpenTelemetry traces already model this structure, and GenAI semantic conventions provide portable attribute names.

What to record on every tool call

A useful tool-call record looks like this:

{
  "trace_id": "run_01J8ZK3M2Q",
  "span_id": "call_004",
  "parent_span_id": "call_003",
  "timestamp": "2026-08-26T14:03:11.482Z",
  "agent": "billing",
  "step": 4,

  "tool_name": "refundOrder",
  "tool_args": { "orderId": "ord_92", "amount": 1200, "reason": "duplicate" },
  "tools_available": ["getOrder", "listOrders", "refundOrder", "voidInvoice"],

  "http": {
    "method": "POST",
    "url": "/v1/orders/ord_92/refund",
    "request_body_hash": "sha256:1f4c...",
    "status": 200,
    "duration_ms": 412,
    "retry_count": 1,
    "idempotency_key": "9f2b7c14-6d3a-4b18"
  },

  "outcome": "success",
  "tokens": { "prompt": 8420, "completion": 96 },
  "policy": { "approval_required": true, "approved_by": "user_31", "dry_run": false }
}

These fields provide disproportionate value:

tool_args — Log the arguments before your executor normalizes them. This is where incorrect IDs and values become visible.
tools_available — Explains tool selection. If the model chose an unexpected tool, you can immediately see its alternatives.
retry_count — Distinguishes a slow API from an API that failed twice before succeeding.
outcome — Use an explicit enum such as success, failed, timed_out, blocked_by_policy, or rejected_by_human. A blocked call is a working guardrail, not necessarily an error.
policy — Provides the approval audit trail. It should complement the enforcement described in AI agent guardrails.

Log the decision, not only the action

The hardest agent bugs are selection errors. Record enough information to reconstruct the decision.

Version your tool definitions

Store the tool definitions used for the run, or at least a hash of them. A small description change can alter selection behavior. Comparing hashes quickly shows whether the tool set changed between a successful and failed run.

See tool schema design for why tool descriptions influence behavior.

Record model configuration

Store the:

Model ID
Temperature
Prompt version
Tool-set hash

Model behavior changes across versions. Without these fields, a model update can look like an application regression.

Store prompt size and identity

Full prompts can be expensive and sensitive. A token count plus a hash usually provides most of the diagnostic value. If a prompt suddenly doubles in size, something was likely appended unexpectedly.

Preserve raw tool results

If your executor trims a response before returning it to the model, store the full payload in the trace first. Otherwise, you cannot tell whether the API omitted data or your executor removed it.

This is especially important when keeping tool responses out of the context window.

Redact before you store

Agent traces contain both requests and the context around them. Prompts also tend to accumulate personal data.

Never store credentials

Strip:

Authorization headers
API keys
Cookies
Signed URLs

Log a credential identifier, such as a key ID, rather than the secret itself. Least-privilege API keys for agents explains why identifying the key is still useful.

Redact at the boundary

Redact in logging middleware before the record leaves the process. Query-time filtering is too late: the secret may already be on disk, in a replica, or in a backup.

Hash bodies you cannot store

A request-body hash lets you prove that two calls were identical without retaining the payload. That is enough for many duplicate-request investigations.

Tier retention by sensitivity

For example:

Full traces for one week
Redacted summaries for one year

Most debugging occurs within days, while audit questions may arrive months later.

Turn traces into tests

Good tracing produces more than faster debugging. It creates realistic test cases.

Replay failed runs

Every failed run is a scenario. Extract the tool calls from the trace, replay them against the API, and keep the reproduction as a regression test.

In Apidog, rebuild the failing request as a saved case, assert the corrected behavior, and run it in CI. An incident then becomes permanent coverage.

Build mocks from real traffic

Use your traces to identify:

The endpoints agents call most
The failure statuses they actually encounter
The responses that influence later decisions

Build mocks around observed traffic instead of assumptions. See running agents against mocks instead of production.

Track behavioral drift

Review these metrics weekly:

Tool selection distribution
Retry rate per endpoint
Calls per completed task
Percentage of runs blocked by policy

A change in any of them can signal an incident before users report one. API contract testing can catch upstream changes that often cause this drift.

Three investigations your trace must support

“The agent charged the wrong customer”

You need:

The arguments generated by the model
The resolved URL
The preceding tool result
The step where the ID was selected

Often, an earlier tool returns multiple matches and the model picks the first. A complete trace exposes the ambiguity and the choice. Without tool_args, all you have is a successful 200 and an unhappy customer.

“It stopped working on Tuesday”

Compare a successful run with a failed run field by field:

Model ID
Tool-set hash
Prompt version
Average response size

One of these usually identifies the change. This is why configuration belongs on the run record, not only in deployment metadata.

“Did anyone approve this?”

The policy record should answer this directly:

{
  "approval_required": true,
  "approved_by": "user_31",
  "approved_at": "2026-08-26T14:03:10.917Z",
  "dry_run": false
}

Record the approval at the moment of the decision. Reconstructing it later is unreliable.

None of these questions can be answered by “the tool returned 200.” They require fields that are cheap to write and impossible to recover afterward.

Sampling: what never to sample

Full-fidelity tracing can be expensive at scale, but agent traffic is not uniform.

Always retain:

Failed runs
Runs that hit a policy block
Runs containing a write operation

Sample successful, read-only runs because they make up most traffic and are less interesting individually. Keep enough to establish reliable baselines.

The SRE monitoring chapter explains why sampling should preserve signal rather than simply reduce volume.

Even when dropping payloads, keep a skeleton trace containing:

Tool names
Outcomes
Durations
Trace and span IDs

Bodies and prompts are usually the expensive parts, so drop those first.

With tail sampling, decide what to retain only after the outcome is known. A run that looks healthy at step three may fail at step nine, so buffer enough data to retain the complete trace.

Where the trace should live

If your agent is a service calling your APIs, centralized trace storage is usually the right choice.

For coding agents running on developer machines, storing the trace in a terminal session is a poor retrieval model.

Sharkly takes a different approach: it attaches the execution trace to the Task assigned to the agent. Run history, execution logs, results, goals, status, and human review comments live together.

The practical benefit is retrieval. “Why did the agent do that?” becomes a question answered by opening the task instead of finding the right machine, session, and scrollback.

This does not replace runtime tracing or the runtime itself. Claude Code and Codex still perform the work; it changes where the record ends up when the agent is not a service you deployed.

Watch four numbers

Traces are useful only if someone reviews them. These metrics belong on a dashboard:

Calls per completed task

This is the clearest efficiency measure. If it rises, the agent may be exploring more because a tool description worsened or an endpoint began failing.

Retry rate by endpoint

This ranks unreliable dependencies and highlights degradation. Agent error recovery covers how to respond to the worst offenders.

Blocked-by-policy rate

This should remain low and stable. A spike may mean the agent is attempting forbidden actions, or that a policy has become too restrictive.

Time to first tool call

A slow start often indicates a bloated prompt. Prompt size is one of the easiest forms of complexity to grow unnoticed.

Implementation checklist

Use one trace ID per run and one span ID per tool call.
Stamp both IDs across the reasoning, tool, and HTTP layers.
Log model arguments before normalization.
Record the available tool list on every call.
Store outcomes as explicit enums, including policy blocks.
Track retries separately from call count.
Store model ID, temperature, prompt version, and tool-set hash.
Preserve raw tool results before trimming.
Strip credentials in middleware.
Hash bodies that cannot be stored.
Use retention tiers based on sensitivity.
Convert failed traces into replayable test cases.
Include idempotency keys for write operations. See idempotency keys for AI agents.

The goal is simple: when someone asks why the agent did something, answer from the record instead of guessing. Download Apidog to replay calls from a trace and preserve the reproductions as tests.

Frequently asked questions

Should I use OpenTelemetry or a purpose-built agent observability tool?

Use OpenTelemetry for transport and the trace model. It already handles correlation, and most infrastructure supports it. Agent-specific tools can add useful views, but the underlying data should remain portable.

How much does full tracing cost to store?

Less than expected if you tier it. Keep full payloads for a few days and structured records without bodies for longer. Prompt dumps are usually the most expensive data, so hash and size them instead of storing them by default.

Do I need to log the model’s reasoning text?

Usually not. The selected tool, generated arguments, and available alternatives explain most decisions. If a provider exposes reasoning content, store it only for failed runs and treat it as sensitive.

How do I trace across multiple agents?

Use one trace ID for the whole task and give each agent its own span. Record handoffs as events. See multi-agent handoff for the fields that belong in a handoff record.

What if the agent runs on a customer’s machine?

Log locally, redact aggressively, and send only aggregate metrics unless the user opts in. Tool names, outcomes, and durations are usually enough for fleet-level monitoring without sending payloads off the device.

Is a request-body hash actually useful?

Yes. It proves that two calls were identical, which resolves many duplicate-write investigations without retaining the payload. Pair it with idempotency keys, which should prevent duplicate operations in the first place.

DEV Community