Originally published at kunalganglani.com — read it there for inline code, hero image, and live links.

Execution Trace Tree for AI Agents: Build One in 60 Minutes

Last week I watched someone try to debug an agent by tailing logs and “just increasing verbosity.” It produced 40,000 lines of JSON and exactly zero insight.

You’re going to end this tutorial with an execution trace tree for AI agents that you can:

• capture on every run (LLM calls, tool calls, retrieval, memory, guardrails)
• replay from checkpoints without guessing
• diff run A vs run B and point at the exact node where behavior diverged
• export into your existing tracing stack via OpenTelemetry

Give yourself ~60 minutes if you already have an agent loop running.

2026 is when this stopped being optional. People are finally writing about tracing agent workflows with systems like AWS X‑Ray, and the “why does it fail only in prod?” stories all rhyme. Raw logs don’t show causality. A structured trace tree does.

I’ll be blunt: if you’re building AI agents for anything real, log-only debugging is malpractice. You need a deterministic model of what happened.

What is an execution trace tree for AI agents?

An execution trace tree for AI agents is a structured parent/child history of a single agent run. Every step that can change behavior (LLM call, tool call, retrieval, memory read/write, guardrail decision, human approval) becomes a node with enough metadata to reproduce the run and compare it to another run.

It’s “distributed tracing,” but pointed inward at your agent loop instead of outward at your microservices.

In a web request, fan-out is usually HTTP calls. In an agent run, fan-out is weirder: a planning step branches into tool calls, retrieval, retries, “let me ask again with a different prompt,” and sometimes a human gate.

Two rules make trace trees actually useful (instead of “yet another telemetry thing”):

1. Every node is a span. Timing, parent, attributes. Same mental model as OTel.
2. Every node is replayable. Persist the minimum inputs/outputs (or hashes + artifact references) so you can restart from checkpoints.

If you already read my take on vendor-neutral observability, this is the debugging model that sits underneath the dashboards: How to Build Vendor-Neutral LLM Observability Monitoring [2026].

Why logs aren’t enough for agent debugging (and when they actively mislead you)

When agents get flaky in production, teams almost always do the same thing first. Crank log volume up by 10x. It feels responsible. It also turns your on-call into archaeology.

Here’s what log-only systems consistently fail at in agentic systems.

1) Causality beats chronology

Logs give you a timeline. Agents fail because of dependencies.

A classic failure mode: the agent generates a tool call with a slightly different argument shape. The tool returns a subtly different payload. That changes retrieval. That changes the final answer.

In logs, this is 600 lines sprinkled across async workers and retries.

In a trace tree, it’s one path you can walk:

plan → tool.search → retrieval → answer

That’s the difference between “I think it was the search tool?” and “this exact search result changed, and everything downstream followed.”

2) Async fan-out kills “grepability”

Real agent runs do parallel tool calls, retries, background evaluators, and queues. Once you have concurrency, grepping logs becomes a coping mechanism.

If you’ve ever fought broken tracing context in a web stack, you know the smell: traces fragment, or worse, leak across requests.

Ahmed Mahmoud documented this kind of context leakage in Next.js instrumentation. Agents are even more sensitive because the “request” isn’t a single handler. It’s a long-running orchestration with lots of side quests.

3) Diffing runs is basically impossible

Most agent regressions aren’t hard failures. They’re the annoying ones:

• “It answered differently today.”
• “It used to call the tool. Now it doesn’t.”
• “It started citing the wrong doc.”

To debug that, you need a comparison across runs:

• model changed (gpt-4.1 → gpt-4.1-mini)
• temperature changed (0.2 → 0.7)
• tool output changed (external API returned different data)
• retrieval docs changed (index update)

A trace tree makes those changes obvious and diffable. Logs don’t.

If you care about production AI, this is the line between demo debugging and actual engineering.

Spans, parent/child relationships, and how they become an execution tree

Distributed tracing already solved the shape problem. A trace is a collection of spans. Spans have IDs and parent IDs. That gives you a tree.

OpenTelemetry (OTel) standardized the model and the plumbing. You don’t need to adopt every piece of OTel on day one. You do need to stop inventing your own half-trace format that can’t join the rest of your telemetry.

If you’re already instrumenting services, your agent trace should connect to the same trace graph as:

• the API request that triggered the agent
• the database calls the agent made
• the queue workers it fanned out to

That only happens if you treat the agent run as a first-class trace, not “some logs in a side table.”

A minimal agent run trace is usually 20–200 spans per run. That’s not a vibe. That’s a planning number. If your agent does 8 tool calls with 2 retries each and a planner loop of 6 steps, you’re already there.

If you want the broader “agent observability stack” context, I wrote about wiring this into exporters here: OpenTelemetry Instrumentation for AI Agents [2026]: Ship It.

A minimal trace-tree schema (node types + required fields)

This is the part most posts dodge. They say “add tracing” and call it a day. Then you implement it, and six weeks later you realize you didn’t record the one field that would have explained the incident.

My bias: a good schema is:

• small enough you’ll ship it
• strict enough that diffing and replay aren’t a fantasy
• compatible with OpenTelemetry span attributes

Node (span) types you should support

You don’t need 30 event types. You need the ones that change control flow or create nondeterminism.

1. root.run — one per agent run
2. router/planner — “what should we do next?” decisions
3. llm.call — model invocation (system prompt + messages)
4. tool.call — anything with side effects (HTTP, DB, shell, email)
5. retrieval — anything RAG-like (query, topK, doc IDs)
6. memory.read / memory.write — long-term memory and state
7. guardrail — allow/deny/redact decisions
8. human.approval — HITL gates
9. checkpoint — replay boundary

This maps cleanly to how most agent framework runtimes behave.

If your agent does RAG, make retrieval a first-class span. Otherwise you’ll be stuck arguing about “why did it cite that doc?” with nothing but vibes.

Required fields per span

Here’s the compact table I use when I’m designing instrumentation.

Span type	Required fields (minimum)	Why it matters
root.run	run_id, agent_version, input_hash, user_id_hash, start_ts, end_ts	identity, grouping, compliance
router/planner	policy_version, decision, candidates, reason_code	explains control flow
llm.call	provider, model, temperature, max_tokens, prompt_hash, messages_hash, tool_schema_hash, response_hash, input_tokens, output_tokens	replay + cost + drift
tool.call	tool_name, tool_version, args_hash, result_hash, side_effect=true/false, status_code, latency_ms	determinism boundaries
retrieval	index_version, query_hash, top_k, doc_ids, reranker_version	explains context changes
memory.*	store, key_hash, value_hash, op=read/write	state drift
guardrail	rule_id, decision, redaction_count	safety + debugging
human.approval	approver_role, decision, wait_ms	latency + governance
checkpoint	checkpoint_id, state_hash, artifact_refs	restart points

Two concrete limits I’d enforce:

• cap any single span attribute blob to 16 KB (anything larger belongs in an artifact store)
• cap the total persisted trace payload per run to 256 KB before sampling kicks in

If you ignore this, you’ll accidentally turn “observability” into your newest cost center. If you’re thinking about LLM cost, trace payload size is part of that equation.

For more on metrics you can attach to these spans (latency, cost, quality), see: How to Pick LLM Application Observability Metrics [2026].

Checkpoints, deterministic replay, and trace diffing (the workflow that actually saves you)

Most teams stop at “we can see the spans.” That’s table stakes. The payoff is replay + diff.

Step 1: Define what “deterministic” means for your agent

Your agent is deterministic if, given the same:

• prompt/messages
• tool schema
• retrieval corpus + index version
• tool outputs (or captured tool responses)
• model + sampling params

…it produces the same downstream actions.

That’s why I treat tool.call as the determinism boundary. External APIs change. Time changes. Randomness changes. Your “perfect prompt” is not the thing that makes the run reproducible.

Step 2: Add checkpoints at control-flow boundaries

Put a checkpoint span after events that are expensive to recompute or likely to fork behavior.

My default list (5 checkpoints):

1. after initial user input normalization
2. after planning/router decision
3. after retrieval context is assembled
4. before any irreversible tool side effect (side_effect=true)
5. after final answer is generated

That’s enough to replay most bugs without re-running the entire world.

If you’re already thinking in control-flow terms, this pairs nicely with: AI Agent Control Flow Patterns [2026]: Retries, HITL, Checkpoints.

Step 3: Capture “tool snapshots” for replay

To replay safely, you have two options:

• record-and-replay: store the tool output (or a reference to it) and return it during replay
• sandbox replay: re-run tools in a sandbox with fixed fixtures

In practice, you’ll do both:

• record-and-replay for third-party APIs
• sandbox replay for your own services

This is where workflow engines earned their reputation. Temporal’s whole pitch is replaying workflow code against an event history. Even if you’re not using Temporal, the determinism concept transfers cleanly.

If you want the full workflow-engine framing: Temporal Workflow Engine: The Reliability Layer Your Distributed System Is Missing [2026 Guide].

Step 4: Diff two runs node-by-node

Diffing is the killer feature. The only question that matters in a regression is: what changed?

A practical diff algorithm:

1. align root spans by agent_version and input_hash
2. walk the tree in execution order
3. compare each node’s *_hash attributes
4. stop at the first divergence and expand the subtree

What you’ll usually find:

• retrieval.doc_ids differs because the index version changed
• tool.result_hash differs because an API returned new data
• llm.call.model differs because routing changed
• llm.call.temperature differs because someone “tuned creativity”

That’s the moment debugging becomes boring. Boring is good.

This plugs straight into evals too. I built this site’s publishing pipeline as a multi-step, idempotent system with deterministic gates. The lesson transferred cleanly: deterministic structure beats “more model.” Based on the pipeline I run for this site (261+ posts published and weekly automated feedback loops), deterministic gates catch failures earlier than just upgrading the review model.

You can tie trace diffing into your eval workflow here: Agent Evaluation Harness [2026]: Replay, Rubrics, CI Gates and AI Engineering Evals: Regression Gates for Prompts, Tools, RAG [2026].

OpenTelemetry context propagation, redaction, sampling, and cost control

This is where most “agent tracing” tutorials die in production. The demo works. The first incident hits. Context fragments, or you accidentally store secrets, or your tracing bill starts looking like a second LLM bill.

Context propagation: don’t let your trace fragment (or leak)

If you have async tool calls, queues, or background workers, you need to propagate context.

Hard requirements:

• a stable trace_id for the whole run
• a new span_id per node
• propagation across process boundaries (HTTP, queues)

When context propagation breaks, you get:

• orphan spans
• partial traces
• cross-run contamination (the scariest one)

That’s why Ahmed Mahmoud’s Next.js story matters here. The exact same bug class shows up in agent orchestrators when spans are stored in globals or threadlocals incorrectly.

If you’re building agent infrastructure, you want an orchestration layer that makes context explicit, not implicit. See: AI Agent Observability Logging Schema [2026]: OTel + Redaction.

Redaction and security: store less than you think

Agent traces contain the worst kind of data:

• user input (often PII)
• prompts (often secrets and system rules)
• tool outputs (often internal data)

A sane baseline:

• store raw prompts only in dev, never in prod
• store hashes in prod by default (prompt_hash, messages_hash)
• store structured “safe excerpts” only after redaction
• keep artifact blobs in a separate store with retention controls

This is also where prompt injection becomes an observability problem. If your trace store is searchable and you persist raw tool outputs, you created a new exfiltration surface.

If you’re building RAG-heavy systems, the redaction patterns carry over directly from: How to Implement Field-Level Redaction for RAG Pipelines [2026].

And if your team needs the broader posture: AI Security Leader Playbook [2026]: 10 Controls That Ship.

Sampling and cost control: trace trees can get expensive fast

My default:

• keep 100% of traces for failures
• keep 10% of traces for successes
• keep 1% of traces for high-volume endpoints

Attach token/cost metadata at the llm.call spans so you can see which node blew the budget.

If you want to go deeper on cost math, start here: Agent Per-Task Cost Calculation [2026]: Retries, Tools, Caching.

Data anchor (site-owned): the GSC keyword winnability dataset I run for kunalganglani.com shows this topic neighborhood has ~1,083 related impressions already and a demand estimate around ~11,000 searches/month across related agent observability queries. That’s the clearest signal I have that “agent debugging models” are actively being searched for.

Using trace trees for evaluation (regressions, scorecards, and alerts)

Once every run is a tree, evaluation stops being hand-wavy.

Three concrete ways I’d wire it:

1. Label traces: pass/fail, task_type, customer_tier, release_sha
2. Score nodes: attach a rubric score to the exact llm.call or retrieval span that caused failure
3. Alert on drift: when retrieval.doc_ids churn spikes after an index deploy, page someone

A good eval program doesn’t need a PhD. It needs a weekly habit and a stable artifact format. I wrote a lightweight approach here: Agent Evaluation Roadmap for Small Teams [2026]: The 30-Min/Week Plan.

If you’re running any kind of AI in production, you eventually run into the quiet truth: evaluation is observability with opinions.

Where I think this goes next

By the end of 2026, “agent frameworks” that don’t natively emit an execution trace tree will feel like web frameworks that don’t support request IDs. You can technically ship without it. Serious teams won’t.

If you’re building agents now, here’s my challenge: implement the minimal schema above and force yourself to debug the next incident by diffing two trace trees, not by grepping logs. Once you feel how fast that is, you’ll start getting annoyed at any system that can’t do it.

---

Originally published on kunalganglani.com