I Built a Local Observability Stack for Claude Code: Traces, Tool Calls, Tokens, and Cost

Yasvanth Udayakumar — Sun, 26 Jul 2026 05:22:50 +0000

I was building a Claude Code plugin — a few skills, a couple of MCP servers, some hooks, the usual pile of scripts. It mostly worked. That was the problem: "mostly worked" was the highest resolution I had.

Some prompts came back quickly. Others chewed through minutes and noticeably more money for work that looked the same size. The terminal showed me the same thing either way: text scrolling past, a result, a cursor. It could not tell me how many tools a prompt actually called, whether a skill I had written fired at all, or which of my prompts was quietly the expensive one.

I have spent years insisting that services get instrumented before they ship. It felt strange to be debugging an agent workflow by squinting at scrollback.

Claude Code exports OpenTelemetry data. So I pointed it at a local LGTM stack and built dashboards against what actually came out.

The observability problem

Terminal output renders the conversation. It does not render the execution.

Past a certain plugin size, the questions I care about stop being answerable from the transcript:

How many tool calls did this prompt actually take?
Which skills and which plugin MCP servers were involved?
How many tokens went into this session, split by billing class?
What did that one prompt cost?
How long was the session actually active?
Can I correlate a log line with a span?
Did Claude take a path with unnecessary steps in it?

These are ordinary questions. If this were a checkout service, nobody would consider it exotic to want per-request latency, a dependency map, and cost attribution. The agent case is the same shape: a request fans out into a loop of model calls and tool executions, each with a duration and a price, and you want to see the fan-out.

The one thing that differs: tokens are the bill. In a normal service, cost is an infra line item you review monthly. Here it is a per-request property that varies by an order of magnitude depending on how the agent chose to work.

What I built

The repository is xops-labs/claude-observability. It is small on purpose — a Compose file, one collector config, three dashboards, and a bootstrap script per platform.

Claude Code exports OTLP over gRPC (HTTP is also accepted) to a standalone OpenTelemetry Collector running on the host. The collector limits memory, tags every record with the local environment and route, batches, and forwards all three signals to the collector embedded in Grafana's otel-lgtm image. From there metrics land in Mimir, logs and events in Loki, traces in Tempo, and Grafana reads all three.

Each piece has one job. Claude Code produces the telemetry. The standalone collector is the ingestion and processing boundary. The LGTM image supplies storage plus preconfigured data sources. The dashboards are where I actually spent my time.

Why not an existing tool?

Worth answering before you spend a weekend on this, because the space is not empty.

Grafana Cloud ships an official Claude Code integration with prebuilt stats, usage-breakdown and productivity dashboards; it takes metrics and logs/events, but not traces. There is a community Claude Code dashboard on Grafana's site built on Azure Monitor and Application Insights, and several good write-ups routing Claude Code metrics into Prometheus or VictoriaMetrics to watch usage against plan limits. They all answer the same question: how much is being spent, by whom, over time. If that is your question, use one of those — they are considerably less work than this.

The LLM observability platforms — Langfuse, Helicone, Arize Phoenix, OpenLLMetry — solve a different problem again. They instrument an application you wrote, where you own the call sites and can decide what a span means. Claude Code is not that. It is a closed agent you drive, and you get whatever it chooses to emit.

What I wanted was neither org-level spend nor app instrumentation, but the execution path of one prompt I had just typed: which tools it called, in what order, how long each took, what it cost. That is why this stack is Loki-first with prompt and session drill-downs, and why traces matter enough to turn on a beta flag. It is a debugger for your own workflow, not a usage report.

Architecture

Claude Code
    |  OTLP/gRPC :4317  (or OTLP/HTTP :4318)
    v
OpenTelemetry Collector
    |  memory_limiter -> resource -> batch
    |  OTLP/gRPC on the Docker network
    v
Grafana OTEL-LGTM ---> Mimir  (metrics)
                  |--> Loki   (logs & events)
                  '--> Tempo  (traces)
                          |
                          v
                       Grafana :3300

Every published port binds to 127.0.0.1. The LGTM image's own OTLP ports stay internal to the Docker network, so host 4317 and 4318 belong only to the standalone collector — deliberately, so nothing can accidentally bypass the processing layer.

Why a standalone collector?

I could have pointed Claude Code straight at the LGTM image, which has its own OTLP receivers. I did not, because collapsing ingestion into storage means every future change to how telemetry is handled becomes a change to how telemetry is stored.

What the collector config does today, and nothing more:

OTLP receivers on gRPC :4317 and HTTP :4318.
memory_limiter — 256 MiB limit with a 64 MiB spike allowance, checked every second. This runs on a laptop already hosting Docker, an editor, and the agent itself.
resource — upserts deployment.environment.name=local and telemetry.route=local-otel-collector onto every record, so the origin is visible from inside the data.
batch — 1024 records or 5 seconds, whichever comes first.
OTLP gRPC exporter to lgtm:4317 with sending_queue and retry_on_failure enabled, so a restarting LGTM container does not silently drop a session's telemetry.

Traces, metrics, and logs each get their own pipeline, all three running the same processor chain. There is no sampling, filtering, or redaction processor in here. If I later want to strip an attribute before storage, that is a change in one file that never touches the backends.

The three dashboard perspectives

The dashboards are provisioned from grafana/dashboards/*.json into a Claude Observability folder, with UI edits disabled so the JSON stays authoritative.

One thing surprised me while building them, and it shaped everything: almost all of the useful detail arrives as logs, not metrics. Session IDs, prompt IDs, per-request cost, token splits, tool names, tool inputs, hook executions — these come through as Loki events with structured metadata. The Prometheus counters exist, but they are coarse, and (observed on Claude Code 2.1.218) the VS Code extension and claude -p do not emit them at all. So the dashboards are Loki-first, with the native metrics parked in a clearly-labelled collapsed row.

Claude Code Telemetry (overview)

The entry point. Stats over the selected range — total cost, distinct sessions, input, output, cache read and cache creation tokens — then time series for cost by model, token usage by billing class, API requests by model, request duration at p50 and p95, and errors split into API errors and failed tool executions.

Top of the overview: total cost, distinct sessions, and the four token classes over the selected range.

Then the part I use most: Tool calls by tool, Skill & plugin activity (skill invocations, plugin MCP tool calls, hook executions, plugins loaded), and MCP tool calls by server. Plus a Cost by prompt table for the top 25 prompts in range, a Top MCP tools bar gauge, a Sessions breakdown table, Context size by session, and a raw event log.

One wrinkle worth knowing. The raw tool_result events carry generic tool names — every MCP call is mcp_tool and every skill invocation is Skill. The real names live inside tool_parameters. So the dashboards use LogQL label_format with regexReplaceAll to pull them back out, rendering MCP calls as server: tool and skills as skill: name. Without that, the most interesting panel for plugin work is one enormous bar labelled mcp_tool.

Tool calls by tool. Note claude_ai_Vercel: list_deployments — that name is reconstructed from tool_parameters, not emitted directly.

Skill & plugin activity: plugin hook executions and plugins loaded at session start.

The collapsed Native metrics (Prometheus) row holds claude_code_cost_usage_USD_total, claude_code_session_count_total, and claude_code_token_usage_tokens_total. Active time (claude_code_active_time_seconds_total) is available for ad hoc queries but I have not put it on a panel yet.

Claude Code — Session Drill-down

A session ID textbox variable (regex, .* shows everything) scopes the whole dashboard. You do not usually type it — you click a row in the overview's Sessions breakdown and land here.

It gives session cost, prompt count, API requests and tool calls, peak context, token split, and errors. Then a Prompts in this session table where each row links through to the prompt view, context size over time, token usage by type, a ranked Tool, MCP & skill calls gauge, a Tool spans (Tempo) table, the full tool call log, and the session's prompts in order.

Sessions breakdown. Each session_id links straight through to the session drill-down; note the $6.03 session with 865K peak context next to seven sessions under $1.

Two details worth being precise about. Sessions have no name in telemetry — only a UUID — so the first prompt is the closest thing to a title, which is why the prompt list is prominent. And peak context is derived, not exported: input + cache_read + cache_creation on the largest single API request. It tracks the context meter you see in VS Code, but the per-category breakdown behind that meter is not exported.

Tool spans read from Tempo. Every span is named claude_code.tool; the real name lives in the tool_name attribute, selected here as a column.

Claude Code — Prompt Drill-down

One prompt, end to end. Cost, API requests and tool calls, token split, cache tokens, errors, hook executions. An event timeline by type where the agent loop shows up as bursts of tool_result between api_request bars. The prompt text itself. A ranked tool call gauge. A chronological tool/MCP/skill log with inputs, success flags, and durations. The full event trail.

Then the honest bit. Spans carry session.id but not prompt_id. So the Tempo panel here is range-scoped rather than prompt-scoped — it lists every claude_code.interaction trace in the window, not just this prompt's. The workaround is a small Loki panel above it, Traces for THIS prompt, which pulls the distinct trace_id values off that prompt's api_request events. Expand a line, click through, and you are in the right waterfall.

Inside that waterfall every tool span is named claude_code.tool; the actual tool name sits in the tool_name span attribute, which is why the session dashboard's Tempo table selects it as an explicit column.

Event timeline by type. The bursts are agent-loop iterations: api_request, then tool_result, then another api_request.

The ranked tool gauge, here scoped to the whole window rather than one prompt — Bash at two thirds of everything, and the Vercel MCP call resolved to a real name.

Running it locally

You need Docker with Compose v2, Claude Code installed and authenticated, and PowerShell or a POSIX shell.

Windows PowerShell:

.\start.ps1

macOS, Linux, or WSL:

sh ./start.sh

The bootstrap checks Docker, generates an ignored .env with a random local Grafana password on first run, merges the telemetry environment from claude-settings.example.json into your user-level ~/.claude/settings.json (preserving whatever else is in there), starts both services, waits for the LGTM healthcheck and the collector health endpoint, then verifies all three dashboard UIDs are present in Grafana before declaring success. Re-running is safe. Because the settings go to the user-level file, telemetry applies to every project, not just this one.

Restart Claude Code — CLI or editor extension — after the first bootstrap. Then use it normally. Grafana is at http://localhost:3300, and the export intervals are short (10s metrics, 5s logs and traces), so give it a few seconds.

To check health, and to stop or wipe everything including stored telemetry:

curl -fsS http://localhost:13133/
docker compose ps
docker compose down
docker compose down --volumes

What the telemetry actually shows you

Rather than describe this in the abstract, here is my own last 24 hours — the same window as the screenshots above.

Cost is wildly uneven, and a daily total hides it. Eight sessions, $8.19. One session accounts for $6.03 of that, about three quarters of the day, while five others come in under eleven cents each. The expensive one is not expensive because it ran longer or sent more prompts — it sent two. It is expensive because its peak context reached 865K tokens against roughly 30K for the cheap ones. Cost and peak context sitting in the same row make that legible immediately.

Cache reads dominate, by a margin I did not expect. Over the same window: 12K uncached input tokens against 5 million cache reads, plus 525K of cache creation. That ratio is the system working as intended, history and tool schemas arriving at roughly a tenth of the input rate. It also reframes what "reduce token usage" means — the lever is not the input number, it is whatever invalidates the cache and forces re-creation.

The tool mix is more lopsided than it feels from the transcript. 45 tool calls across 40 API requests: 31 Bash, 9 Read, 2 Write, 1 Edit, 1 ToolSearch, and a single MCP call out to Vercel. Bash at two thirds of everything is the sort of fact that changes where you put a hook, or what you bother to cache.

Durations live in the spans, not the metrics. The Tempo table puts numbers on it: Bash spans from 384ms up to 7.84s, ToolSearch at 5.57ms, the Vercel MCP call at 621ms. When something feels slow, this is where you find out whether it was the model or one shell command.

What this does not do: tell you whether Claude's answer was correct. There is no evaluation, accuracy scoring, hallucination detection, or intent matching anywhere in this repository. It shows the execution path and what it cost. Judging the output is still your job.

Security and privacy — read this part

This configuration deliberately captures sensitive data. It enables OTEL_LOG_USER_PROMPTS, OTEL_LOG_ASSISTANT_RESPONSES, OTEL_LOG_TOOL_DETAILS, OTEL_LOG_TOOL_CONTENT, and OTEL_LOG_RAW_API_BODIES. That means prompts, assistant responses, source file contents, command output, tool inputs, API conversation bodies, and OAuth account identity attributes can end up in the Docker volume claude-observability_lgtm-data.

Local is not the same as harmless. That volume is a plaintext record of your work sitting on your disk. I found this out concretely while picking screenshots for this article: half the panels I captured had real prompt text, an email address, and private project paths in them, and could not be published without redaction.

The precautions the repo takes, and the ones it asks of you:

All published ports bind to 127.0.0.1. Keep them there.
Grafana anonymous auth is disabled; the bootstrap generates a random password rather than shipping a default.
Use synthetic test data. Do not point this at work involving customer data or production secrets.
Delete the volume after a sensitive test run: docker compose down --volumes.
Know which flags you have on. Turning the logging flags off is a one-line edit in ~/.claude/settings.json.

Two limits worth knowing: inline bodies are truncated at Claude's content limit, and extended thinking stays redacted. Prompt logging is also not retroactive — anything sent before you enabled it stays <REDACTED>.

Limitations

Being straight about what this is:

Local development and testing only. Not hardened, not multi-user, no auth story beyond a local Grafana password.
Trace export is a beta capability gated behind CLAUDE_CODE_ENHANCED_TELEMETRY_BETA=1. If your build or account does not support it, metrics and events still export fine — you lose the trace views.
Detailed hook tracing has additional gates and is left disabled.
Everything is subject to export intervals. Data arrives in seconds, not instantly.
Native Prometheus metrics did not appear from VS Code extension or claude -p sessions in my testing (Claude Code 2.1.218). The Loki panels cover all session types, which is why they carry the dashboards.
The dashboards are only as good as the fields Claude Code emits. The MCP and skill name extraction in particular depends on the current shape of tool_parameters.
The dashboards query service_name="claude-code". Other service names need their own queries.
This is not an evaluation platform.

Lessons learned

Output does not reveal execution. A good answer can come from an efficient path or a wasteful one, and looks identical either way. Only the telemetry distinguishes them.

Cost is actionable only when correlated. "You spent $8.19 today" is trivia. "One session took $6.03 of that across two prompts, with peak context at 865K while the cheap sessions sat near 30K" points at something you can change.

More telemetry is more responsibility. The moment I turned on prompt and response logging, I created a new place where sensitive content lives — one holding my source files, my command output, and my account identity. That is a trade-off, not a checkbox.

Wrapping up

This started as a way to stop guessing about my own plugin, and became something I now open reflexively when a session feels off. It is a small stack: a collector, an LGTM image, three dashboards, one bootstrap script.

The repository is at github.com/xops-labs/claude-observability. Clone it, run one script, and your own sessions will be on screen in about a minute.

I would like to know how other people are approaching this. Plenty of setups track Claude Code spend; far fewer try to reconstruct what happened inside a single prompt, and anyone who has tried has almost certainly hit problems I have not — particularly around trace correlation and evaluation signals. If you are running something similar, or you try this and find a panel that is wrong or a query that could be better, open an issue or send a PR. I am more interested in comparing approaches than defending mine.

DEV Community: Yasvanth Udayakumar