DEV Community

Cover image for Claude Code agent observability with Docker Compose, Grafana and Microsoft Fabric: Build an agent cost and token dashboard
Jubin Soni
Jubin Soni Subscriber

Posted on

Claude Code agent observability with Docker Compose, Grafana and Microsoft Fabric: Build an agent cost and token dashboard

#claude #docker #grafana #microsoft

AI coding agents consume tokens in ways that are difficult to see. A single request in Claude Code can start a subagent, and that subagent can call several tools, each of which sends more context back to the model. The developer reads the final answer. The invoice shows a monthly total. Neither one explains which agent or which model produced the cost.

Claude Code can export this information as OpenTelemetry metrics and log events. Metrics answer the live question, which is what the agent is spending right now on this machine. Log events answer the historical question, which is how the cost and the tool behavior of many developers changed over the last two weeks. The two questions suit two different tools.

This article builds a viewer for both. A Docker Compose stack on the developer machine runs an OpenTelemetry Collector, Prometheus and Grafana, so the metrics appear in a dashboard within seconds. The same collector forwards the log events to a Microsoft Fabric Eventstream. An Eventhouse stores them, and a Real-Time Dashboard reads them with KQL for the team view.

Pipeline

A note on scope. The Compose file, the collector configuration and the Grafana provisioning files were checked for syntax only. The Fabric steps follow the product documentation. Run each stage against a real Claude Code session and compare the result with the descriptions here. The dashboard images in this article use illustrative data.

What Claude Code emits

Claude Code exports telemetry through the standard OpenTelemetry environment variables. The documentation lists eight metrics. The following table shows the ones that this article uses.

Metric What it counts Attributes used here
claude_code.cost.usage Cost of the session in USD model, query_source, agent.name
claude_code.token.usage Tokens used type (input, output, cacheRead, cacheCreation), model
claude_code.session.count Sessions started start_type
claude_code.lines_of_code.count Lines of code modified type (added or removed)
claude_code.code_edit_tool.decision Accept or reject decisions on edits decision, tool_name

The log events carry more detail, because Claude Code writes one event for each occurrence instead of a running total. Two events matter for cost and reliability. The claude_code.api_request event describes one request to the model, with the attributes model, cost_usd, duration_ms, input_tokens, output_tokens, cache_read_tokens, cache_creation_tokens and query_source. The claude_code.tool_result event describes one tool call, with tool_name, success and duration_ms.

The attribute query_source is the one that turns a cost report into an agent report. It identifies the part of the system that issued a request. The main conversation and each subagent have different values, and background work such as context compaction has its own value, so the cost of delegation becomes visible.

One agent run, two kinds of telemetry

Prompt text is a separate matter. Claude Code redacts prompts, assistant responses and tool parameters by default, and each has its own opt-in variable. This article leaves all of them disabled. The user.id attribute is a random anonymous identifier and is not derived from the account.

Enable telemetry

Telemetry is off until the variable CLAUDE_CODE_ENABLE_TELEMETRY is set to 1. The following file sets the exporters, the protocol and the endpoint that the collector will expose. A shell that sources it starts every later Claude Code session with telemetry enabled.

# Source this file in the shell that starts Claude Code: source claude-telemetry.env
export CLAUDE_CODE_ENABLE_TELEMETRY=1
export OTEL_METRICS_EXPORTER=otlp
export OTEL_LOGS_EXPORTER=otlp
export OTEL_EXPORTER_OTLP_PROTOCOL=grpc
export OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4317
export OTEL_METRIC_EXPORT_INTERVAL=10000
export OTEL_LOGS_EXPORT_INTERVAL=5000
# Prompts, responses and tool parameters stay redacted unless you opt in. Leave these unset.
Enter fullscreen mode Exit fullscreen mode

Claude Code has no default protocol, so the file sets OTEL_EXPORTER_OTLP_PROTOCOL explicitly. The short metric interval of ten seconds is useful during setup. A longer interval reduces the number of samples once the stack works as intended.

Run the collector, Prometheus and Grafana

The Compose file starts three containers. The collector receives OTLP from Claude Code. Prometheus scrapes the metrics that the collector exposes. Grafana reads Prometheus and loads a dashboard from provisioning files.

services:
  otel-collector:
    image: otel/opentelemetry-collector-contrib:latest
    command: ["--config=/etc/otelcol/config.yaml"]
    environment:
      FABRIC_KAFKA_BOOTSTRAP: ${FABRIC_KAFKA_BOOTSTRAP:-localhost:9093}
      FABRIC_KAFKA_TOPIC: ${FABRIC_KAFKA_TOPIC:-claude-events}
      FABRIC_KAFKA_CONNECTION_STRING: ${FABRIC_KAFKA_CONNECTION_STRING:-unset}
    volumes:
      - ./otel-collector-config.yaml:/etc/otelcol/config.yaml:ro
    ports:
      - 4317:4317
      - 4318:4318

  prometheus:
    image: prom/prometheus:latest
    command: ["--config.file=/etc/prometheus/prometheus.yml"]
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml:ro
      - prometheus-data:/prometheus
    ports:
      - 9090:9090
    depends_on:
      - otel-collector

  grafana:
    image: grafana/grafana-oss:latest
    environment:
      GF_AUTH_ANONYMOUS_ENABLED: "true"
      GF_AUTH_ANONYMOUS_ORG_ROLE: Admin
    volumes:
      - ./grafana/provisioning:/etc/grafana/provisioning:ro
      - ./grafana/dashboards:/var/lib/grafana/dashboards:ro
      - grafana-data:/var/lib/grafana
    ports:
      - 3000:3000
    depends_on:
      - prometheus

volumes:
  prometheus-data:
  grafana-data:
Enter fullscreen mode Exit fullscreen mode

The collector configuration defines one pipeline for each signal.

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318

processors:
  # Prometheus needs cumulative counters. This processor converts delta sums.
  deltatocumulative: {}
  batch:
    timeout: 5s

exporters:
  # Local: Prometheus scrapes this endpoint, Grafana reads Prometheus
  prometheus:
    endpoint: 0.0.0.0:8889

  # Team-wide: Fabric Eventstream through its Kafka-compatible custom endpoint
  kafka/fabric:
    brokers:
      - ${env:FABRIC_KAFKA_BOOTSTRAP}
    logs:
      topic: ${env:FABRIC_KAFKA_TOPIC}
      encoding: otlp_json
    tls:
      insecure: false
    auth:
      sasl:
        mechanism: PLAIN
        username: $$ConnectionString
        password: ${env:FABRIC_KAFKA_CONNECTION_STRING}

service:
  pipelines:
    metrics:
      receivers: [otlp]
      processors: [deltatocumulative, batch]
      exporters: [prometheus]
    logs:
      receivers: [otlp]
      processors: [batch]
      exporters: [kafka/fabric]
Enter fullscreen mode Exit fullscreen mode

Several details in this file deserve attention. The deltatocumulative processor converts delta metrics to cumulative ones by keeping a running total in memory, and its documentation lists the stability level as alpha. The processor is present because the Prometheus exporter serves cumulative values, and the documentation that I read does not state the temporality that Claude Code uses by default. The SASL username contains a literal dollar sign, so the configuration escapes it as $$ConnectionString. The encoding value otlp_json makes each Kafka message a readable JSON document that the Eventhouse can ingest later.

Prometheus needs only one scrape target.

global:
  scrape_interval: 15s

scrape_configs:
  - job_name: claude-code
    static_configs:
      - targets: ["otel-collector:8889"]
Enter fullscreen mode Exit fullscreen mode

Start the stack from a folder named claude-telemetry, which also becomes the Compose project name.

docker compose up -d
source claude-telemetry.env
claude
Enter fullscreen mode Exit fullscreen mode

Without Fabric values in a .env file, the Kafka exporter logs connection retries. The metrics pipeline continues to work, because each pipeline has its own queue. Comment out the logs pipeline if the retry messages distract during the first test.

Docker Desktop showing the otel-collector, prometheus and grafana containers

Chart agent cost and tokens in Grafana

Grafana reads its configuration from files, so the dashboard exists as soon as the container starts. Two small files connect it to Prometheus and tell it where to find dashboard JSON.

apiVersion: 1

datasources:
  - name: Prometheus
    uid: prometheus
    type: prometheus
    access: proxy
    url: http://prometheus:9090
    isDefault: true
Enter fullscreen mode Exit fullscreen mode 
apiVersion: 1

providers:
  - name: claude-code
    folder: Claude Code
    type: file
    options:
      path: /var/lib/grafana/dashboards
Enter fullscreen mode Exit fullscreen mode

The dashboard file grafana/dashboards/claude-code-agents.json holds eight panels. Each one is a short PromQL expression. The cost panel, for example, sums the increase of the cost counter per model.

sum by (model) (increase(claude_code_cost_usage_USD_total[$__rate_interval]))
Enter fullscreen mode Exit fullscreen mode

The panel for delegated work uses the same counter and groups by the attribute that identifies the source of each request.

sum by (query_source) (increase(claude_code_cost_usage_USD_total[$__rate_interval]))
Enter fullscreen mode Exit fullscreen mode

Metric names change on the way into Prometheus. The exporter normalizes names by default and can append a unit suffix and a counter suffix. Open http://localhost:9090 after the first session and search for claude_code. If the names differ from the ones in the dashboard file, edit the file and reload Grafana. The cost metric also supports attribution attributes such as agent.name and skill.name, and a query that groups by agent_name shows the cost of each named subagent when the attribute is present.

Open http://localhost:3000 to see the dashboard. The Grafana image in the Compose file is the open source edition, so no license is required.

Grafana dashboard showing Claude Code cost, tokens, sessions, cost by model and cost by query source

Send the log events to Fabric

Grafana with Prometheus answers questions about the current session and the last few days, because Prometheus keeps data on the local volume. A team lead needs a longer record from many machines, and a query language that can join and rank. The Fabric half of the stack provides both.

The Fabric side has three items. An Eventstream receives the events, an Eventhouse stores them, and a Real-Time Dashboard displays them. The Eventstream accepts Kafka clients through a custom endpoint source, which is the reason the collector can use its standard Kafka exporter.

  1. Create an Eventhouse named claude-telemetry. Fabric creates a default KQL database with the same name.
  2. Create an Eventstream and add a Custom endpoint source.
  3. Open the source, select the Kafka tab, and choose SAS Key Authentication. The page shows the bootstrap server, the connection string for the primary and secondary keys, and the topic name.
  4. Copy these values into a .env file next to the Compose file. 
# Copy to .env. Values come from the Kafka tab of the Eventstream custom endpoint.
FABRIC_KAFKA_BOOTSTRAP=<bootstrap-server-from-sas-key-authentication-page>
FABRIC_KAFKA_TOPIC=<topic-name-from-the-kafka-tab>
FABRIC_KAFKA_CONNECTION_STRING=<connection-string-primary-key>
Enter fullscreen mode Exit fullscreen mode

The Fabric documentation describes the authentication settings as security.protocol of SASL_SSL, sasl.mechanism of PLAIN, a username equal to the literal text $ConnectionString, and the connection string as the password. Copy the bootstrap server exactly as the page shows it, including the port.

Before the Eventstream can write to the Eventhouse, the tables must exist. Run the following script in the KQL queryset of the database. It creates a landing table, a typed table, a function that flattens the OTLP structure, and an update policy that runs the function for every ingested batch.

.create table RawClaudeLogs (resourceLogs: dynamic)

.create table ClaudeEvents (
    Timestamp: datetime,
    EventName: string,
    SessionId: string,
    PromptId: string,
    Model: string,
    QuerySource: string,
    CostUsd: real,
    DurationMs: real,
    InputTokens: long,
    OutputTokens: long,
    CacheReadTokens: long,
    CacheCreationTokens: long,
    ToolName: string,
    Success: string
)

.create-or-alter function ExpandClaudeEvents() {
    RawClaudeLogs
    | mv-expand rl = resourceLogs
    | mv-expand sl = rl.scopeLogs
    | mv-expand lr = sl.logRecords
    | mv-apply a = lr.attributes on (
        summarize attrs = make_bag(bag_pack(tostring(a.key), a.value))
    )
    | extend EventName = replace_string(tostring(attrs["event.name"].stringValue), "claude_code.", "")
    | project
        Timestamp = unixtime_nanoseconds_todatetime(tolong(lr.timeUnixNano)),
        EventName,
        SessionId = tostring(attrs["session.id"].stringValue),
        PromptId = tostring(attrs["prompt.id"].stringValue),
        Model = tostring(attrs["model"].stringValue),
        QuerySource = tostring(attrs["query_source"].stringValue),
        CostUsd = todouble(coalesce(tostring(attrs["cost_usd"].doubleValue), tostring(attrs["cost_usd"].stringValue), tostring(attrs["cost_usd"].intValue))),
        DurationMs = todouble(coalesce(tostring(attrs["duration_ms"].doubleValue), tostring(attrs["duration_ms"].stringValue), tostring(attrs["duration_ms"].intValue))),
        InputTokens = tolong(coalesce(tostring(attrs["input_tokens"].intValue), tostring(attrs["input_tokens"].stringValue), tostring(attrs["input_tokens"].doubleValue))),
        OutputTokens = tolong(coalesce(tostring(attrs["output_tokens"].intValue), tostring(attrs["output_tokens"].stringValue), tostring(attrs["output_tokens"].doubleValue))),
        CacheReadTokens = tolong(coalesce(tostring(attrs["cache_read_tokens"].intValue), tostring(attrs["cache_read_tokens"].stringValue), tostring(attrs["cache_read_tokens"].doubleValue))),
        CacheCreationTokens = tolong(coalesce(tostring(attrs["cache_creation_tokens"].intValue), tostring(attrs["cache_creation_tokens"].stringValue), tostring(attrs["cache_creation_tokens"].doubleValue))),
        ToolName = tostring(attrs["tool_name"].stringValue),
        Success = tostring(coalesce(tostring(attrs["success"].boolValue), tostring(attrs["success"].stringValue)))
}

.alter table ClaudeEvents policy update
@'[{"IsEnabled": true, "Source": "RawClaudeLogs", "Query": "ExpandClaudeEvents()", "IsTransactional": false, "PropagateIngestionProperties": false}]'

.alter-merge table RawClaudeLogs policy retention softdelete = 1d
Enter fullscreen mode Exit fullscreen mode

Each Kafka message holds one OTLP export request, so each row in RawClaudeLogs holds one resourceLogs array. The function expands it level by level and converts the attributes of each log record into a property bag. OTLP JSON stores every attribute as a typed object, such as {"intValue": "412"}. Numeric attributes may arrive as strings or as typed numbers, so the function tries each form in turn. Inspect one real record in RawClaudeLogs before trusting the typed table, and adjust the attribute names if they differ.

Back in the Eventstream, add the Eventhouse as a destination. Select direct ingestion, choose the RawClaudeLogs table, and set the data format to JSON. When the mapping page appears, map the top-level resourceLogs field to the column of the same name. Publish the Eventstream, then restart the collector so that it reads the .env values.

docker compose up -d --force-recreate otel-collector
Enter fullscreen mode Exit fullscreen mode

After a few prompts in Claude Code, the query ClaudeEvents | take 10 should return rows. If it does not, query RawClaudeLogs | take 1 first. An empty landing table points to the Kafka connection, and a populated landing table with an empty ClaudeEvents points to the update policy or to an attribute name that differs from the one in the function.

Build the Real-Time Dashboard

Create a Real-Time Dashboard in the same workspace and add the Eventhouse database as a data source. Each tile runs one KQL query. The queries below answer six questions that engineering leads ask about agent use.

// Cost per day by model
ClaudeEvents
| where EventName == "api_request" and Timestamp > ago(30d)
| summarize CostUsd = sum(CostUsd) by bin(Timestamp, 1d), Model
| render columnchart

// Cost by query source: main thread versus subagents
ClaudeEvents
| where EventName == "api_request" and Timestamp > ago(7d)
| summarize Requests = count(), CostUsd = sum(CostUsd) by QuerySource
| order by CostUsd desc

// Cache hit ratio per day
ClaudeEvents
| where EventName == "api_request" and Timestamp > ago(30d)
| summarize CacheRead = sum(CacheReadTokens), Input = sum(InputTokens) by bin(Timestamp, 1d)
| extend CacheHitRatio = round(todouble(CacheRead) / (CacheRead + Input), 3)
| project Timestamp, CacheHitRatio
| render timechart

// Model latency, median and 95th percentile
ClaudeEvents
| where EventName == "api_request" and Timestamp > ago(7d)
| summarize p50 = percentile(DurationMs, 50), p95 = percentile(DurationMs, 95) by Model

// Tool reliability
ClaudeEvents
| where EventName == "tool_result" and Timestamp > ago(7d)
| summarize Calls = count(), Failures = countif(Success == "false"), p95 = percentile(DurationMs, 95) by ToolName
| extend FailureRate = round(todouble(Failures) / Calls, 3)
| order by Calls desc

// Most expensive sessions
ClaudeEvents
| where EventName == "api_request" and Timestamp > ago(7d)
| summarize CostUsd = sum(CostUsd), Requests = count(), Subagent = countif(QuerySource != "repl_main_thread") by SessionId
| top 10 by CostUsd desc
Enter fullscreen mode Exit fullscreen mode

The queries use percentiles for latency, because the mean hides slow requests when most requests are fast. The query that groups by QuerySource is the team version of the Grafana panel for delegated work, and it keeps its history for as long as the Eventhouse retention policy allows. The cache hit ratio is the share of input that the model read from its cache. A ratio that falls after a change in prompts or tools is an early sign that the same context is being paid for repeatedly.

Microsoft Fabric Real-Time Dashboard showing Claude Code cost per day, cost by query source, cache hit ratio, tool reliability and expensive sessions

What the two dashboards tell you about an agent

The dashboards support four decisions about agent design.

The first decision concerns delegation. If subagents account for a large share of cost, review what each subagent loads into its context and whether the task needed delegation at all. The split between the main thread and the subagents gives the first evidence.

The second decision concerns model choice. The cost by model panel shows how much work runs on the larger model and how much on the smaller one, which is the starting point for moving routine tasks to the cheaper model.

The third decision concerns context. A low cache hit ratio means that the model reads the same material at the full input price. Changes that rewrite the start of the context on every turn lower the ratio, and the dashboard shows the day on which it began to fall.

The fourth decision concerns tools. A tool with a high failure rate or a long 95th percentile costs time, because the agent waits and often retries. The tool_result events identify the tool, and the same table can answer whether failures cluster in a few sessions.

Roll it out to a team

On a team, the endpoint should not depend on each developer setting variables by hand. Claude Code reads managed settings, and the documentation shows a configuration that sets the OpenTelemetry variables centrally. When managed settings define the endpoint, the protocol or the credentials, Claude Code removes conflicting developer values at startup, so developers cannot redirect telemetry to another collector.

{
  "env": {
    "CLAUDE_CODE_ENABLE_TELEMETRY": "1",
    "OTEL_METRICS_EXPORTER": "otlp",
    "OTEL_LOGS_EXPORTER": "otlp",
    "OTEL_EXPORTER_OTLP_PROTOCOL": "grpc",
    "OTEL_EXPORTER_OTLP_ENDPOINT": "http://collector.example.com:4317"
  }
}
Enter fullscreen mode Exit fullscreen mode

A shared collector replaces the local one in this setup, and it needs transport security and authentication before it accepts traffic from many machines. The exporter selector variables are not locked by the endpoint setting, so include them in the managed file as shown.

Where this leaves you

The local stack answers what an agent is spending during the current session. The Fabric stack answers how agent use changed across a team over weeks. Both read telemetry that Claude Code produces without any change to the agent itself.

Several limits apply. The deltatocumulative processor has alpha stability, so pin the collector image version once the stack works. Metric names and attribute types come from the products involved, so the dashboard JSON and the KQL function are the two places to adjust after an update. High-cardinality attributes such as the session identifier increase the size of Prometheus data, and the documentation describes variables such as OTEL_METRICS_INCLUDE_SESSION_ID that reduce it.

References

  1. Claude Code monitoring and usage (OpenTelemetry)
  2. Consume events from a Fabric Eventstream with the Kafka endpoint
  3. OpenTelemetry Collector Contrib: Kafka exporter
  4. OpenTelemetry Collector Contrib: deltatocumulative processor
  5. OpenTelemetry Collector Contrib: Prometheus exporter
  6. Grafana documentation: provision dashboards and data sources

Top comments (0)