I let three different AIs work in my homelab every day — a coding assistant, a second CLI for pair-work, and a small agent that triages alerts overnight. One evening I realized I couldn't answer a basic question: what is all of this actually costing me? Two burn subscription quota I've already paid for, one spends real API dollars, and none showed up on the Grafana dashboards I'd built for everything else in the rack.
So I fixed it. Every call from every AI in the lab — tokens, latency, cache hits, and dollars — now lands in Prometheus and one Grafana dashboard. This walks through the four measurement legs, the PromQL traps that made my first dashboard lie, and the privacy scrub that makes the screenshots publishable.
Make the addresses your own. Every machine-specific value here is a placeholder: the monitoring host
10.0.0.5, agent host10.0.0.7, Ollama nodes10.0.0.1–10.0.0.3, exporter ports, and any/home/youradminpaths.
Why measure AI at all?
The AI layer has a genuinely weird cost structure. Two interactive CLIs run on flat subscriptions, so their "cost" is quota — a percentage of a weekly allowance. The automated agent calls a hosted API and pays per token. Same lab, three billing models. So the dashboard has two columns: quota burn (a percentage that resets) and real dollars (the metered agent).
Leg 1: Claude Code already speaks OpenTelemetry
The coding CLI needs no wrapper — it has native OpenTelemetry support. Switch it on with env vars:
export CLAUDE_CODE_ENABLE_TELEMETRY=1
export OTEL_METRICS_EXPORTER=otlp
export OTEL_EXPORTER_OTLP_PROTOCOL=grpc
export OTEL_EXPORTER_OTLP_ENDPOINT=http://10.0.0.5:4317
Every session exports claude_code.token.usage (by type — input, output, cache read), claude_code.cost.usage in USD, and claude_code.session.count. Those arrive over OTLP, which Prometheus doesn't scrape directly — so an OpenTelemetry Collector on the monitoring host listens on :4317, applies processors, and re-exposes everything in Prometheus format on :8889.
Leg 2: tail the agent's log
The automated agent doesn't speak OTel, but it writes an honest log line per call: provider, model, token counts, latency, cache hit, and the occasional "fallback activated". A ~200-line stdlib-only Python exporter tails the log with a regex, keeps counters in memory, and serves them on :9109. It also reads the agent's task database (read-only) for queue-depth gauges — so the dashboard shows quality alongside spend: is the agent finishing work, or just burning tokens?
Leg 3: Ollama, measured from the outside
My three-node Ollama cluster was stubborn: Ollama ships no Prometheus endpoint (as of the 0.30 releases). But it leaks what you need in two places — its journald request log (status, latency, caller IP) and the documented /api/ps endpoint (which models are loaded right now). One exporter per node on :9110. The caller-IP label turned out useful: it shows who is using the local models.
Leg 4: the laptop, via Pushgateway
The second CLI (Codex) records tokens and quota in session files, but the laptop sleeps and moves, so Prometheus can't reliably scrape it. That's exactly the Pushgateway case: a systemd user timer parses the session files every 5 minutes and pushes lifetime totals plus quota percentage to :9091.
Wiring it together is four scrape jobs:
scrape_configs:
- job_name: ai_claude_code
static_configs: [{ targets: ["10.0.0.5:8889"] }] # OTel Collector
- job_name: ai_codex
honor_labels: true
static_configs: [{ targets: ["10.0.0.5:9091"] }] # Pushgateway
- job_name: ai_agent
static_configs: [{ targets: ["10.0.0.7:9109"] }] # log-tail exporter
- job_name: ai_ollama
static_configs: [{ targets: ["10.0.0.1:9110", "10.0.0.2:9110", "10.0.0.3:9110"] }]
Three PromQL traps that made v1 lie
Trap 1: per-session counters break increase(). Claude Code's counters are per-session and ephemeral; a short session leaves one sample, and range functions need two, so increase() returns nothing while a real 30k-token session sits invisible. Read the last value each session reported and sum:
sum(max_over_time(claude_code_token_usage_tokens_total[1d]))
Trap 2: composite cost math collapses on absent series. In PromQL, arithmetic with an empty operand makes the whole expression empty — so before any cache reads existed, real spend rendered as $0.00. Guard every component with or vector(0):
(sum(rate(input_tokens[1h])) or vector(0)) * 1.00 / 1e6
+ (sum(rate(cache_tokens[1h])) or vector(0)) * 0.10 / 1e6
+ (sum(rate(output_tokens[1h])) or vector(0)) * 5.00 / 1e6
Trap 3: histogram_quantile returns literal NaN over idle windows. Documented behavior with zero observations — my latency panel drew garbage across every quiet hour. Consumers need to drop non-finite samples; Grafana panels just go sparse when the lab is idle, which is the honest picture.
The privacy scrub: make dashboards publishable
Claude Code's telemetry attaches identity by default — your email, account ids, org id — as labels on every metric. Useful in a company; radioactive on a public screenshot. Going forward, the OTel Collector deletes those before they reach Prometheus:
processors:
attributes/scrub:
actions:
- { key: user.email, action: delete }
- { key: user.account_uuid, action: delete }
- { key: user.account_id, action: delete }
- { key: user.id, action: delete }
- { key: organization.id, action: delete }
Delete by key, not value, so anyone who ever exported from that laptop gets scrubbed. For history already on disk, open a temporary Prometheus admin window (--web.enable-admin-api), delete the identity-labeled series, then close it:
curl -X POST 'http://10.0.0.5:9090/api/v1/admin/tsdb/delete_series?match[]={user_email!=""}'
curl -X POST 'http://10.0.0.5:9090/api/v1/admin/tsdb/clean_tombstones'
# now remove --web.enable-admin-api from the unit and restart again
One deliberate non-deletion: I kept session_id. Dropping it merges per-session cumulative counters into one series (last write wins, totals undercount). Scrub identity; keep cardinality that's structurally load-bearing.
Don't page yourself over a sleeping laptop
My Node Down alert was up == 0. A sleeping laptop is not an outage, so it's now scoped to exclude the AI jobs — a dead AI exporter shows as a dashboard gap, real infra still pages:
up{job!~"ai_.*"} == 0
What I actually watch
Not spend — the cache-hit rate. At ~78% cached input, the agent's metered bill stays in coffee money. The day that rate drops is the day something changed in how it builds prompts, and now I'll see it the same morning. Total cost of the measurement layer: two tiny Python exporters, one collector, one gateway, and an evening.
Read the full version
- Track AI Token Spend in Grafana (full guide)
- Monitor Proxmox with Grafana and Prometheus
- Build a Read-Only Homelab MCP Server
Originally published on peira.dev.
Top comments (6)
The per-session counter trap is a nasty one — I hit the same class of bug with agent-side counters I exported myself:
increase()over a window that caught a single-sample series silently returned empty, and a genuinely busy day rendered as idle. Reading each session's last reported value withmax_over_timeand summing is the only pattern I've found that survives ephemeral sessions, and it's almost never documented anywhere near token metrics.The quota-vs-dollars split is the honest way to model it. I ran a multi-agent setup where one agent paid per token and everything else sat on flat subscriptions, and the dashboard only became trustworthy once I stopped trying to convert quota into an equivalent dollar figure — a percentage that resets weekly and a metered rate are different units, and forcing them into one column made both misleading.
Watching cache-hit rate rather than spend is a sharp choice. My equivalent leading indicator was prompt size drift: when the agent started stuffing more context into each call, the bill moved before anything else did.
One question: do you alert on spend beyond the Node Down scoping? I keep meaning to wire something like "metered spend in the last hour exceeds 3x the trailing weekly average" but I'm wary of alert fatigue from retry and fallback bursts that are technically expected. Did the fallback lines in your agent's log turn out to be spend events worth alerting on, or noise?
Good question — and no, I don't alert on spend at all, on purpose. The only alert in play is the Node Down one, and I actually scoped it to exclude the AI jobs (a sleeping laptop leg isn't an outage). A dead AI exporter just shows as a gap on the dashboard. Spend and cache-hit rate stay there, and the number I watch by eye is cache-hit rate, not dollars.
Your instinct about the "3x trailing average" rule is exactly why I haven't wired it — retry and fallback bursts are technically-expected noise, and I'd rather not train myself to tune out the channel. If I did alert, it'd be on a sustained cache-hit-rate drop rather than raw spend, since that's what moves before the bill does (same leading-indicator logic as your prompt-size drift).
On the fallback lines specifically: in my setup "fallback activated" means the agent dropped to a local Ollama model — so it's zero marginal API cost, the opposite of a spend event. Not worth alerting on as spend, but a genuinely useful behavioral signal (usually the paid provider was down or throttling), so I keep it on the dashboard via the caller-IP label rather than paging on it.
Treating the cache-hit rate as the primary leading indicator rather than aggregate dollar spend gets the economics right. In agentic workflows, prompt caching is what converts quadratic context growth into manageable baseline opex. When a refactor or dynamic preamble silently breaks prefix cache alignment, the cost explosion shows up instantly in token volume before task failure rates ever trigger an alert.
The distinction between subscription quota burn and metered API spend also mirrors options pricing versus spot exposure. Flat-rate CLI subscriptions are essentially capped-downside call options on compute subsidized by the provider's unutilized capacity. Metered agent calls are pure spot consumption. Forcing them into the same dashboard column obscures the fact that one has fixed zero-marginal cost until the ceiling hits, while the other carries linear financial tail risk on every retry loop.
That options-pricing framing is a sharp way to put it — the flat CLI subscriptions really are a capped-downside call on subsidized compute, and the metered agent is pure spot. It's exactly why I refused to collapse them into one dollar column: a weekly-resetting percentage and a metered rate are different units, and any conversion between them made both lie. Keeping quota-burn and metered spend as separate strips was the only version I trusted. And your point about prefix-cache alignment breaking on a dynamic preamble is the real teeth of it — that's precisely the failure the cache-hit panel is there to catch before the token volume even moves.
Nice breakdown. One thing that saved me from misleading totals was labeling metrics with model and source at ingest, instead of trying to recover it from Grafana later. I also track cache hits separately, otherwise a cheaper cached call can look like a sudden usage drop.
Yeah — labeling with model and source at ingest is what made the whole thing tractable; trying to recover it in Grafana after the fact is a losing game. Cache hits as their own series too, for exactly the reason you say. The one that bit me anyway was the composite cost math: multiply the input, cache-read, and output rates by price and sum them, and on a day the lab had produced no cache reads the empty series made the whole PromQL expression empty — real spend rendered as $0.00 until I wrapped every component in "or vector(0)".