DEV Community: Jasmine Park

What wakes you at 2am when an enterprise operator deploys your agent

Jasmine Park — Thu, 02 Jul 2026 08:13:02 +0000

Month one of our enterprise rollout: three 3am pages. None of them were code bugs. None of them were model quality issues. All of them were operational surprises we had not accounted for before the handoff.

We were production-ready. We were not operator-ready. Those are different things.

Here's what the three incidents were, what they cost, and the alert set I now require before any enterprise agent deployment.

Incident one: rate limits hit at 6pm EST

The operator's usage pattern had a daily spike between 5pm and 7pm EST. Their team was closing out the day, reviewing agent outputs, running follow-up queries. Peak usage was 4x average.

Our OpenAI rate limit was sized for average load. At 6:04pm on day eight, we hit it. Every request after that returned 429 errors for eleven minutes. The agent returned a generic error message. Eleven minutes of silent failure during peak business hours.

Nobody had tested for burst traffic. We had tested for average load. The cost of finding this in production instead of before handoff: one executive complaint, two support tickets, and an emergency weekend call.

The fix was multi-provider routing with automatic failover: when provider A returns 429, route to provider B. We had this wired up within 48 hours. We should have had it before day one.

What this incident taught us about operator-readiness: size for the operator's peak, not your average. The operator's usage pattern will not match your test environment.

Incident two: $11,800 overage in 18 days

The operator had one very active team. Four engineers running batch analysis jobs against our agent at hours that made no operational sense (11pm local, Saturdays). Within 18 days, their usage represented 68% of total API cost.

We had a total cost ceiling. We did not have a per-tenant ceiling. When we hit the total ceiling, the entire deployment slowed down. The heavy team's cost was invisible to us until the month-end bill.

The question from the customer's VP of operations on day 20: "Can you break down the cost by team?"

We could not.

This is a FinOps problem that looks like a monitoring problem. Per-tenant cost tagging needs to be in the request headers before the first request goes out. Not added after month-end when someone asks the question.

Cost of finding this in production instead of before handoff: $11,800 overage, one executive conversation, two weeks of retroactive tagging work to approximate the breakdown.

Minimum viable cost attribution setup: tag every request with operator ID, team ID, and use-case ID at the gateway layer. Aggregate daily by tag. Alert when any single tag hits 70% of its monthly budget by day 15.

Incident three: no audit trail for a compliance request

Six weeks in, the operator's compliance officer needed to reconstruct a decision the agent made on a specific document at a specific time. Customer data question. The agent had made a routing decision that the compliance team needed to audit.

We had trace spans in our observability system. We did not have an immutable per-request audit log that showed: which document, which agent version, which model version, which prompt version, what the output was, what confidence score.

Trace spans are not audit logs. They're operational data. An audit log needs to be write-once, timestamped, and correlated with the business object (the document, the customer record, whatever the operator's domain object is).

We spent four days building a retroactive audit log approximation. It was not what the compliance officer needed, but it was the best we could do. The real audit log went live in week eight.

Cost of finding this in production instead of before handoff: one compliance near-miss, four days of engineering time, one uncomfortable meeting.

What operator-ready means vs. what production-ready means

Production-ready is about your system. Uptime, latency, error rates. Metrics you control.

Operator-ready is about what happens when your system runs inside someone else's organization, with their usage patterns, their cost constraints, their compliance requirements, their data.

The three incidents above were all foreseeable. The operator's usage pattern is discoverable before handoff (just ask them). Per-tenant cost attribution is a gateway configuration decision that takes a day to implement. Audit log requirements for regulated industries are documented in their compliance frameworks.

We didn't discover any of these things before the handoff because we didn't ask the right questions before the handoff.

The pre-operator readiness checklist I run now

Before any enterprise agent deployment, I go through five operational questions. These are not code reviews. They're operational configuration checks.

Rate limit sizing: What is the operator's expected peak usage, and what is our per-provider rate limit at that peak? If peak usage exceeds 60% of the rate limit, configure burst handling or multi-provider routing before go-live.

Cost attribution: Is every request tagged with the minimum attribution set (operator, team, use-case) at the gateway layer? If not, implement it before the first request. Do not add it retroactively.

Audit log schema: Does the operator operate in a regulated industry? If yes, map their compliance requirements to a specific log schema before deployment. Generic trace spans do not satisfy financial services, healthcare, or legal audit requirements.

Failover configuration: Is there a secondary provider configured for automatic failover? If not, is there a documented manual procedure and a stated SLA for the outage window?

Cost ceiling configuration: Is there a per-tenant monthly budget ceiling with an alert at 70% of budget? Not a per-account ceiling. Per tenant. An over-active team should not consume another team's budget.

These five checks take about two hours to complete and review. Three incidents in month one took about three weeks to fully resolve, plus the relationship cost.

What I'd page on

If I were setting up the alert set for a new enterprise agent deployment, these are the four alerts I'd configure first:

Alert one: per-tenant request rate above 80% of the tenant's configured rate limit. Not 100%. Leave 20% headroom to investigate before hitting the ceiling.

Alert two: per-request cost moving average above threshold for any single tenant (set the threshold based on the expected per-request cost, alert at 3x). Catches batch jobs and runaway loops before month-end.

Alert three: agent response with no trace ID in the audit log. Means the audit trail has a gap. You want to know about this in real time, not when a compliance officer asks.

Alert four: first-request p99 latency above 2x the steady-state p99. Catches cold-start regressions before the operator's peak usage hits them.

None of these alerts require custom infrastructure. They require that your gateway and your agent emit the right metadata on every request.

Get the metadata right before handoff. Fix the alerts before go-live. The 3am page is not a production incident. It's a pre-deployment checklist item you deferred.

The Langfuse migration that cost us a sprint: how I now budget LLM observability

Jasmine Park — Fri, 26 Jun 2026 21:37:56 +0000

We moved off our first tracer in month eight. The migration took one engineer the better part of a sprint, because the trace data lived in a schema we did not own. Nobody costed that line item on day one. I am writing this so you can.

I run reliability for a small team shipping LLM features. When the pager goes off at 2am, I do not care which dashboard is prettiest. I care about two numbers: what this tool costs me per month, and what it costs me to leave. Those two numbers are the whole story, and they are almost never on the comparison page.

So here are six Langfuse alternatives. For each I tracked both numbers: the monthly bill on the invoice, and the exit bill that only shows up the day you migrate. I compared Helicone, Arize Phoenix, LangSmith, Braintrust, Laminar, and Future AGI traceAI. They all trace LLM calls (prompts, tokens, retrieval spans, latency). The axis that decides your exit cost is whether the trace format is OpenTelemetry-native or a vendor schema. Get that wrong and the migration bill lands later, with interest.

The cost nobody puts on the pricing page

Your monthly invoice is the visible cost. The exit cost is the invisible one: re-instrumenting the app, rebuilding integrations, and losing historical traces when the schema does not travel. If your spans are OTel, the exit cost trends toward zero because the data is portable by construction. If they are proprietary, you are paying a deferred bill every month you stay. Sort on that first.

Helicone. The gateway-first option. You proxy model calls through it and get logging, cost tracking, and analytics with almost no code change. Apache-2.0, self-hostable, roughly 5,800 GitHub stars as of June 2026. On pure observability ergonomics this is one of the strongest picks, and the proxy model means low setup cost. The thing to watch at scale: a gateway in the request path is one more hop to reason about when latency spikes.

Arize Phoenix. The open-source OTel option. Tracing plus evals, self-hostable, around 10,000 stars as of June 2026. Because it is OTel-native, your exit cost stays low. The commercial Arize AX tier adds ML monitoring and enterprise features. If portability is your top line, this and traceAI are the two that keep the invisible bill near zero.

LangSmith. The LangChain-native option. If you live in LangChain or LangGraph, instrumentation is automatic and the developer experience is strong. Proprietary and closed-source, tightly coupled to the LangChain ecosystem. This is the most lock-in of the group: the day-one cost is the lowest, the day-200 cost is the highest. Worth it only if you are certain you are never leaving LangChain.

Braintrust. The polished SaaS option. One of the better eval and observability experiences, and the people who do not page (PMs, leads) tend to like the UI. Proprietary trace schema, closed-source, managed by default. Even on enterprise deployments you operate inside their format, so the exit cost stays on the books.

Laminar. The newer open-source entrant. OTel-based tracing with evals, smaller and younger than Phoenix, in the low-thousands of stars as of June 2026. Lower lock-in on the same OTel logic. The cost to weigh here is maturity, not portability: a smaller project means fewer battle-tested edges, which matters more for an on-call rotation than a demo.

Future AGI traceAI. The instrumentation-layer option. Worth being precise here, because it is not the same kind of thing as the others. traceAI is not an observability dashboard. It is an Apache-2.0, OpenTelemetry-native instrumentation SDK (pip install fi-instrumentation-otel) that emits portable OTel spans for 50-plus frameworks as of June 2026. The spans go wherever you point your collector. Future AGI's broader platform adds evals on top (50-plus metrics under one evaluate() call as of June 2026), but on raw observability ergonomics Helicone and Phoenix are more mature dashboards. Where traceAI earns its place on this list is the exit-cost column: because it speaks OTel, the cost of leaving is roughly the cost of changing a collector endpoint. Code: github.com/future-agi/traceAI.

The two numbers, side by side

Visible cost is easy: read the pricing page, multiply by your span volume, done. Invisible cost is the one that bit me. The open-source OTel tools (Phoenix, Laminar, traceAI as the instrumentation layer) keep your exit near free. The proprietary ones (LangSmith, Braintrust) front-load convenience and back-load the migration. Helicone sits in between: open and portable, with a proxy hop to account for. Pick the lock-in profile you can afford in month eight, then argue about features.

What I'd page on

If I were standing this up again, here is the dashboard and alert set I would build before I cared about anything else:

Trace export success rate below 99 percent over 5 minutes. A silent collector drop is invisible until you need the trace you do not have.

Span ingestion cost per day trending above your budget line. Token spend gets watched; span volume does not, and it scales with traffic too.

P99 added latency from the tracing path above your SLO budget. If the tracer (or proxy) adds tail latency, that is a reliability cost masquerading as observability.

Percent of spans in a portable (OTel) format. This is your exit-cost gauge. If it drifts down because someone added a proprietary integration, you just took on migration debt. Page on it before it compounds.

Dropped-trace rate during incidents specifically. Tracing tends to fail exactly when load is highest, which is exactly when you need it. Alert on the correlation, not just the absolute.

Build those five first. The dashboard you actually page on is cheaper than the migration you did not plan for.

The gateway tax: 6 OpenAI-compatible gateways.

Jasmine Park — Fri, 26 Jun 2026 21:35:09 +0000

On March 14, 2026, our LLM bill came in at $9,140 for the month, up from about $5,200, and I could not tell you which team spent it. The gateway in front of every provider emitted one cost line and one trace span per request, all tagged service=llm-gateway, so the platform team ate the whole overage in the FinOps review while three product teams shrugged.

That month is the reason I now treat cost attribution as a gateway design decision, not an afterthought. If you cannot answer "which team, which feature, which key spent this" from the layer every call already passes through, you will answer it never. This is a comparison of the OpenAI-compatible LLM gateways I have evaluated for exactly that job: LiteLLM, Portkey, Helicone, Cloudflare AI Gateway, and Bifrost, plus one newer open-source entrant I introduce in the comparison table below. The lens is an SRE lens. What does it cost you in p99, and how granularly can you bill it back.

TL;DR

Cost attribution belongs at the gateway, not in each app's SDK and not in your provider's dashboard. The gateway is the one chokepoint every call crosses, so it is the only place where per-team, per-feature, per-key spend is both complete and consistent.

Every OpenAI-compatible gateway you put in that path adds latency. Call it the gateway tax. It is real, it is usually single-digit milliseconds at the proxy hop, and it varies with what you turn on (caching, guardrails, semantic lookups). The tax is not the deciding factor for most teams, because provider latency dwarfs it. What actually differs across gateways, by a lot, is attribution granularity: whether you can slice spend by virtual key, by route, by user, and whether the cost shows up as a first-class OpenTelemetry span attribute or as a number you have to scrape out of a dashboard later.

So the decision rule is short. Pick the gateway whose tax you can afford at your p99 budget, and whose attribution you can actually bill against. Most teams over-index on the first half and never check the second. Then March happens.

One honesty note up front, because it matters for how you read everything below. We did not re-run a latency benchmark across these six gateways on one rig. Anybody who hands you a clean cross-vendor p99 table either ran a heroic apples-to-apples harness (rare) or is quietly comparing numbers each vendor measured on different hardware against different upstreams (common). Where I cite latency, it is the vendor's own published number, labeled as such. The capability columns (self-host, caching type, attribution granularity, OTel-native, guardrails, license) are checked against each project's public docs and READMEs, because those are verifiable and they are what you will actually live with.

Why not the app SDK, and why not the provider dashboard

Before the table, kill the two alternatives, because most teams reach for one of them first and it is why their numbers never reconcile.

Cost attribution does not belong in each app's SDK. The pitch is seductive: every service instruments its own OpenAI client, tags spend with its own team name, ships it to your metrics backend. In practice you now have N implementations of "compute token cost" drifting against each other. One team is on an old pricing table. One forgot to count cached input tokens at the discounted rate. One service calls the provider directly in a cron job and bypasses instrumentation entirely, so that spend is simply invisible. When the provider changes per-token pricing (they do, quietly), you are editing N codebases to stay correct. SDK metering is great for in-process latency spans. It is a bad system of record for dollars, because the source of truth is smeared across every repo and every deploy cadence.

Cost attribution does not belong in the provider dashboard either. The OpenAI or Anthropic billing console knows your org spent the money. It does not know your org chart. It cannot tell you that team-checkout spent $4k and team-search spent $300, because your teams are not a concept the provider has. The best you get is per-API-key, and only if you had the discipline to mint one key per team up front and never share them, which under load nobody does. Multi-provider makes it worse: now you are stitching three billing consoles, three export formats, three currencies of "cost," into one spreadsheet a human maintains by hand. That spreadsheet is wrong by the second week.

The gateway is the only layer that sees every request, knows which credential made it, can compute cost once against one pricing table, and can stamp that cost onto a span before the response leaves the building. That is the whole argument. Now, which gateway.

Definitions, so the table means something

Two terms do all the work in this post. Pin them down before you read the comparison.

Cost-attribution granularity is the finest dimension along which the gateway can split spend without you doing post-hoc log surgery. I rank it in three tiers:

Per-key: the gateway issues virtual keys (its own keys, mapped to upstream provider keys) and tracks spend and budget per virtual key. You hand team-checkout a virtual key, and its spend is isolated. This is the floor for billing back, and honestly it is enough for most orgs.
Per-route / per-model: spend split by which model or endpoint served the call, so you can see that GPT-4-class traffic is 80% of cost while being 10% of calls.
Per-user / per-metadata: arbitrary tags (end-user id, feature flag, tenant) attached at request time and queryable later. This is what you need for usage-based billing to your customers, not just internal chargeback.

A gateway that only gives you per-key is fine for internal FinOps. A gateway that gives you per-user metadata is what you need if you resell LLM features and bill your customers per seat.

The gateway tax is the latency the gateway hop adds on top of provider latency. It has a floor (the proxy itself: parse, auth, route, re-serialize) and a variable part (every feature you enable adds a little: an exact-cache lookup is cheap, a semantic-cache vector search is not free, each inline guardrail is a synchronous scan). The tax is paid on every request that is not a cache hit. On a cache hit you skip the provider entirely and the gateway saves you latency, which is the one case where the tax goes negative. The mistake teams make is benchmarking the bare proxy, seeing 2 ms, and budgeting as if guardrails and semantic cache are free. They are not. Measure the tax with your real feature set on, or do not quote it.

And again, the number you measure on your rig is not comparable to the number a vendor measured on theirs. Different CPU, different upstream, different concurrency, different request body size. Treat every cross-vendor latency claim, including the ones in this post, as directional.

The comparison

Read this as capabilities first, latency last. The capability columns are what you live with daily. The latency column is vendor-published and not re-run by us, so it is the least load-bearing thing here.

Gateway	Self-host?	Caching (exact / semantic)	Cost-attribution granularity	OTel-native?	Inline guardrails?	License	Verdict
LiteLLM	Yes	Exact (Redis/in-mem/disk/S3/GCS) + semantic (Qdrant/Redis)	Per-key, per-team, per-user (virtual keys + budgets + spend tags)	Via OTel callback/integration	Via plugins + Guardrails hooks	MIT (OSS); paid Enterprise tier	Broadest provider + ecosystem coverage. Default pick if you want the biggest model zoo.
Portkey	Yes (gateway is OSS; full platform is SaaS)	Simple (exact) + semantic	Per virtual key + metadata tags; rich SaaS dashboards	Partial / via integrations	Yes (integrated Guardrails)	Gateway MIT; platform proprietary SaaS	Most polished managed dashboards and config UI. Default if you want a hosted control plane, not a DIY one.
Helicone	Yes (self-host available)	Exact-match only (cache-key hash)	Custom properties (per-user / per-feature) via metadata; per-key	OTLP ingest (observability-first)	Limited / not the focus	OSS (observability platform)	Observability-first, not a routing-heavy gateway. Default if logging + analytics is the job.
Cloudflare AI Gateway	No (Cloudflare edge, cloud-only)	Caching (exact); no documented semantic cache	Per-request analytics, basic metadata; provider/token/cost metrics	No documented OTel export	Not the focus	Proprietary (managed service)	Zero-ops edge gateway. Default if you are already all-in on Cloudflare and want one toggle.
Bifrost	Yes	Semantic caching (exact also supported)	Hierarchical budgets: virtual keys, teams, customers	Yes (Prometheus + OTel/tracing)	Yes (plugin middleware / enterprise guardrails)	Apache-2.0 (Go)	Fast Go OSS gateway with strong budget hierarchy. Default if you want OSS + native budgets and live in Go.
Future AGI Agent Command Center	Yes (single Go binary)	Exact (6 backends) + semantic (4 backends)	Per virtual key budgets/quotas + per-request cost on the span	Yes, OTel-native (W3C trace context) + Prometheus `/metrics`	Yes, 18 built-in scanners + external adapters	Apache-2.0 (Go)	End-to-end OSS platform where the gateway is one piece beside eval/observability. Default if you want OTel + Prometheus + caching + guardrails in one binary.

Notes on the latency column, deliberately kept out of the table because it is not comparable: LiteLLM publishes proxy-overhead figures in the single-digit-millisecond range on their own harness; Future AGI publishes a vendor benchmark of roughly +1.4 ms P95 added by three inline guardrails and a lower added-latency figure than LiteLLM measured on Future AGI's own rig (their numbers, their methodology, not verified by us); Bifrost publishes its own low-microsecond internal-selection numbers. None of these were measured against each other. Do not put them in a slide as if they were.

Gateway by gateway

LiteLLM

The one with the longest provider list and the deepest ecosystem. If a model exists, LiteLLM probably has a route to it, and the litellm SDK is already in half the agent frameworks you will touch. For attribution it is genuinely strong: virtual keys, budgets, and spend tracking down to key, team, and user, plus cache (exact via Redis and friends, semantic via Qdrant). OpenTelemetry is available through its callback/integration system rather than being the native wire format, which means you wire it up rather than getting it for free. The tax is the usual proxy hop; LiteLLM publishes single-digit-ms overhead on their own harness. The cost of all that breadth is configuration surface: there is a lot of it, and a lot of ways to hold it wrong.

Choose LiteLLM when your priority is provider coverage and ecosystem fit, and you have someone who will own the config.

Portkey

The most polished managed experience. The gateway core is open source and you can run it with npx @portkey-ai/gateway, but the part people actually pay for is the hosted control plane: the dashboards, the config UI, the virtual-key and metadata management without you standing up storage. Caching is simple plus semantic, guardrails are integrated, attribution is per-virtual-key plus metadata tags. If you want to hand a non-platform team a screen where they can see their own spend without you building it, Portkey is the shortest path. The trade is that the nice parts are SaaS and proprietary, so the dependency is on Portkey-the-company, not just Portkey-the-binary.

Choose Portkey when you want a managed control plane and dashboards out of the box, and SaaS dependency is acceptable.

Helicone

Observability-first. Helicone is excellent at logging every request, tagging it with custom properties, and giving you analytics over that, including per-user and per-feature cost slicing via metadata. Caching is exact-match only (the cache key is a hash of URL, body, and relevant headers, so "Hello" and "Hi" are different entries). It is self-hostable and open source, and it leans into OTLP-style ingest because its center of gravity is the observability plane, not heavy multi-provider routing or failover. If your real problem is "I cannot see what my LLM calls are doing," Helicone solves that cleanly. If your real problem is "I need 15 routing strategies and inline guardrails," it is not aimed there.

Choose Helicone when logging, analytics, and per-feature cost visibility are the job and routing is secondary.

Cloudflare AI Gateway

The zero-ops option. It runs on Cloudflare's edge, so there is no binary to operate and no SPOF you own (you inherit Cloudflare's). It does caching and gives you analytics: request counts, tokens, cost. What you do not get, per the public docs, is self-hosting, a documented OpenTelemetry export, or deep per-team attribution beyond request-level metadata. It is the right answer when you are already on Cloudflare, you want one dashboard and one toggle, and your attribution needs stop at "roughly how much, roughly where."

Choose Cloudflare AI Gateway when you want a managed edge gateway with near-zero ops and you already live on Cloudflare.

Bifrost

A fast Go OSS gateway (Apache-2.0) with a genuinely good cost model: hierarchical budgets across virtual keys, teams, and customers, which maps cleanly onto chargeback. It ships native Prometheus metrics and distributed tracing / OTel, semantic caching, and a plugin middleware system for analytics and guardrail-style logic. It is newer and the ecosystem is smaller than LiteLLM's, so you trade provider breadth for a tight, performant core and a budget hierarchy that is built in rather than bolted on.

Choose Bifrost when you want OSS, native budget hierarchy, and Prometheus + OTel, and you are comfortable in the Go ecosystem.

Future AGI Agent Command Center

An OpenAI-compatible gateway shipped as a single Go binary, Apache-2.0, open source (repo at github.com/future-agi). As of June 2026 it ships 15 routing strategies, two-tier caching (6 exact-match backends and 4 semantic backends), and 18 built-in guardrail scanners plus adapters for external guardrail vendors. The piece that matters for this post: it is OpenTelemetry-native using W3C trace context and also exposes a Prometheus /metrics endpoint, and it tracks per-virtual-key budgets and quotas, so cost can ride on the span rather than living only in a dashboard. It also ships a committed, reproducible benchmark harness (a bench/ directory with a mock upstream), which I respect more than a marketing number, because it means you can re-run their claim instead of trusting it.

On their own published benchmark (vendor numbers, not verified by us), three inline guardrails add roughly +1.4 ms at P95, and they claim lower added latency than LiteLLM measured on their rig. Same caveat as everywhere else: their hardware, their upstream, their methodology. The honest positioning: LiteLLM still has the broadest provider and ecosystem coverage, and Portkey has the more polished managed SaaS and dashboards. Future AGI's actual edge is that the gateway is one component of an end-to-end open-source platform that also does eval and observability, with native OTel plus Prometheus and built-in caching and guardrails in a single binary, so you are not assembling four tools to get attribution onto a span.

Choose Agent Command Center when you want OTel + Prometheus + caching + guardrails in one OSS binary, and you value the gateway being part of one eval/observability platform.

The diagram you should draw on your whiteboard

Figure: the gateway is the one layer every call crosses. Stamp cost on the OpenTelemetry span at GOVERN/COST and attribution stays complete and consistent.

The single most important thing in that diagram is where the span is emitted. It is emitted inside the gateway, at the govern/cost control point, after the gateway has resolved the credential and computed the cost. That is what makes attribution complete (every call crosses it) and consistent (one pricing table, one cost function). Move that emission into each app and you reintroduce every drift problem from the "why not the SDK" section above.

Honest limitations: where every one of these adds risk

No gateway is free of downside. If you put one in your hot path, you have signed up for these, regardless of vendor.

Single point of failure. Every request now depends on the gateway being up. A managed edge service (Cloudflare) trades your SPOF for theirs, which may be a better or worse bet than your own uptime. A self-hosted binary (LiteLLM, Bifrost, Future AGI) is yours to make HA: run more than one replica, put a real load balancer in front, and test failover before you need it. "We deployed one gateway pod" is not a control plane, it is an incident waiting for a node drain.

Cache poisoning and stale answers. Semantic caching is the feature most likely to bite you. A vector-similarity hit can return a cached answer for a prompt that is close but not equivalent, and now one user sees another user's response, or a stale answer to a changed question. Exact caching is safer but still leaks across users if your cache key does not include the right scoping. Scope cache keys per tenant where correctness matters, and keep semantic caching off for anything with PII or per-user state until you have measured the false-hit rate.

Span-cardinality blowup. The fix for attribution (rich tags on every span) is also the way you melt your metrics backend. Put end_user_id as a label on a Prometheus metric and you have just created one time series per user. That is a cardinality bomb. Keep high-cardinality identifiers (user id, request id) on traces and logs, where high cardinality is fine, and keep your metric labels low-cardinality (team, model, provider, cache_hit). Conflating the two is the most common way an attribution rollout pages the observability team instead of the FinOps team.

A pasteable artifact: per-key budget plus OTel export

Here is a minimal, runnable setup for one gateway (LiteLLM, because its config is the most widely deployed and the spend tracking is mature), showing a per-virtual-key budget and OpenTelemetry export, plus the queries that turn it into a bill-back.

docker-compose.yml:

services:
  litellm:
    image: ghcr.io/berriai/litellm:main-stable
    ports:
      - "4000:4000"
    environment:
      OPENAI_API_KEY: ${OPENAI_API_KEY}
      LITELLM_MASTER_KEY: ${LITELLM_MASTER_KEY}
      DATABASE_URL: postgres://litellm:litellm@db:5432/litellm
      # Send OTel spans to your collector
      OTEL_EXPORTER_OTLP_ENDPOINT: http://otel-collector:4317
    command: ["--config", "/app/config.yaml", "--port", "4000"]
    volumes:
      - ./config.yaml:/app/config.yaml:ro
    depends_on:
      - db

  db:
    image: postgres:16
    environment:
      POSTGRES_USER: litellm
      POSTGRES_PASSWORD: litellm
      POSTGRES_DB: litellm
    volumes:
      - litellm-pg:/var/lib/postgresql/data

volumes:
  litellm-pg:

config.yaml:

model_list:
  - model_name: gpt-4o
    litellm_params:
      model: openai/gpt-4o
      api_key: os.environ/OPENAI_API_KEY

litellm_settings:
  # Emit an OpenTelemetry span per request, with cost + tokens as attributes.
  callbacks: ["otel"]
  # Track and persist spend so it can be queried per key/team/user.
  store_model_in_db: true

general_settings:
  master_key: os.environ/LITELLM_MASTER_KEY
  database_url: os.environ/DATABASE_URL

Mint a virtual key for one team, with a hard monthly budget, so March cannot happen silently:

curl -s http://localhost:4000/key/generate \
  -H "Authorization: Bearer $LITELLM_MASTER_KEY" \
  -H "Content-Type: application/json" \
  -d '{
        "key_alias": "team-checkout",
        "models": ["gpt-4o"],
        "max_budget": 500,
        "budget_duration": "30d",
        "metadata": {"team": "checkout", "cost_center": "cc-4471"}
      }'

That key now refuses traffic once team-checkout crosses $500 in a 30-day window, and every call it makes carries team=checkout into the spend store and onto the OTel span.

Attributing spend to a team comes from the gateway's own spend store. With LiteLLM's spend logs in Postgres, the bill-back for last month is one query:

SELECT
  metadata ->> 'team'      AS team,
  COUNT(*)                 AS requests,
  ROUND(SUM(spend)::numeric, 2) AS usd
FROM "LiteLLM_SpendLogs"
WHERE "startTime" >= date_trunc('month', now()) - interval '1 month'
  AND "startTime" <  date_trunc('month', now())
GROUP BY 1
ORDER BY usd DESC;

And for the live alerting view, scrape low-cardinality cost metrics into Prometheus and rank current-month spend by team. With a gateway that exposes a per-team cost counter (label team, deliberately low-cardinality), the PromQL is:

topk(5,
  sum by (team) (
    increase(llm_gateway_cost_usd_total[30d])
  )
)

Keep team, model, and provider as metric labels. Keep end_user_id and request_id out of metrics and on the trace instead. That one discipline is the difference between an attribution dashboard and a cardinality incident.

Paste this into your PRD

A scenario matrix for the decision review, so the next person does not re-derive it.

Scenario	Priority	Default pick	Escalate to	Why
Internal chargeback, many providers	Provider breadth + per-team spend	LiteLLM	Bifrost (if you want native budget hierarchy in Go)	Biggest model zoo, mature virtual keys and spend tracking; budgets get you per-team bill-back.
Non-platform teams need their own spend screen	Managed dashboards, low build cost	Portkey	LiteLLM self-host (if SaaS dependency is a no)	Hosted control plane and config UI mean you do not build the dashboard yourself.
"I cannot see what my LLM calls do"	Logging + per-feature cost visibility	Helicone	Future AGI ACC (if you also need routing + guardrails)	Observability-first with custom-property attribution; exact-match cache.
Already on Cloudflare, want near-zero ops	One toggle, no binary to run	Cloudflare AI Gateway	Any self-hosted gateway (when you outgrow request-level attribution)	Edge-managed, no SPOF you operate; attribution stops at request-level metadata.
Want OTel + Prometheus + cache + guardrails in one OSS binary	One platform, attribution on the span	Future AGI Agent Command Center	LiteLLM (for wider provider coverage) or Portkey (for managed dashboards)	Native OTel (W3C) + Prometheus, two-tier cache, 18 guardrail scanners in one Go binary, part of an eval/observability platform.
Resell LLM features, bill your customers per seat	Per-user / per-metadata attribution	LiteLLM or Portkey (rich metadata)	Helicone (for the analytics layer on top)	You need arbitrary per-user tags queryable later, not just per-key.

What I'd page on

This is the on-call checklist for a gateway in your hot path. If you adopt one of these gateways and do not wire these alerts, you are flying blind and the next $9k month is already in flight.

Gateway p99 latency, by route. Page if p99 of the gateway-added overhead (gateway span duration minus upstream span duration) exceeds your budget for 5 minutes. This is the gateway tax going bad. Separate the proxy hop from provider latency or you will blame the wrong layer at 2am.
Gateway error rate and saturation. Page on 5xx rate from the gateway above baseline, and on CPU saturation, because at high concurrency CPU is the bottleneck, not the network. A saturated gateway fails every team at once.
Per-team budget burn. Page (or auto-throttle) when any virtual key crosses, say, 80% of its monthly budget before the month is 80% over. This is the alert that would have caught March on March 6, not March 31.
Total spend rate-of-change. Page on day-over-day total LLM spend up more than X%. A runaway retry loop or a new feature shipping uncapped shows up here first, hours before the invoice.
Cache hit rate drop. Page if cache hit rate falls below your assumed floor, because your cost model and your latency budget both silently assumed those hits. A cache that quietly stopped hitting is a bill increase and a latency regression in one.
Semantic-cache false-hit signal. If you run semantic caching on anything user-facing, alert on user reports or eval-detected wrong answers correlated with cache hits. This is correctness, not cost, and it is the one that becomes a postmortem instead of a FinOps slide.
Span cardinality / metrics ingestion. Page if your metrics backend's active series count jumps after a deploy. That is usually someone putting a user id on a metric label. Catch it before it takes down the observability stack.
Provider failover events. Alert (not page) when the gateway fails over between providers, so a silent provider degradation does not hide inside your routing logic until the bill from the more expensive fallback shows up.

Pick the gateway whose tax you can afford and whose attribution you can bill against. Then wire the eight alerts above, because the gateway is now load-bearing infrastructure, and load-bearing infrastructure gets a pager.

Capability claims here reflect each project's public docs and READMEs as of June 2026. Latency figures are vendor-published on each vendor's own harness, not re-run on a common rig, and are not comparable across vendors. Future AGI's gateway (Agent Command Center) is open source at github.com/future-agi.

Langfuse alternatives: 6 LLM observability tools, sorted by the thing that bites you in month eight

Jasmine Park — Fri, 19 Jun 2026 09:33:19 +0000

TL;DR

I went looking for Langfuse alternatives after living with a proprietary tracer for eight months and then paying to migrate off it.

I compared six options:

Helicone
Arize Phoenix
LangSmith
Braintrust
Laminar
Future AGI traceAI

They all trace LLM calls.

The axis that actually mattered was OpenTelemetry-native (OTel) vs proprietary tracing, because that's what determines whether you can leave without re-instrumenting everything.

Four of the six are open-source, ranging from roughly 200 to 10,000 GitHub stars (June 2026). That spread turned out to predict almost nothing about what I actually cared about: portability.

The Axis That Bites You in Month Eight: Whose Traces Are These?

Every tracer captures:

LLM calls
Prompts
Tokens
Retrieval events

The question nobody asks on day one and everybody regrets on day 200:

Is the trace format OpenTelemetry (portable) or the vendor's own schema (locked to their dashboard)?

If it's proprietary, switching tools later often means:

Re-instrumenting your application
Rebuilding integrations
Losing historical trace data

I learned this the expensive way.

So today I sort observability tools by lock-in first and features second.

The Six Tools, Sorted by Lock-In

1. Helicone

The gateway-first open-source pick.

Often the first Langfuse alternative people mention.

You proxy model calls through Helicone and get:

Logging
Cost tracking
Analytics

with very little code change.

Highlights

Open-source (Apache-2.0)
Self-hostable
~5,800 GitHub stars (June 2026)

Best for: teams that want a fast observability layer with minimal engineering effort.

2. Arize Phoenix

The open-source OTel pick.

Phoenix combines:

OTel-based tracing
Evaluations
Self-hosted deployment

The core project is free and open-source.

Arize's commercial offering (AX) adds:

Enterprise capabilities
Advanced ML monitoring

Highlights

Open-source
OTel-native
Self-hosted
~10,000 GitHub stars (June 2026)

Best for: teams that want portable tracing and full ownership.

3. LangSmith

The LangChain-native pick.

If you're already using LangChain or LangGraph, LangSmith provides:

Automatic instrumentation
Deep framework integration
Strong developer experience

The tradeoff is coupling.

Highlights

Proprietary
Closed-source
Closely tied to the LangChain ecosystem

Best for: teams fully committed to LangChain.

Most lock-in of the group.

4. Braintrust

The polished SaaS pick.

Braintrust has one of the strongest experiences for:

Evaluations
Observability
Cross-functional visibility

Non-technical stakeholders tend to like the UI.

Highlights

Proprietary trace schema
Closed-source
Managed SaaS by default
Enterprise deployment options available

Even in enterprise deployments, you're still operating within their trace format.

Best for: organizations that prioritize product polish over portability.

5. Future AGI traceAI

The no-lock-in instrumentation pick.

traceAI is different from the others.

It is not an observability platform.

It is an Apache-2.0 OpenTelemetry instrumentation layer that captures:

LLM calls
Prompts
Tokens
Retrieval
Agent steps

and exports them to any OTel-compatible backend:

Datadog
Grafana
Jaeger
Vendor platforms

In other words:

It handles instrumentation, not dashboards.

If you want a polished product out of the box, tools like Langfuse or Helicone are more complete.

If you want portable traces that you own, this is the lightest approach I found.

Highlights

Apache-2.0
OTel-native
Backend-agnostic
~200 GitHub stars (June 2026)

Best for: teams optimizing for long-term portability.

It's also the youngest project here, so think of it as a bet on the instrumentation-first approach rather than a mature platform.

6. Laminar

The newer open-source pick.

Laminar combines:

OTel-based observability
Evaluations
Modern architecture

It is newer than Phoenix but worth evaluating.

Highlights

Apache-2.0
OTel-native
~3,000 GitHub stars (June 2026)

Best for: teams looking for a modern open-source observability stack.

My Take

I'm not crowning a winner.

Different tools optimize for different priorities.

If you want...	Look at...
Fastest onboarding	Helicone
Self-hosted OTel observability	Phoenix or Laminar
Deep LangChain integration	LangSmith
Polished SaaS workflows	Braintrust
Pure OTel instrumentation	traceAI

The proprietary tools are completely reasonable choices.

Until the day you want to leave.

What I Would Do Differently

I would choose OTel-native instrumentation from day one.

Not because proprietary tools are bad.

In many cases they're actually easier and more pleasant to use.

But the cost of switching is paid later through:

Re-instrumentation
Migration effort
Lost historical context

And on day one, you have no idea whether you'll outgrow the tool.

The argument is simple:

Instrument once with OpenTelemetry. Point it at whatever backend you want. Change backends without changing application code.

That's the entire case for OTel.

And it's the one strong opinion I came away with.

FAQ

Is Langfuse bad?

No.

Langfuse is genuinely good.

It's self-hostable, widely adopted, and has roughly 29,000 GitHub stars as of June 2026.

This post is about alternatives and tradeoffs, not criticism.

If I'm OTel-native, does that mean I don't get a dashboard?

No.

You still need somewhere to view traces.

Examples include:

Grafana
Datadog
Jaeger
Vendor platforms

OTel-native simply means you can change the backend later without changing instrumentation.

Should I self-host or use a managed service?

Self-host if you need:

Data residency
Infrastructure control
Lower long-term platform dependency

Managed if you want:

Faster setup
Less operational burden
A more polished experience

Open Question

Lock-in is easy to reason about.

What I still struggle to quantify is the value difference between:

A polished proprietary platform
A portable-but-rawer OTel stack

The proprietary tools genuinely save time every day.

The portability benefits only pay off if you actually switch.

I don't have a clean framework for valuing that optionality.

If you've found a useful way to think about that tradeoff, I'd love to hear it.

Per-project LLM cost attribution with OTel spans: the wiring

Jasmine Park — Thu, 04 Jun 2026 21:01:52 +0000

TL;DR. If your LLM bill is one line item on a cloud invoice, you cannot answer "which team spent that." We fixed this by tagging every gateway span with team.id, project.id, and feature.id, plus the OpenInference token-count attributes, shipping those spans through an OTel collector into Tempo, and rolling cost up per team with TraceQL in Grafana. The payoff that sold it internally: one team's monthly spend quietly went from a few hundred dollars to over a thousand because of a retry loop, and the org-level dashboard never flinched. The per-team view caught it in a day. Below is the wiring, the collector config, the rollup query, the alert, and the attributes I tried and threw away.

1. The problem is attribution, not collection

Most teams already collect LLM telemetry. Spans exist, tokens get counted, traces land somewhere. What is missing is the dimension that finance and eng leads actually ask about: who owns this spend. The provider invoice gives you one number per month per API key. If you share keys across services (most people do at some point), that number is useless for chargeback. You cannot tell the platform team's spend from the support-bot team's spend.

So the design goal was narrow. Every LLM call has to carry enough labels that I can group spend by team, by project under that team, and by feature inside that project. Three levels. No more, because deeper than feature and nobody reads the dashboard. I standardized the whole pipeline on OpenTelemetry and OpenInference, and I will state the one opinion plainly: I want the labels, the wire format, and the storage to be things I can swap without rewriting instrumentation. We tag spans with open semantic conventions so the day we change a backend or a dashboard tool, the gateway code does not move. That is a portability decision, not a verdict on anyone's product.

2. Which attributes get tagged, and where

Tag at the gateway, not in each service. We run an LLM gateway (every call to every provider goes through it), so it is the one place that sees model, token counts, and request context together. A new service gets attribution for free as long as it routes through the gateway and forwards the three context headers.

The cost-math group comes straight from OpenInference semantic conventions: llm.model_name, llm.token_count.prompt, llm.token_count.completion. The attribution group is custom, set from request headers: team.id, project.id, feature.id. Cost is not a span attribute. I compute it at query time from token counts and a small price lookup, because prices change and I do not want last quarter's spans frozen at last quarter's rates.

Attribute	What it buys you	Keep or drop
`team.id`	Top-level chargeback. The number a director asks for.	Keep
`project.id`	Splits a team's spend across its services.	Keep
`feature.id`	Which feature drove a spike inside a project.	Keep
`llm.model_name`	Lets you weight tokens by per-model price.	Keep
`llm.token_count.prompt`	Input side of the cost.	Keep
`llm.token_count.completion`	Output side. Usually the expensive half and the one that runs away.	Keep
`user.id`	Per-user spend, in theory. A privacy liability in traces.	Drop
`request.id`	Already covered by the trace and span IDs.	Drop

3. The collector config

OTLP in, batch, set anything the gateway missed, Tempo out. The one processor worth calling out is transform: I use it to backfill team.id with a sentinel when a service forgets the header, so unlabeled spend shows up as unattributed instead of vanishing. Cost with no label is cost you will never find.

receivers:
  otlp:
    protocols:
      grpc: { endpoint: 0.0.0.0:4317 }
      http: { endpoint: 0.0.0.0:4318 }

processors:
  batch: { timeout: 5s, send_batch_size: 1024 }
  transform/attribution:
    trace_statements:
      - context: span
        statements:
          - set(attributes["team.id"], "unattributed") where attributes["team.id"] == nil
          - set(attributes["project.id"], "unknown") where attributes["project.id"] == nil
          - set(attributes["feature.id"], "unknown") where attributes["feature.id"] == nil

exporters:
  otlp/tempo:
    endpoint: tempo:4317
    tls: { insecure: true }

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [transform/attribution, batch]
      exporters: [otlp/tempo]

Two notes from running this. Put transform before batch so the backfill happens per span while the data is still cheap to touch. And keep the price table out of the collector. I tried encoding per-model rates as collector attributes once. Every price change became a config deploy, and the rates drifted out of sync with what we were actually billed. Pricing lives next to the query now.

4. Rolling cost up by team in Grafana

Tempo stores spans, not dollars. So the rollup is two steps: TraceQL pulls token sums grouped by the attribution attributes, and a small price map turns tokens into cost downstream. I start from this, which aggregates output-token counts (the number I watch most, because completion tokens are usually where the money and the runaways are):

{ .team.id = "support-platform" && .llm.token_count.completion > 0 }
| select(.project.id, .feature.id, .llm.model_name, .llm.token_count.prompt, .llm.token_count.completion)
| by(.team.id, .project.id, .llm.model_name)
| sum(.llm.token_count.completion)

Drop the team.id filter and group by it instead for the all-teams board. The grouping by llm.model_name matters: a mini-tier model and a frontier model can differ by more than an order of magnitude per token, so summing raw tokens across models hides which team is expensive because of volume versus model choice. The dollar step is deliberately dumb: a lookup from llm.model_name to input-price and output-price per thousand tokens, multiplied through, summed per team. Keeping it dumb and external is what lets me re-price history when a provider changes rates.

5. The alert: page when a team's output tokens jump

Cost attribution is reporting. The thing that earns its keep is the page. The rule I run is week-over-week on output tokens per team: if this week's completion-token total for any team is more than 2x the same window last week, page. Output tokens, not input, because the runaway failure modes (retry storms, an agent that loops, a prompt-chaining bug that re-asks) all show up as generation volume first. Why 2x week-over-week and not a fixed dollar ceiling: a fixed ceiling either pages constantly for your big teams or never fires for your small ones. A relative jump normalizes across team size on its own. The team whose spend doubled in the story above would have tripped a 2x rule on day one. It did not trip our dollar alert because the absolute number was still small against the org total. Small against the org, doubled for the team, is exactly the blind spot per-team attribution exists to close. Route it to whoever owns the team's budget, not a shared channel where it gets ignored.

6. What I tagged and then dropped

user.id. Per-user spend sounds useful and is occasionally asked for. But putting a user identifier on every span means every trace is now PII, and your whole tracing backend inherits the retention, access, and deletion obligations that come with that. The attribution win did not come close to justifying the compliance surface. Dropped it, have not missed it.

request.id. Pure redundancy. A trace already has a trace ID and every span has a span ID. Anywhere I thought I wanted it, the trace ID was already there and already correct.

The pattern in both: an attribute is only worth tagging if it answers a question the cheaper attributes cannot, and if its cost (privacy, plumbing, drift) is lower than that answer is worth.

FAQ

Why compute cost at query time instead of writing a cost attribute on the span? Prices change and I want to re-cost history when they do. A cost attribute freezes the rate at write time.

Do I need the gateway, or can each service tag its own spans? You can tag per service. I prefer the gateway because it sees model and tokens and request context in one place, so a new service gets attribution by routing through it and forwarding three headers.

Why Tempo specifically? It is what we run, and TraceQL's aggregation over span attributes does the rollup I need. The attribute conventions are OpenInference, so the labels are not tied to Tempo. The point of standardizing on open conventions is that this choice is reversible.

What if a service forgets the attribution headers? The collector backfills unattributed. The spend still shows up, just in a bucket whose name tells me to go fix the instrumentation.

Is week-over-week 2x too noisy? For steady traffic, no. For genuinely spiky workloads, raise the ratio or widen the comparison window. I bias toward a slightly noisy page over a silent doubling.

Open questions

Caching breaks the token-to-cost math. Cached prompt tokens bill at a different rate (sometimes free), and I do not yet tag cache hits cleanly enough to price them right. Streaming and cancelled generations: if a client disconnects mid-stream, what is the honest output-token count, and does the provider bill for tokens generated after the cancel? Feature-level granularity has a ceiling, and I keep wanting per-prompt-version attribution but every level deeper is one more label nobody reads. And whether 2x week-over-week should itself be per-team, since some teams are spiky by nature and one global ratio serves both imperfectly. If you have wired cached-token pricing into a span-based cost model in a way that survives a provider changing its cache rates, I want to hear how.

Span attributes that catch LLM cost regressions before billing does

Jasmine Park — Tue, 02 Jun 2026 19:09:31 +0000

The default OTel + OpenInference span has llm.tokens.input and llm.tokens.output as numeric attributes. Useful for trace-level debugging. Not useful for per-team cost regressions, because nothing groups traces by team.

The 3 attribute additions that earned their keep:

team.id. Tagged on every span at the gateway layer (before the call routes to the LLM provider). This is the column that makes the cost rollup possible. Without it, you can attribute spend to an org but not to a team inside the org.

feature.id. The product feature that triggered the call (chat_assistant, summarizer, rag_search). Lets you see when one feature's token cost spikes vs the overall trend.

llm.model. Already standard in OpenInference but worth flagging: without this, you cannot separate a cheap mini-tier model's spikes from a frontier model's spikes when both are in the same feature.

The daily Tempo + Grafana query (TraceQL):

{ resource.service.name = "llm-gateway" }
| histogram_quantile(0.95, llm.tokens.output_total) by (team.id, feature.id, llm.model)

The alert rule: page when 7-day-trailing average of output-tokens-per-team-per-feature jumps more than 2x week-over-week. We caught a runaway retry loop last quarter that the org-level spend dashboard missed because the total stayed within budget while one team's bill quietly doubled.

What we tried and dropped:

user.id tagging: privacy concerns at scale, and the rollup-by-team covered the use case.

request.id tagging: redundant with the trace_id; just adds cardinality.

Drafted with AI assistance, edited and verified by author.