Jason Lee

Posted on Aug 11

The LLM Observability Tool You Pick Today May Not Exist As You Know It Tomorrow

#llm #observability #ai #devtools

In a five-week stretch between mid-January and late February 2026, the three companies that most engineering teams weigh when picking an LLM observability stack all changed what they fundamentally are. ClickHouse bought Langfuse outright as part of a $400M Series D. Mintlify acquired Helicone and put it into maintenance mode — no more new features, ever, just security patches. And Braintrust raised an $80M Series B and stayed independent, doubling down on being the "eval-first" pure-play.

If you're choosing one of these three right now, that's not trivia. It's the actual decision. The feature checklists for Langfuse, Helicone, and Braintrust have converged enough that on a Tuesday afternoon comparing docs, you'd struggle to tell them apart — all three trace LLM calls, all three run evals, all three ingest OpenTelemetry spans. What differs, and what almost nobody writing "top LLM observability tools" listicles bothers to check, is which of these three still has a team whose job is to keep building the thing you're about to wire into production.

This is a comparison of what each platform does today, but more importantly, of what kind of company you're actually buying into for the next two years.

Why this comparison matters now

LLM observability went from "nice to have" to "the thing that tells you your agent silently started hallucinating refund amounts" sometime in 2025, as more teams moved from demos to production agents that call tools, retrieve documents, and chain multiple model calls per request. Traditional APM tools like Datadog or New Relic were built for exceptions and latency percentiles, not for "the model returned syntactically valid but factually wrong JSON." That gap created an entire product category — trace the LLM call, score the output, catch regressions before they ship — and Langfuse, Helicone, and Braintrust became the three most-adopted names in it.

Then, within one quarter, the category's competitive landscape reshuffled. A team picking today isn't just comparing pricing tables; they're picking a bet on who's still actively shipping in 2027. That's a genuinely different question than "which dashboard has more chart types," and it's the one this article tries to actually answer instead of skating past it.

There's also a structural reason this specific category attracts acquisitions rather than staying independent forever: LLM observability is fundamentally a data-infrastructure problem wearing a developer-tools UI. Agent traces are large, deeply nested (a single request can spawn dozens of child spans across retrieval, tool calls, and sub-agent hops), and need to be both written fast and queried flexibly for evaluation. That's exactly the workload profile that analytics-database companies like ClickHouse are built to serve, which is part of why "an LLM observability startup gets bought by the database it's built on" isn't a one-off story — it's closer to the shape this market was always going to take.

What each one actually does

Langfuse started as an open-source (MIT-licensed core) LLM engineering platform: tracing, prompt management with versioning, LLM-as-a-judge and code-based evaluators, dataset management, and a playground for iterating on prompts against real data. It ships typed SDKs for Python and JS/TS, integrates natively with LangChain, LlamaIndex, the OpenAI SDK, and LiteLLM, and accepts OpenTelemetry spans from anything else. Deployment is genuinely flexible — Langfuse Cloud, a five-minute Docker Compose install, a VM, Kubernetes via Helm, or Terraform modules for AWS/Azure/GCP. As of mid-2026 it sits around 32,900 GitHub stars, and at acquisition time ClickHouse cited over 26,000 SDK installs per month, 6,000+ Docker pulls, and adoption inside 19 of the Fortune 50.

Helicone takes a different architectural stance: it's primarily an AI gateway — a proxy you route requests through — that layers observability, cost tracking, caching, and rate limiting on top, rather than a tracing SDK you instrument your code with. That means integration is often a one-line base-URL swap instead of wrapping every LLM call, which is genuinely the fastest path to "I can see my OpenAI/Anthropic/Gemini spend by endpoint" of the three. It's also open source and self-hostable, and until March 2026 was on a clear trajectory to add deeper eval and agent-observability features to compete more directly with Langfuse and Braintrust.

Braintrust is built eval-first: the trace and the evaluation score live in the same object from day one, not bolted on after the fact. Its differentiators are CI/CD-integrated evals that can block a merge when a quality score regresses, a proprietary trace database called Brainstore built specifically for large nested agent traces (which traditional row-oriented databases handle poorly), and "Topics," a feature that automatically clusters production traces into patterns so teams can spot emergent failure modes without hand-writing every eval upfront. It ships SDKs for Python, TypeScript, Go, Ruby, and C#, and — notably — has no meaningful self-hosted open-source tier; Enterprise offers hybrid/on-prem deployment, but the product is fundamentally cloud-first and commercially licensed.

A fourth name worth a sentence: Arize AI, particularly its open-source Phoenix project, sits adjacent to all three with a heavier focus on ML-style evaluation rigor and drift detection carried over from traditional model-monitoring roots. It's a legitimate option for teams that already think in terms of embeddings and statistical drift, but it's a narrower fit for teams whose primary need is "trace my agent and catch regressions," so I'm treating it as a footnote rather than a fourth full contender here.

Worth being precise about what "evaluation" means across these tools, because the word gets used loosely. All three support three broad scoring mechanisms: LLM-as-a-judge (another model grades the output against a rubric), code-based evaluators (deterministic checks — did the JSON parse, does the output contain a banned phrase, is the extracted number within tolerance), and human-in-the-loop review queues for cases neither of the first two can reliably judge. Where they differ is workflow integration: Braintrust wires evaluators directly into CI so a regression fails the build; Langfuse treats evals as a first-class object you can run ad hoc or schedule against production traces and datasets; Helicone's evaluation surface is the thinnest of the three, reflecting its gateway-first origin — historically closer to "flag this trace for review" than "block this deploy."

How each is actually built

The architecture differences explain a lot of the trade-offs that marketing pages gloss over.

Langfuse's core data layer is ClickHouse itself — the company literally built v3 of its product on the database that would go on to acquire it, switching from Postgres because Postgres became the ingestion/analytics bottleneck at scale. That's not a coincidence; it's the reason the acquisition made technical sense in the first place, and it's also why Langfuse's self-hosted deployment is non-trivial: running it well at production scale means running ClickHouse well, which is its own operational skill.

Helicone's proxy architecture means it sees every request by definition — no SDK instrumentation gaps, no missed spans because someone forgot to wrap a call — but it also means your LLM traffic physically routes through Helicone's infrastructure (or your self-hosted instance of it) before hitting OpenAI or Anthropic. That's a latency and availability dependency that SDK-based tracing doesn't introduce, and it's a meaningfully different risk profile to sign up for, especially now that the team maintaining that proxy has been folded into a documentation company.

Braintrust's Brainstore is the most opinionated architectural bet: a purpose-built database for nested, tree-shaped agent trace data, which the company claims outperforms "traditional" databases on full-text search, write latency, and span load time for this specific workload (Braintrust doesn't publish the comparison numbers on the public pricing/marketing pages, so treat the specific multipliers as vendor-claimed until you benchmark your own traces). The trade-off is that Brainstore is proprietary — you can't self-host your way out of it, and there's no open-core version to fall back on if you want to leave.

There's a second layer to Braintrust's architecture worth flagging: "Loop," a built-in agent that can generate prompts, scorers, and datasets on request, and "Topics," which continuously clusters live production traces against custom facets you define (use case, customer segment, compliance sensitivity, tone) without you having to hand-write a taxonomy first. Both are genuinely novel relative to Langfuse and Helicone's current feature sets — neither ships an equivalent today — but both also mean more of your evaluation logic lives inside Braintrust's proprietary layer rather than in code you own and could port elsewhere.

On the security and compliance axis, all three have converged toward the same checklist at their upper tiers: SOC 2 Type II, HIPAA support, and SSO/SAML are available from Braintrust (Enterprise), Langfuse (Pro and up, with BAAs available), and Helicone (Team and up). None of the three differentiates meaningfully here anymore — treat compliance as table stakes you should verify at your specific tier rather than a deciding factor between them.

What changed vs. before

Eighteen months ago, "which LLM observability tool" was a feature-and-pricing question. Now it's also a going-concern question, and here's the timeline that made it one:

January 16, 2026 — ClickHouse announced it had acquired Langfuse as part of closing a $400M Series D led by Dragoneer Investment Group, with Bessemer, GIC, Index, Khosla, Lightspeed, and others participating. Langfuse's founders (Marc Klingen, Max Deichmann, Clemens Rawert) and the whole team joined ClickHouse. In their own announcement, the founders were explicit: no licensing changes, self-hosting remains first-class, the team keeps building Langfuse, roadmap unchanged.
February 17, 2026 — Braintrust announced an $80M Series B led by ICONIQ Capital, with a16z, Greylock, Elad Gil, and basecase capital returning. Publicly named customers include Notion, Vercel, Ramp, Stripe, Replit, Zapier, Airtable, and Instacart. The stated use of funds: expanding engineering and go-to-market teams, new offices, and new products revealed at Braintrust's user conference.
March 3, 2026 — Mintlify announced it had acquired Helicone, with founders Justin Torre and Cole Gottdank and the team joining Mintlify in San Francisco. Mintlify's own announcement states Helicone moves into maintenance mode: security updates, bug fixes, and "new models" keep shipping, but — in the founders' own words on their blog — "we will not be shipping new features going forward." A follow-up post from the founders frames the move as a bet that the "knowledge layer" (Mintlify's docs/retrieval business) matters more than the observability layer they'd spent three years building.

Read plainly: one of these three vendors told its own users, in writing, that active development has stopped.

Why developers should actually care

Cost. Helicone's published pricing starts free (10K requests/month), $79/month Pro, $799/month Team, and custom Enterprise. Langfuse Cloud runs Free, $29/month Core, $199/month Pro (with a $300/month add-on for enterprise SSO/RBAC), and $2,499/month Enterprise, all with usage-based overage on top of included units. Braintrust's model is structurally different — a platform fee ($0 Starter, $249/month Pro) plus consumption-based charges for processed data ($4/GB then $3/GB) and scored outputs ($2.50 then $1.50 per 1,000), which means your bill scales with eval volume in a way the other two don't as directly. None of these are apples-to-apples without knowing your actual trace volume and eval cadence — get real numbers from each vendor's calculator before assuming the sticker price is the bill you'll get.

Self-hosting and lock-in. This is where the three diverge hardest. Langfuse's MIT-licensed core is self-hostable with no seat caps, no retention limits, and no usage caps — genuinely free if you're willing to run and scale the ClickHouse-backed stack yourself (real talk: that's roughly $3,000–4,000/month in infra and DevOps overhead at medium scale, against $199–300/month for the equivalent managed Pro tier, so "free" self-hosting isn't actually free once you count engineering time). Helicone is also open source and self-hostable, though a maintenance-mode project only gets you as far as its last shipped release — no future architecture improvements are coming. Braintrust has essentially no meaningful self-hosted or open-core path; you're a cloud customer of a VC-backed company, full stop, and your evaluation history and Brainstore-formatted traces live in their infrastructure.

Latency and blast radius. Helicone's gateway model means an outage or slowdown on their infra is a direct outage or slowdown on your model calls, not just a gap in your dashboards. Langfuse and Braintrust's SDK/OTel-based tracing degrades more gracefully — a tracing backend hiccup means you lose some observability data, not that your app stops calling the LLM.

Maintainability, i.e., "will this still be a product in two years." This is the part the acquisition wave actually answers, and it answers it unevenly. Langfuse's founders stayed on their own product with a funded, aligned parent whose core business (real-time analytics infra) benefits directly from Langfuse's continued growth — that's about as good an acquisition outcome as an open-source infra project gets. Braintrust just took venture money specifically to keep building faster, and remaining independent means its roadmap answers to its own customers rather than a parent company's separate product strategy. Helicone's founders were explicit that new feature development has stopped; whatever gap exists today between Helicone and Langfuse/Braintrust on evals, agent observability, or anything else is now a permanent gap, not a temporary one, unless a fork or a well-resourced competitor picks up where they left off.

Practical use cases

Pick Langfuse if you need to keep sensitive trace data inside your own infrastructure (regulated industries, contractual data residency requirements), you already run or are comfortable running ClickHouse-adjacent infra, and you want a genuinely complete open-source tool rather than an open-core teaser. It's also the strongest fit if your stack already leans LangChain/LlamaIndex, given the native integration depth.

Pick Braintrust if evals are the actual bottleneck — you have a quality bar you want enforced automatically pre-merge, not just visibility after the fact — and you're comfortable being a cloud customer of a well-funded startup. The CI/CD quality-gate workflow and Topics' automatic pattern discovery are the most differentiated capabilities in this comparison; nothing in Langfuse or Helicone matches "block the deploy because eval scores regressed" as a first-class, built-in workflow today.

Pick Helicone, cautiously, if you specifically want gateway-style cost tracking and caching with the absolute lowest integration lift (swap a base URL, done), your needs are already fully met by its current feature set, and you're comfortable that "current feature set" is now permanent. It remains a reasonable choice for teams that only need routing, logging, and cost/caching — just budget migration effort later, because "maintenance mode" is a polite way of saying sunset clock, even an unusually slow one.

Mixed/hybrid approaches are common in practice. Plenty of teams run Helicone (or another gateway) purely for routing/caching/fallback in front of multiple model providers, while sending traces to Langfuse or Braintrust for the actual observability and eval workflow — the gateway and the trace-and-eval platform aren't mutually exclusive, and given Helicone's status, that division of labor is arguably the more defensible way to keep using it going forward.

A few more concrete scenarios worth naming, since "pick the eval-first one" only helps if you know what eval-first workflows actually look like day to day. A support-bot team iterating on a system prompt weekly gets the most direct value from Braintrust's playground-to-dataset-to-CI loop: turn last week's worst production traces into a regression dataset with one click, then gate every future prompt change against it. A platform team building an internal LLM gateway for a dozen product teams to share — centralized rate limiting, per-team cost attribution, response caching to cut redundant spend — is closer to Helicone's original design center than either of the other two, frozen roadmap notwithstanding. And a team building agents inside a compliance-heavy environment (health data, financial data) that needs both traces and prompts to physically never leave a VPC has, realistically, one option here: Langfuse self-hosted.

What the marketing pages don't say

Braintrust's Brainstore performance claims (faster full-text search, write latency, span load vs. "the competition") are stated without published comparison methodology or numbers on the public marketing pages — reasonable to be skeptical of specific multipliers until you run your own trace volume through it.

Langfuse's "self-hosting is free" pitch is true of licensing and false of total cost once you count the infrastructure and the DevOps time to run ClickHouse well at production trace volume — several independent teardowns put realistic self-hosted TCO in the same ballpark as, or above, the managed Pro tier once you're past hobby scale.

Helicone's and Mintlify's announcement posts both frame maintenance mode gently ("services will remain live," "security updates... keep shipping") without emphasizing what that means operationally: no roadmap response to competitive feature gaps, no new provider integrations as new model APIs ship, and a support org that isn't growing with your usage.

And all three vendors' comparison pages (yes, including the ones aggregated in this piece) are written by the vendor being compared favorably — treat any "X vs Y" page hosted on x.com's own domain as marketing copy first, evidence second.

One more thing none of the acquisition announcements dwell on: all three deals were framed, by the acquired company's own founders, as unambiguously positive for users — "no changes," "roadmap continues," "your data is safe." That's the correct thing for a founder to say in an acquisition announcement, and in Langfuse's case the evidence since January (continued commits, continued open-source releases) backs it up so far. But five months of continuity after an acquisition is a thin sample size, and "maintenance mode" for Helicone was announced with the same reassuring tone as Langfuse's "nothing changes" — the actual difference between the two outcomes only became visible once you read past the headline to what each post committed to shipping next. Read the fine print, not the framing, when a vendor you depend on gets acquired.

Comparison table

Dimension	Langfuse	Helicone	Braintrust
Core model	SDK/OTel tracing + evals, open-source-first	AI gateway/proxy + logging	SDK/OTel tracing, eval-first, CI/CD gating
License	MIT core, open source	Open source	Proprietary, cloud-only
Self-hosting	Full-featured, unlimited on free tier	Available, frozen at current feature set	Enterprise-only hybrid/on-prem
Entry cloud price	Free / $29 mo (Core)	Free / $79 mo (Pro)	Free / $249 mo (Pro)
Pricing model	Included units + $8/100k overage	Included requests + usage-based	Platform fee + $/GB processed + $/1k scores
Standout feature	Open-source completeness, prompt mgmt	Zero-code proxy integration, caching	CI/CD quality gates, automatic pattern discovery (Topics)
Primary DB	ClickHouse (post-v3)	Proprietary logging store	Brainstore (proprietary, purpose-built)
Corporate status (as of Aug 2026)	Acquired by ClickHouse, Jan 2026 — team intact, roadmap continuing	Acquired by Mintlify, Mar 2026 — maintenance mode, no new features	Independent, $80M Series B Feb 2026
Best fit	Regulated/data-residency-sensitive teams, LangChain-heavy stacks	Multi-provider routing/cost-tracking with minimal code changes	Teams that need enforced quality gates in CI/CD
Biggest risk	Self-hosted TCO creep at scale	Feature set is now frozen	No self-host exit ramp; fully dependent on vendor

Independent read

None of these is a bad tool today — that's precisely what makes the decision harder, not easier. If the three had wildly different feature depth, the choice would make itself. Instead you're picking based on second-order bets: whether you trust an infra database company to keep an open-source observability product healthy inside its portfolio (Langfuse/ClickHouse looks like a genuinely good pairing so far, for what a five-month-old acquisition is worth), whether a well-funded eval-first pure-play stays focused as it scales its GTM team post-Series-B (Braintrust, still to be proven, but the incentives point the right way), or whether "maintenance mode" is a real long-term option for infrastructure you're about to depend on in production (Helicone, and the honest answer is: for narrow use cases yes, for anything you expect to grow into, no).

Which reader profile should pick which

If you're a regulated-industry team or one with hard data-residency requirements, Langfuse's self-hosted MIT core is the only one of the three that actually satisfies "our trace data never leaves our VPC" without an enterprise contract — pick it, and budget for the ClickHouse operational overhead going in.

If your actual pain point is "we ship an agent update and have no idea if quality regressed until a customer complains," Braintrust's CI/CD-gated evals are purpose-built for exactly that workflow, and it's worth the lack of a self-host option if you're fine being a cloud customer of a company that just took funding specifically to keep building.

If you're early-stage, multi-provider, and mostly need cost visibility and caching without touching your codebase, Helicone still does that job well today — just don't plan your two-year roadmap around features it doesn't already have.

And if you're not sure yet, that's a legitimate answer too: all three have generous free tiers, and the actual cost of trying two of them for two weeks on a real workload is lower than the cost of guessing wrong on the vendor whose long-term trajectory you can't fully know in advance.

Which of these three would you actually bet your team's roadmap on for the next two years, and does a vendor's post-acquisition status change your calculus as much as it changed mine writing this?

Sources:

DEV Community