Jason Lee

Posted on Aug 18

LangSmith, Langfuse, Braintrust, and Helicone Aren't Actually Competing With Each Other

#ai #llm #opensource #observability

If you shipped an LLM feature in 2024, you probably bolted on a logging table and called it observability. If you're shipping an agent in 2026 — something that plans, calls tools, loops, and sometimes fails in ways that don't show up until step fourteen of a nine-step task — that table stopped being enough a while ago, and you're now choosing between four products that all claim to solve "LLM observability" but were actually built to solve four different problems.

That's the part the comparison posts tend to flatten. LangSmith, Langfuse, Braintrust, and Helicone get lined up in a feature-matrix and scored on the same axes, as if the decision were "which one has the most checkmarks." It isn't. Each one metabolizes a different unit of work — a trace, an eval score, an API call — and that choice of unit shapes everything downstream: what it costs you at scale, what it can and can't diagnose, and how painful it is to leave.

This piece is about that dividing line, not a leaderboard.

Why this comparison is a live decision right now

Three things changed in the last year that make this an actual decision point instead of a nice-to-have.

First, production LLM usage moved from single-turn completions to multi-step agents — tool calls, sub-agent delegation, retries, human-in-the-loop approval. A single user request can now fan out into dozens of LLM calls, and "what went wrong" stopped being answerable by reading a request/response log. You need the whole execution graph.

Second, teams that got burned by regressions started treating evals the way they treat unit tests: something that runs in CI, blocks a merge, and produces a score, not a vibe. That's a meaningfully different product requirement than "show me a nice trace viewer."

Third, self-hosting stopped being a niche ask. GDPR and data-residency requirements, plus a general fatigue with usage-based SaaS bills that don't scale predictably, pushed teams to actually price out running observability infrastructure themselves — and discovered that "open source" in this category ranges from genuinely free to bait for an upsell.

Put those three together and the question isn't "which tool is best" — it's "which unit of work do I actually need to optimize for, and am I willing to operate my own database to get it."

There's a fourth driver too, quieter but just as real: LLM API spend stopped being a rounding error on the infra bill. Once a team is running agents that fan out into dozens of calls per request, finance starts asking which feature, which customer, or which prompt is burning the budget — and that's a question none of these four tools answer equally well, because "cost" means something different depending on whether your unit of analysis is a trace, a score, or a raw request.

What each one actually does

LangSmith is LangChain's hosted observability and eval platform. It traces LLM calls, chains, and — increasingly its main selling point — LangGraph agent executions, showing node-by-node state diffs across a graph run. It also does dataset management, prompt versioning, and annotation queues for human review.

Langfuse is an open-source (MIT-licensed core, with a separate ee — enterprise edition — folder under a source-available license) platform covering tracing, evaluation, prompt management, and datasets, shippable as a fully self-hosted stack or as a managed cloud product. It's framework-agnostic by design: it integrates with LangChain, LlamaIndex, the OpenAI SDK, and raw API calls equally, rather than being built around one orchestration framework.

Braintrust treats the eval, not the trace, as the primary object. You define scorers — code-based, LLM-graded, or human — run them against datasets, and Braintrust tracks score deltas across experiments and model/prompt changes over time. Tracing exists, but it's downstream of the eval, not the other way around.

Helicone is architecturally different from the other three: instead of an SDK that instruments your code, it's primarily a proxy. You point your OpenAI/Anthropic-compatible traffic at Helicone's endpoint (or use an async logging integration), and it captures cost, latency, and usage per request with minimal code changes. It layers in caching, rate limiting, and prompt experimentation as proxy-level features.

How they're actually built — and why the architecture matters more than the feature list

This is the part worth slowing down on, because it's the thing that determines what each tool is good at diagnosing, not just what's in the pricing page.

Proxy vs. SDK. Helicone sits in the request path. Every LLM call goes through its infrastructure before it reaches the model provider. That buys you the simplest possible install — often a base-URL change and nothing else — plus request-level features like caching and rate limiting for free, because the proxy is already touching every call. The cost is a genuine architectural trade-off: you've added a network hop to your hot path, and it's a request-level view of the world, not an execution-level one. Helicone can group requests into sessions, but it doesn't reconstruct the full state of a multi-step agent run the way a purpose-built agent tracer does — it sees API calls, not the reasoning between them.

LangSmith, Langfuse, and Braintrust instead instrument your code — SDK wrappers, decorators, or callback hooks — which means no added network hop, but real code changes and framework-specific integration work.

Trace-first vs. eval-first vs. session-first. LangSmith and Langfuse are fundamentally trace-first: the unit you page through is a request or a run, and evaluation is something you attach to traces after the fact. Braintrust inverts this — the unit is a score, generated by running a scorer against a dataset, and traces are there to explain a score, not the other way around. That inversion is what makes Braintrust's CI/CD story clean: a GitHub Action can fail a build because a score dropped, in the same way a test suite fails a build because an assertion failed. Bolting that workflow onto a trace-first tool is possible but always feels bolted on.

Both framings, notably, still treat the request as the atomic unit and add agent features on top. A newer crop of "agent-first" tools flips this again, treating the full multi-turn session as the atomic unit from day one — a reminder that even this four-way comparison isn't the end state of the category.

License architecture. Langfuse's core is MIT — genuinely, unlimited, no seat caps, no retention window, run it on your own Postgres/ClickHouse and keep everything. The commercial layer is thin by design: SSO, SCIM, audit logs, project-level RBAC. That's a meaningfully different posture than "open core" products that gate basic usability behind a paywall. LangSmith and Braintrust both offer self-hosted or VPC deployment for regulated customers, but neither ships a free, fully-featured open-source edition the way Langfuse does — self-hosting them is an enterprise-tier commercial arrangement, not a docker compose up.

What changed versus a year ago

A few shifts are worth flagging because they change the calculus from what you'd have read in a 2024 comparison:

Langfuse leaned harder into "self-host without limits" as its wedge, explicitly positioning the free self-hosted edition as feature-complete rather than a trial. The catch is operational, not licensing: production self-hosting means running ClickHouse yourself (or paying for a managed ClickHouse Cloud/BYOC add-on), and ClickHouse at scale is real infrastructure work most application teams don't want to own.
Braintrust doubled down on eval-as-CI, with native GitHub Actions support that lets teams gate merges on scorer output — a workflow that barely existed as a first-class feature two years ago, when evals were mostly notebooks people ran manually before a release.
LangSmith's differentiation increasingly comes from LangGraph, not from LangChain generically. If your agent is built on LangGraph, LangSmith can show you node-by-node state diffs across the graph — a depth of integration that's structurally hard for a framework-agnostic tool to match, because it requires knowing the internal shape of the orchestration layer.
Helicone's positioning narrowed rather than widened. Where some proxy-based tools have tried to grow into full observability suites, Helicone's current strength is explicitly cost and usage visibility with the simplest possible integration — a "put this in front of your API calls and see your spend" tool, not a multi-turn agent debugger. That's a legitimate niche, but it means evaluating it against the other three on "agent tracing depth" is comparing it on an axis it isn't optimized for.

Why developers should actually care

Cost. All four meter something, and the something differs, which makes head-to-head price comparison misleading unless you convert to your own volume. LangSmith and Braintrust both start metering above free tiers around the same per-thousand-unit price point, but LangSmith counts traces while Braintrust counts scores — so a workflow that runs five scorers per request costs five times more on Braintrust's meter than the trace count would suggest on LangSmith's. Langfuse counts "units" as traces + observations + scores combined, which front-loads cost for chatty multi-step agents that generate many observations per trace, but it's also the only one of the four where you can sidestep metered pricing entirely by self-hosting the open-source edition. Helicone's model is closer to per-request logging cost, which tends to scale more predictably with raw API call volume than with how deep your instrumentation goes.

Latency. Proxy architectures add a hop. For most applications the added latency is small and worth the simplicity, but if you're already latency-sensitive (voice agents, low-latency chat), routing every model call through a third party's infrastructure is a real tax that SDK-based instrumentation doesn't impose in the same way — SDK wrappers can log asynchronously without blocking the response path.

DX. Helicone wins on integration speed by a wide margin — often a base-URL swap. LangSmith wins on DX specifically if you're already in the LangChain/LangGraph ecosystem, because the tracing is nearly free once you're using the framework's callback system. Langfuse and Braintrust both require deliberate instrumentation work regardless of framework, which is more upfront cost but also more portable if you later change frameworks.

Lock-in. This is where the license architecture actually bites. Langfuse is the only one of the four where "we're leaving" doesn't mean "we're re-exporting years of trace data from someone else's database" — because you can have owned the database the whole time. That's a genuinely different risk profile for a team that expects to be running this for five years, versus a team that just wants the fastest path to a working eval loop this quarter.

Security and data residency. If you're in a regulated industry, LangSmith and Braintrust both offer VPC/self-hosted deployment at the enterprise tier, and Langfuse's Enterprise self-host tier adds the compliance surface (SCIM, audit logs, RBAC) on top of a base that's already self-hostable for free. Helicone's proxy architecture is the one place this gets structurally awkward: your model traffic is, by design, passing through a third party's infrastructure before it reaches the provider, which is a harder sell in a security review than an SDK that logs after the fact.

Maintainability. The eval-first vs. trace-first split shows up here too. Teams that adopt Braintrust's CI-gated eval workflow report that it forces evaluator rubrics to stay current, because a stale scorer either blocks merges it shouldn't or lets through ones it shouldn't — that pressure keeps the eval suite honest in a way a trace viewer you check manually after an incident doesn't.

Prompt management. This dimension gets less attention than tracing but ends up mattering just as much day to day. LangSmith and Langfuse both ship prompt versioning and rollback as first-class features — you can edit a prompt in a UI, tag a version, and roll back without a deploy, which matters once non-engineers (PMs, prompt engineers) are iterating on wording. Braintrust treats prompts as part of the experiment/dataset workflow rather than a standalone versioned artifact, which fits its eval-first framing but is a worse fit for a team that just wants a lightweight prompt CMS. Helicone's proxy position lets it do prompt experimentation at the request layer — routing a percentage of traffic to a variant — but it isn't a prompt-authoring tool in the way the other two are.

Practical use cases per option

LangSmith — you're building on LangGraph, want agent execution graphs debugged at the node level with minimal extra instrumentation, and you're fine with a hosted product for a team that's already deep in the LangChain ecosystem.
Langfuse — you need GDPR-grade data residency, want to avoid a growing usage-based bill as trace volume scales, or you simply don't trust a vendor with years of production trace data and want an exit that doesn't require an export ticket.
Braintrust — you're moving from "we eyeball outputs before a release" to "a bad score blocks the merge," and you want that gate to live next to your existing CI, not in a separate dashboard someone has to remember to check.
Helicone — you want cost and usage visibility across a fleet of API keys or teams with the least possible integration lift, and your debugging needs are closer to "which key spent what" than "why did this agent loop four times."

Concretely: a two-person team shipping a RAG chatbot on top of the OpenAI API, with no orchestration framework, gets the fastest path to "we can see what we're spending and where latency is going" from Helicone — a base-URL change on a Friday afternoon. A platform team supporting a dozen internal LangGraph agents, where the recurring support ticket is "the agent got stuck in a loop and nobody knows why," gets far more value from LangSmith's graph-level state diffs than from either of the other three, because the failure mode is specifically about control flow inside the graph, not about cost or eval scores. A team that just had a prompt regression ship to production undetected — the classic "someone tweaked the system prompt and support tickets tripled" incident — is the textbook Braintrust case: the fix isn't better logging, it's a scorer in CI that would have caught the regression before merge. And a compliance-constrained team (health data, EU customer data, a security review that flags "third-party sees your prompts") ends up at Langfuse almost by elimination, because it's the only one of the four where "keep the data in our own infrastructure" doesn't require an enterprise sales call.

What the marketing pages don't say

Every vendor's pricing page reads like the decision is obvious. It isn't, and here's where the gap between the pitch and the reality tends to show up:

Langfuse's "self-host for free, no limits" claim is true for the open-source edition — and materially less true the moment you want compliance features, at which point you're either paying for enterprise self-host (which still requires you to operate the ClickHouse layer) or moving to their managed cloud, where the Free→Core→Pro jump is mostly about retention windows and rate limits, not raw included volume. The infrastructure savings from self-hosting are real, but they're an ops-hours trade, not a free lunch.
Braintrust's eval-first pitch undersells how much scorer design work sits on your side of the line. The platform will happily run and track any scorer you give it, but a bad scorer produces a confidently wrong CI gate — the tool doesn't protect you from writing an eval that doesn't actually measure the thing you care about, and teams that skip the work of validating their scorers end up with a green checkmark that means less than it looks like.
LangSmith's deepest value is conditional on LangGraph, and that's understated in general positioning that talks about "any LLM app." Teams not using LangChain's orchestration layer get a competent but not distinctive tracer — most of what makes LangSmith's traces genuinely better than a generic tracer is the LangGraph-specific state-diff view, and that's tied to the framework.
Helicone's "add one line, get full observability" framing glosses over the ceiling. It's genuinely the fastest tool to integrate, but if your actual problem is "why did my agent take the wrong path on step six," a proxy watching individual API calls structurally cannot answer that as well as a tool tracing execution state, because the proxy never sees the parts of your program that aren't LLM calls — the branching logic, the tool outputs, the state passed between steps.

None of this is dishonest marketing — it's the normal gap between "what the product does at its best" and "what it does for your specific workload," and it's exactly the gap a feature-matrix comparison won't show you.

Comparison table

Dimension	LangSmith	Langfuse	Braintrust	Helicone
Primary unit of analysis	Trace / run	Trace + observation + score	Eval score	API request
Architecture	SDK / callback instrumentation	SDK instrumentation	SDK instrumentation	Proxy (+ async logging option)
Open source	No (closed, VPC option at enterprise)	Yes — MIT core, source-available `ee` add-ons	No (closed, BYOC at enterprise only)	Partially, varies by component
Free self-hosting with full features	No	Yes	No	Limited
Deepest strength	LangGraph node-level agent tracing	Framework-agnostic self-hosted observability + evals	CI/CD-gated eval workflows	Fastest integration, cost/usage visibility
Structural weak point	Value drops outside LangChain/LangGraph	Enterprise self-host still requires ops-heavy ClickHouse	Scorer quality is entirely on you	Shallow on multi-step agent state, adds proxy latency
Entry pricing signal	Free tier ~5k traces/mo; paid tier ~$99/mo	Free cloud tier ~50k units/mo; self-host free; paid cloud from ~$29/mo	Free tier ~10k scores/mo; paid tier ~$249/mo	Usage-based, generous free tier
Best for	LangGraph-native agent teams	Data-residency-sensitive or cost-scaling-averse teams	Teams moving evals into CI as a merge gate	Teams that mainly need spend/usage visibility fast

(Pricing figures reflect publicly listed tiers as of August 2026 and change frequently — verify current numbers on each vendor's pricing page before budgeting.)

An independent read

If you strip away the positioning, the honest takeaway is that these four products aren't really competing for the same budget line as often as the "LLM observability tools compared" headline implies. A team that adopts Braintrust for CI-gated evals will often also want Langfuse or LangSmith for exploratory trace debugging during incidents, because eval scores tell you something broke but not always why. A team using Helicone for cost visibility across API keys will frequently layer a trace-first tool on top once their agents get complex enough that "which call cost what" stops being the interesting question. The real overlap — and the place an actual either/or decision exists — is between LangSmith and Langfuse, because they occupy the same trace-first niche and genuinely are substitutes for each other, differentiated mainly by framework affinity and hosting philosophy rather than by category.

The self-hosting story deserves one more caveat: "free and open source" is doing a lot of work in Langfuse's pitch, and it's earned — the MIT core is real — but free software still costs engineering time to operate well. A three-person startup evaluating "$199/mo Langfuse Pro" against "free self-hosted Langfuse plus someone's Tuesday afternoons managing ClickHouse" should actually run that math, not assume free means cheaper.

Which reader profile should pick which option

You're building agents on LangGraph and want the deepest possible visibility into graph execution with the least instrumentation effort → LangSmith. Don't fight the framework affinity; it's the whole value proposition.
You need to self-host for compliance, you're wary of usage-based pricing at scale, or you want a tool that works the same whether you're on LangChain, LlamaIndex, or raw API calls → Langfuse. Budget the ops time for self-hosting honestly, or start on their cloud free tier and re-evaluate once volume is real.
Your team already treats broken outputs as bugs and wants that enforced by CI rather than caught in production → Braintrust, but only if you're willing to invest real effort in scorer design — the tool won't do that part for you.
You need cost and usage visibility across many API keys or teams, fast, and your debugging problems are shallow enough that a proxy's view of individual requests is enough → Helicone. Don't reach for it as your only tool once your agents get genuinely multi-step; you'll outgrow its depth before you outgrow its price.

None of these is a universal winner, and any comparison that hands you one is answering a question you didn't ask.

Discussion: for teams running agentic workflows with five-plus LLM calls per user request, at what point does the added network hop from a proxy-based tool like Helicone actually show up in your p99 latency budget — have you measured it, or is it an assumption you're carrying from a single-call mental model?

Sources:

DEV Community