DEV Community

Shubham Palriwala
Shubham Palriwala

Posted on Originally published at agnost.ai

Braintrust vs LangSmith: which to choose for agent evals (2026)

Braintrust vs LangSmith comes down to your framework and how you ship. Choose LangSmith if you build on LangChain or LangGraph, have a small team that suits per-seat pricing, or want operational alerts on latency, cost and errors next to your evals. Choose Braintrust if your releases run on datasets, scorers and experiments, you don't use LangChain, or you want unlimited users billed on data and scores. Both trace agents, accept OpenTelemetry, run online LLM judges on production traffic, and offer self-hosting only on Enterprise.

Both are full eval and observability platforms, so a feature checklist won't separate them. Billing, framework fit and the shape of the eval workflow will. Every fact below comes from Braintrust's and LangSmith's own documentation and pricing pages, checked on November 17, 2026. It is a documentation comparison, not a hands-on benchmark.

Braintrust vs LangSmith at a glance

Braintrust LangSmith
Built by Braintrust LangChain
Center of gravity Evals: datasets, scorers, playgrounds, experiments Tracing and evals for LangChain and LangGraph agents
Offline evals Experiments as "the immutable, comparable record of your eval runs" Datasets, experiments, human, code, LLM-judge and pairwise evaluators
CI Documented: "Run evals on every pull request" pytest plugin, plus Vitest/Jest for JS/TS
Online scoring units Span, trace, group of traces Run, thread
Wait before judging a conversation 30 s idle by default 10 min idle by default, 2 min minimum
Online evaluator types LLM judge, code, autoevals (span only) LLM judge, decision model, code
Alerts Log, time-window and environment alerts to Slack, webhook, PagerDuty Run count, cost, errors, feedback score, latency to Slack, PagerDuty, Dynatrace, webhook
Production clustering Topics (Task, Sentiment, Issues facets) Insights (Plus and Enterprise)
AI assistant Loop investigates logs, experiments and datasets Engine finds issues and can open a PR
OpenTelemetry OTLP traces and logs OTLP traces
Pricing basis Processed GB plus scores; unlimited users Seats plus traces
Free tier 1 GB, 10,000 scores, 14-day retention 1 seat, 5,000 base traces
Paid entry Pro $249/mo: 5 GB, 50,000 scores Plus $39 per seat/mo: 10,000 base traces
Self-hosting Enterprise: on-prem or hosted Enterprise add-on (Kubernetes)

Evals

Braintrust treats evaluation as the main workflow. Every eval has three parts: data, a task and evaluators. You iterate in browser playgrounds, where "re-running a playground overwrites previous generations," then promote a configuration to an experiment, which Braintrust calls "the immutable, comparable record of your eval runs." Scorers can be built-in autoevals, LLM judges or custom code. Its docs recommend running evals on every pull request to catch regressions before production.

LangSmith has the same building blocks: datasets built by hand, from production traces or synthetically; human, code, LLM-judge and pairwise evaluators; and experiments you compare for benchmarking, regression tests or backtesting. Its CI story is concrete: a pytest plugin "lets Python developers define their datasets and evaluations as pytest test cases," with a Vitest/Jest equivalent for JavaScript and TypeScript.

Verdict. Braintrust is the more experiment-centric product, built for teams whose release gate is "did the scores go up." LangSmith is equally capable offline and fits more naturally into a test suite.

Tracing

Both trace multi-step agents and accept OpenTelemetry. Braintrust ingests OTLP traces and logs at api.braintrust.dev/otel. LangSmith accepts traces "from any OpenTelemetry-compatible agent" at its /otel endpoint. Retention differs. Braintrust Starter keeps data for 14 days and Pro for 30, with more available per gigabyte. LangSmith keeps base traces for 14 days and extended traces for 180.

Online scoring

Braintrust online scoring rules run at three scopes. Span scope judges a single LLM response or tool call. Trace scope judges a full trace once it has been idle for 30 seconds by default. Group scope judges related traces linked by a key such as metadata.session_id. Sampling is a percentage, filters are SQL, and the rule form estimates weekly scorer runs before you save. Each score is billed.

LangSmith online evaluators run on individual runs, with a sampling rate and filters on metadata, tool calls, feedback and other evaluators' scores. Separate multi-turn evaluators judge whole threads after 10 minutes idle by default, capped at 10 per workspace. Running an evaluator on a trace moves it to extended retention, and a weekly spend limit can pause evaluators.

Verdict. Both can judge a whole conversation. Braintrust's 30-second default suits short tasks, while LangSmith's 10-minute idle default suits chat, where users pause between turns. Price both on the share of traffic you actually want judged.

Pricing

Braintrust LangSmith
Free Starter: 1 GB, 10,000 scores, unlimited users Developer: 1 seat, 5,000 base traces
Paid Pro $249/mo: 5 GB, 50,000 scores, unlimited users Plus $39 per seat/mo: 10,000 base traces, unlimited seats
Overage $3 per GB and $1.50 per 1,000 scores on Pro Pay as you go; extended traces cost more
Extras Model credits ($10 Starter, $100 Pro) LSUs at $1 for agent features such as Engine
Enterprise Custom Custom

LangSmith is cheaper for a few engineers at moderate volume: three seats on Plus cost $117 a month. Braintrust's flat $249 becomes the better deal as more people need access, as long as data and score volume stay near the allowance. Agent traces with long prompts and tool outputs consume gigabytes quickly, so measure a real week of traffic before you choose. The AI agent monitoring pricing guide lists the fees to ask about.

Self-hosting

Neither is open source, and neither offers a free self-hosted edition. LangSmith self-hosting is "an add-on to the Enterprise plan," deployed on Kubernetes with Helm charts or Terraform. Braintrust Enterprise offers on-prem or hosted deployment. Its self-hosted model splits a data plane in your cloud, which stores your AI data, from a control plane hosted by Braintrust, which handles the UI, auth and metadata. If you need free self-hosting, see the "if neither fits" section below.

Framework coupling

LangSmith is LangChain's product, and LangChain and LangGraph are where it fits most naturally. LangSmith says it "works with many frameworks and providers," lists OpenAI, Anthropic, CrewAI, the Vercel AI SDK and Pydantic AI, and accepts OpenTelemetry. You don't need LangChain to use it.

Braintrust has no framework of its own, so no stack gets favored treatment. It traces through its SDKs and OpenTelemetry.

Verdict. On LangGraph, LangSmith is the lower-friction choice. Anywhere else, the two are roughly even on integration, and the decision moves to eval workflow and pricing.

Which to choose

If your team… Choose
Builds on LangChain or LangGraph LangSmith
Gates releases on experiment scores and scored datasets Braintrust
Wants evals inside pytest or Vitest LangSmith
Has many people who need access and moderate volume Braintrust
Is two or three engineers watching costs LangSmith
Wants alerts on latency, cost and errors as well as scores LangSmith
Wants production logs clustered by task, sentiment and issues Braintrust (Topics) or LangSmith Insights; both work
Needs free self-hosting Neither; see below

If neither fits

Langfuse is the open-source answer. It is MIT-licensed outside its ee folders, free to self-host, and covers tracing, prompt management, datasets, experiments and evaluators. Cloud Core is $29 a month for 100,000 units. Its evaluators run on observations, and Langfuse says it "does not inherently know when a session has concluded," so judging a conversation takes extra work.

Arize comes in two forms. Phoenix is free to self-host under the Elastic License 2.0. Arize AX is the managed platform, with online evals at span, trace or session level, monitors, and Signal for issue discovery, from $50 a month for 50,000 spans on Pro. Arize alternatives explains the split.

Agnost AI is the option when the failures that cost you users only show up across a whole conversation: a wrong policy stated on turn six, or a refund promised but never processed. We make Agnost AI. It ingests OpenTelemetry and has its own SDKs, so it can run next to Braintrust or LangSmith on the same spans. You write rules in plain English. After a conversation has been idle for 15 minutes, an LLM reads the whole conversation in a batch and marks each assistant message pass or fail; verdicts are not scores. Every user message is labelled with an intent and a sentiment. There is no sampling setting, so every conversation is judged up to your plan limit. Alerts go to Slack or email, and a fired alert can trigger Auto Improve, which opens a pull request for human review. It is not real time and has no CI integration or offline experiments; Auto Improve tests a fix against past runs before a person reviews and merges it. That's why teams usually pair it with Braintrust or LangSmith rather than replace them. Plans run from free (1,000 events a month) to $49 (10,000 events) and $499 (1,000,000 events).

Related reading

Frequently asked questions

Braintrust vs LangSmith: which is better?

Neither wins outright. Choose LangSmith if you build on LangChain or LangGraph, want per-seat pricing with a free single-seat tier, or want alerts on latency, cost and errors next to your evals. Choose Braintrust if your releases run on datasets, scorers and experiments, your stack is not LangChain, or you want unlimited users billed on data and scores instead of seats.

Is Braintrust cheaper than LangSmith?

It depends on team size and volume. LangSmith Plus is $39 per seat a month with 10,000 base traces, so cost grows with headcount. Braintrust Pro is $249 a month for unlimited users with 5 GB of processed data and 50,000 scores, then $3 per GB and $1.50 per 1,000 scores. Small teams usually pay less on LangSmith; larger teams with moderate volume can pay less on Braintrust.

Does LangSmith work without LangChain?

Yes. LangSmith says it works with many frameworks and providers, including OpenAI, Anthropic, CrewAI, the Vercel AI SDK and Pydantic AI, and it accepts OpenTelemetry traces from any OpenTelemetry-compatible agent. Its closest integration is still with LangChain and LangGraph.

Can I self-host Braintrust or LangSmith?

Only on Enterprise. LangSmith self-hosting is an add-on to the Enterprise plan, deployed on Kubernetes with Helm or Terraform. Braintrust Enterprise offers on-prem or hosted deployment, with a data plane in your cloud and a control plane hosted by Braintrust. For free self-hosting, look at Langfuse or Arize Phoenix.

Which is better for online evals, Braintrust or LangSmith?

Both judge production traffic with LLM judges and code, with sampling and filters. LangSmith scores runs and whole threads, waiting 10 minutes of idle time by default before judging a thread. Braintrust scores spans, traces and groups of traces, waiting 30 seconds by default. Braintrust bills each score; LangSmith moves evaluated traces to extended retention, which costs more.

What should I use if neither Braintrust nor LangSmith fits?

Langfuse if you need open source and free self-hosting, Arize if you want AX's managed session-level evals or Phoenix's self-hosted tracing, and Agnost AI if the failures that matter only show up across a whole production conversation and you want them turned into reviewed pull requests.


Originally published on agnost.ai.

Top comments (0)