Braintrust vs LangSmith comes down to your framework and how you ship. Choose LangSmith if you build on LangChain or LangGraph, have a small team that suits per-seat pricing, or want operational alerts on latency, cost and errors next to your evals. Choose Braintrust if your releases run on datasets, scorers and experiments, you don't use LangChain, or you want unlimited users billed on data and scores. Both trace agents, accept OpenTelemetry, run online LLM judges on production traffic, and offer self-hosting only on Enterprise.
Both are full eval and observability platforms, so a feature checklist won't separate them. Billing, framework fit and the shape of the eval workflow will. Every fact below comes from Braintrust's and LangSmith's own documentation and pricing pages, checked on November 17, 2026. It is a documentation comparison, not a hands-on benchmark.
Braintrust vs LangSmith at a glance
| Braintrust | LangSmith | |
|---|---|---|
| Built by | Braintrust | LangChain |
| Center of gravity | Evals: datasets, scorers, playgrounds, experiments | Tracing and evals for LangChain and LangGraph agents |
| Offline evals | Experiments as "the immutable, comparable record of your eval runs" | Datasets, experiments, human, code, LLM-judge and pairwise evaluators |
| CI | Documented: "Run evals on every pull request" | pytest plugin, plus Vitest/Jest for JS/TS |
| Online scoring units | Span, trace, group of traces | Run, thread |
| Wait before judging a conversation | 30 s idle by default | 10 min idle by default, 2 min minimum |
| Online evaluator types | LLM judge, code, autoevals (span only) | LLM judge, decision model, code |
| Alerts | Log, time-window and environment alerts to Slack, webhook, PagerDuty | Run count, cost, errors, feedback score, latency to Slack, PagerDuty, Dynatrace, webhook |
| Production clustering | Topics (Task, Sentiment, Issues facets) | Insights (Plus and Enterprise) |
| AI assistant | Loop investigates logs, experiments and datasets | Engine finds issues and can open a PR |
| OpenTelemetry | OTLP traces and logs | OTLP traces |
| Pricing basis | Processed GB plus scores; unlimited users | Seats plus traces |
| Free tier | 1 GB, 10,000 scores, 14-day retention | 1 seat, 5,000 base traces |
| Paid entry | Pro $249/mo: 5 GB, 50,000 scores | Plus $39 per seat/mo: 10,000 base traces |
| Self-hosting | Enterprise: on-prem or hosted | Enterprise add-on (Kubernetes) |
Evals
Braintrust treats evaluation as the main workflow. Every eval has three parts: data, a task and evaluators. You iterate in browser playgrounds, where "re-running a playground overwrites previous generations," then promote a configuration to an experiment, which Braintrust calls "the immutable, comparable record of your eval runs." Scorers can be built-in autoevals, LLM judges or custom code. Its docs recommend running evals on every pull request to catch regressions before production.
LangSmith has the same building blocks: datasets built by hand, from production traces or synthetically; human, code, LLM-judge and pairwise evaluators; and experiments you compare for benchmarking, regression tests or backtesting. Its CI story is concrete: a pytest plugin "lets Python developers define their datasets and evaluations as pytest test cases," with a Vitest/Jest equivalent for JavaScript and TypeScript.
Verdict. Braintrust is the more experiment-centric product, built for teams whose release gate is "did the scores go up." LangSmith is equally capable offline and fits more naturally into a test suite.
Tracing
Both trace multi-step agents and accept OpenTelemetry. Braintrust ingests OTLP traces and logs at api.braintrust.dev/otel. LangSmith accepts traces "from any OpenTelemetry-compatible agent" at its /otel endpoint. Retention differs. Braintrust Starter keeps data for 14 days and Pro for 30, with more available per gigabyte. LangSmith keeps base traces for 14 days and extended traces for 180.
Online scoring
Braintrust online scoring rules run at three scopes. Span scope judges a single LLM response or tool call. Trace scope judges a full trace once it has been idle for 30 seconds by default. Group scope judges related traces linked by a key such as metadata.session_id. Sampling is a percentage, filters are SQL, and the rule form estimates weekly scorer runs before you save. Each score is billed.
LangSmith online evaluators run on individual runs, with a sampling rate and filters on metadata, tool calls, feedback and other evaluators' scores. Separate multi-turn evaluators judge whole threads after 10 minutes idle by default, capped at 10 per workspace. Running an evaluator on a trace moves it to extended retention, and a weekly spend limit can pause evaluators.
Verdict. Both can judge a whole conversation. Braintrust's 30-second default suits short tasks, while LangSmith's 10-minute idle default suits chat, where users pause between turns. Price both on the share of traffic you actually want judged.
Pricing
| Braintrust | LangSmith | |
|---|---|---|
| Free | Starter: 1 GB, 10,000 scores, unlimited users | Developer: 1 seat, 5,000 base traces |
| Paid | Pro $249/mo: 5 GB, 50,000 scores, unlimited users | Plus $39 per seat/mo: 10,000 base traces, unlimited seats |
| Overage | $3 per GB and $1.50 per 1,000 scores on Pro | Pay as you go; extended traces cost more |
| Extras | Model credits ($10 Starter, $100 Pro) | LSUs at $1 for agent features such as Engine |
| Enterprise | Custom | Custom |
LangSmith is cheaper for a few engineers at moderate volume: three seats on Plus cost $117 a month. Braintrust's flat $249 becomes the better deal as more people need access, as long as data and score volume stay near the allowance. Agent traces with long prompts and tool outputs consume gigabytes quickly, so measure a real week of traffic before you choose. The AI agent monitoring pricing guide lists the fees to ask about.
Self-hosting
Neither is open source, and neither offers a free self-hosted edition. LangSmith self-hosting is "an add-on to the Enterprise plan," deployed on Kubernetes with Helm charts or Terraform. Braintrust Enterprise offers on-prem or hosted deployment. Its self-hosted model splits a data plane in your cloud, which stores your AI data, from a control plane hosted by Braintrust, which handles the UI, auth and metadata. If you need free self-hosting, see the "if neither fits" section below.
Framework coupling
LangSmith is LangChain's product, and LangChain and LangGraph are where it fits most naturally. LangSmith says it "works with many frameworks and providers," lists OpenAI, Anthropic, CrewAI, the Vercel AI SDK and Pydantic AI, and accepts OpenTelemetry. You don't need LangChain to use it.
Braintrust has no framework of its own, so no stack gets favored treatment. It traces through its SDKs and OpenTelemetry.
Verdict. On LangGraph, LangSmith is the lower-friction choice. Anywhere else, the two are roughly even on integration, and the decision moves to eval workflow and pricing.
Which to choose
| If your team… | Choose |
|---|---|
| Builds on LangChain or LangGraph | LangSmith |
| Gates releases on experiment scores and scored datasets | Braintrust |
| Wants evals inside pytest or Vitest | LangSmith |
| Has many people who need access and moderate volume | Braintrust |
| Is two or three engineers watching costs | LangSmith |
| Wants alerts on latency, cost and errors as well as scores | LangSmith |
| Wants production logs clustered by task, sentiment and issues | Braintrust (Topics) or LangSmith Insights; both work |
| Needs free self-hosting | Neither; see below |
If neither fits
Langfuse is the open-source answer. It is MIT-licensed outside its ee folders, free to self-host, and covers tracing, prompt management, datasets, experiments and evaluators. Cloud Core is $29 a month for 100,000 units. Its evaluators run on observations, and Langfuse says it "does not inherently know when a session has concluded," so judging a conversation takes extra work.
Arize comes in two forms. Phoenix is free to self-host under the Elastic License 2.0. Arize AX is the managed platform, with online evals at span, trace or session level, monitors, and Signal for issue discovery, from $50 a month for 50,000 spans on Pro. Arize alternatives explains the split.
Agnost AI is the option when the failures that cost you users only show up across a whole conversation: a wrong policy stated on turn six, or a refund promised but never processed. We make Agnost AI. It ingests OpenTelemetry and has its own SDKs, so it can run next to Braintrust or LangSmith on the same spans. You write rules in plain English. After a conversation has been idle for 15 minutes, an LLM reads the whole conversation in a batch and marks each assistant message pass or fail; verdicts are not scores. Every user message is labelled with an intent and a sentiment. There is no sampling setting, so every conversation is judged up to your plan limit. Alerts go to Slack or email, and a fired alert can trigger Auto Improve, which opens a pull request for human review. It is not real time and has no CI integration or offline experiments; Auto Improve tests a fix against past runs before a person reviews and merges it. That's why teams usually pair it with Braintrust or LangSmith rather than replace them. Plans run from free (1,000 events a month) to $49 (10,000 events) and $499 (1,000,000 events).
Related reading
- Braintrust alternatives for AI agent evaluation
- LangSmith vs Langfuse vs Braintrust vs Phoenix vs Agnost
- Best online evaluation tools for AI agents
- Conversation-level vs trace-level online evals
- LangSmith alternatives for agent evals and monitoring (2026)
Frequently asked questions
Braintrust vs LangSmith: which is better?
Neither wins outright. Choose LangSmith if you build on LangChain or LangGraph, want per-seat pricing with a free single-seat tier, or want alerts on latency, cost and errors next to your evals. Choose Braintrust if your releases run on datasets, scorers and experiments, your stack is not LangChain, or you want unlimited users billed on data and scores instead of seats.
Is Braintrust cheaper than LangSmith?
It depends on team size and volume. LangSmith Plus is $39 per seat a month with 10,000 base traces, so cost grows with headcount. Braintrust Pro is $249 a month for unlimited users with 5 GB of processed data and 50,000 scores, then $3 per GB and $1.50 per 1,000 scores. Small teams usually pay less on LangSmith; larger teams with moderate volume can pay less on Braintrust.
Does LangSmith work without LangChain?
Yes. LangSmith says it works with many frameworks and providers, including OpenAI, Anthropic, CrewAI, the Vercel AI SDK and Pydantic AI, and it accepts OpenTelemetry traces from any OpenTelemetry-compatible agent. Its closest integration is still with LangChain and LangGraph.
Can I self-host Braintrust or LangSmith?
Only on Enterprise. LangSmith self-hosting is an add-on to the Enterprise plan, deployed on Kubernetes with Helm or Terraform. Braintrust Enterprise offers on-prem or hosted deployment, with a data plane in your cloud and a control plane hosted by Braintrust. For free self-hosting, look at Langfuse or Arize Phoenix.
Which is better for online evals, Braintrust or LangSmith?
Both judge production traffic with LLM judges and code, with sampling and filters. LangSmith scores runs and whole threads, waiting 10 minutes of idle time by default before judging a thread. Braintrust scores spans, traces and groups of traces, waiting 30 seconds by default. Braintrust bills each score; LangSmith moves evaluated traces to extended retention, which costs more.
What should I use if neither Braintrust nor LangSmith fits?
Langfuse if you need open source and free self-hosting, Arize if you want AX's managed session-level evals or Phoenix's self-hosted tracing, and Agnost AI if the failures that matter only show up across a whole production conversation and you want them turned into reviewed pull requests.
Originally published on agnost.ai.
Top comments (0)