ke yi

Posted on Nov 13 Originally published at fp8.co

Braintrust vs Langfuse vs Promptfoo vs DeepEval: LLM Evals

#aiengineering #testing

Braintrust vs Langfuse vs Promptfoo vs DeepEval: Which LLM Evaluation Platform Should You Use?

TL;DR: Braintrust is a managed eval-and-experiment platform with a hybrid data-plane option on Enterprise. Langfuse is an MIT-licensed observability platform with evaluations that you can self-host for free. Promptfoo is an MIT-licensed CLI for eval matrices and red teaming that joined OpenAI in March 2026. DeepEval is an Apache 2.0 pytest-style metric library whose hosted companion is Confident AI.

Key Takeaways

These four tools sit at two layers. Braintrust and Langfuse are platforms that store experiments and traces. Promptfoo and DeepEval are frameworks that run tests and write results somewhere else.
Braintrust vs Langfuse comes down to who runs the backend. Braintrust is commercial: the free Starter plan has 14-day retention, Pro is $249/month, and the hybrid data plane is Enterprise-only. Langfuse is free to self-host, and its cloud starts at $29/month on Core.
Promptfoo is the only one of the four whose main job is adversarial testing. Its red-team plugins cover failure modes such as harmful content and broken object- and function-level authorization, and its docs reference the OWASP LLM Top 10 and NIST AI RMF.
DeepEval is the open-source part of Confident AI. You can run its metrics (G-Eval, faithfulness, contextual recall, tool correctness and more) with deepeval test run and never log in, so the library is the Confident AI open-source alternative.
If you are looking for a LangSmith or Opik alternative for evals, pick the layer first. Choose Langfuse or Opik if you want a self-hosted platform, Braintrust if you want a managed experiment UI, and Promptfoo or DeepEval if you only need a CI gate.
Many teams combine one platform with one framework. That works only if dataset IDs, rubric versions, and scores stay traceable between them.

Why is "which LLM eval tool" the wrong first question?

Picture a team whose RAG assistant regressed after a prompt tweak. Someone asks "should we buy Braintrust or use Langfuse?" Nobody has written down what a regression is: which inputs, which expected behavior, which threshold. The tools all speak the same vocabulary — datasets, scorers, experiments, LLM-as-judge — so they look interchangeable on a feature grid. They are not, because they disagree on where results live and who operates the system that holds them.

There are three jobs an evaluation stack has to do:

Run a fixed set of cases against a candidate prompt, model, or agent and score them.
Store and compare those results across runs so a human can see what changed and why.
Feed back production failures into the fixed set so the next run catches them.

Promptfoo and DeepEval are strongest at job 1 and can do it with nothing but a terminal. Braintrust and Langfuse are strongest at job 2 and connect to production traces for job 3. Deciding which jobs you need covered — and which you already cover elsewhere — narrows the choice faster than any feature table.

If you are choosing an observability backend first, our LangSmith vs Langfuse vs Phoenix comparison covers tracing in depth. For a two-way look at the production-vs-CI split, see Langfuse vs Promptfoo.

What are Braintrust, Langfuse, Promptfoo, and DeepEval?

Two of these are platforms that store and compare results (Braintrust, Langfuse), and two are frameworks that run tests and hand results elsewhere (Promptfoo, DeepEval). Here is what each one actually is.

Braintrust

Braintrust is a commercial evaluation and observability platform built around experiments: each run of an Eval() over a dataset becomes a comparable experiment in the web UI, with per-case diffs against previous runs. Its open-source AutoEvals library (MIT, Python and TypeScript) supplies scorers such as Factuality, Closed QA, faithfulness, context precision/recall, JSON validity, embedding similarity, and Levenshtein distance. The platform itself is not open source. For data residency, Braintrust documents a hybrid deployment where the data plane (API, PostgreSQL, Redis, object storage and its Brainstore engine) runs in your AWS, GCP, or Azure account via Terraform, while Braintrust's control plane serves the UI, authentication, and metadata only.

Langfuse

Langfuse is an open-source LLM engineering platform whose repository is MIT-licensed except for its ee folders. It combines OpenTelemetry-based tracing, prompt management, datasets and experiments, LLM-as-judge and code evaluators, user feedback, and manual labeling. It self-hosts via Docker Compose for a single machine or Helm on Kubernetes for production, with Terraform templates for AWS, Azure, and GCP. Evaluation in Langfuse is attached to traces: a score can sit on a production trace, on a dataset run, or on a human annotation queue item.

Promptfoo

Promptfoo is an MIT-licensed CLI and library for evaluating and red-teaming LLM applications. You describe prompts, providers, test cases, and assertions in YAML, run promptfoo eval, and review a side-by-side matrix in promptfoo view. Its red-team mode generates adversarial inputs, runs them through your application, and reports failures, with plugins targeting harmful content, broken object-level and function-level authorization, competitor endorsement, and more. On March 9, 2026, Promptfoo announced it was joining OpenAI; the post states that Promptfoo will remain open source and continue to support a diverse range of providers and models.

DeepEval

DeepEval is an Apache 2.0 Python framework that treats LLM evaluation like unit testing. You build LLMTestCase objects, assert them against metrics with assert_test, and run deepeval test run test_file.py — or call evaluate() from a notebook without pytest. Its metric catalogue spans custom criteria (G-Eval), RAG (faithfulness, answer relevancy, contextual precision/recall/relevancy), hallucination, agent metrics (task completion, tool correctness), multi-turn conversation, and MCP. Running deepeval login sends test runs and traces to Confident AI, the vendor's commercial platform; without it, DeepEval runs entirely locally.

How do the four compare at a glance?

Dimension	Braintrust	Langfuse	Promptfoo	DeepEval
Layer	Managed platform	Platform (cloud or self-hosted)	CLI framework	Python framework
License	Proprietary platform; AutoEvals MIT	MIT core (except `ee`)	MIT	Apache 2.0
Primary interface	`Eval()` SDK + web UI	SDK + web UI	YAML config + CLI	pytest-style Python
Languages	Python, TypeScript	Python, JS/TS SDKs, OpenTelemetry	Language-agnostic (YAML; JS/Python hooks)	Python
Result storage	Braintrust (or your data plane)	Langfuse database	Local files; optional sharing	Local; optional Confident AI
Production tracing	Yes	Yes (core strength)	No	Via Confident AI
Red teaming	Not a focus	Not a focus	Core strength	Not a core metric focus
Self-hosting	Hybrid data plane on Enterprise	Free, full platform	Nothing to host	Nothing to host (library)
Entry price	Starter $0	Hobby $0 / self-host $0	Free	Free

Pricing and licensing were read from each vendor's own pages on November 13, 2026, and they change. Check them again before you budget.

Braintrust vs Langfuse: which eval platform should you choose?

This is the head-to-head most people mean when they search for Langfuse alternatives, and the answer depends on three things.

Operating model. Langfuse gives you the whole platform to run yourself at no license cost, and the cloud runs the same code. Braintrust's control plane is always run by Braintrust. Its hybrid option moves only your sensitive data (experiments, logs, traces, datasets, completions) into your own cloud. If your security review requires that no vendor-operated component touches the deployment, Langfuse self-hosted is the only one of the two that passes.

Workflow centre of gravity. Braintrust is built around the experiment: you change a prompt, rerun the eval, and the UI shows which cases improved or regressed against the baseline. Langfuse is built around the trace: you instrument production, attach scores to traces, and promote interesting traces into datasets. Both platforms can do both. What differs is which workflow takes fewer steps.

Pricing shape. These figures come from the vendors' pages on November 13, 2026:

	Braintrust	Langfuse Cloud
Free tier	Starter: 1 GB processed data, 10K scores/month, 14-day retention, unlimited seats	Hobby: 50k units/month, 2 users, 30-day data access
First paid tier	Pro $249/month: 5 GB, 50K scores, 30-day retention (extendable to 180 days at $0.50/GB/month)	Core $29/month: 100k units, unlimited users, 90-day data access
Overage	$3/GB data, $1.50 per 1,000 scores (Pro)	$8 per 100k units
Next tier	Enterprise (custom; hybrid/on-prem)	Pro $199/month (3-year data access); Enterprise $2,499/month

Braintrust meters processed gigabytes and scores. Langfuse meters units, where a unit is a trace, an observation, or a score. So a workload with large payloads but few spans, and one with small payloads but deeply nested agent spans, can rank the two vendors in opposite orders. Price one week of your real traffic on both calculators rather than comparing headline numbers.

Promptfoo vs DeepEval: which eval framework belongs in CI?

Both run in CI and both fail a build on a threshold, but they suit different teams.

Promptfoo fits polyglot teams and security work. Its YAML config does not care whether the app is written in TypeScript, Go, or Python, and it can call an HTTP endpoint as the target. The model-comparison matrix is the fastest way to answer "which of these three prompts holds up on these 40 cases." Red teaming is the deciding feature. If you need evidence that your agent resists prompt injection or will not call tools outside the user's authorization scope, Promptfoo generates those attacks for you.

DeepEval fits Python teams that already think in pytest. Test cases live next to application code, metrics are Python objects you can subclass, and deepeval test run slots into an existing pytest job. Its research-backed RAG and agent metrics (faithfulness, contextual recall, tool correctness, task completion) give you named, documented scores rather than a hand-written judge prompt. That helps when you have to explain to a reviewer what "0.82 faithful" actually measured.

A minimal DeepEval gate looks like this:

# test_support_agent.py
from deepeval import assert_test
from deepeval.test_case import LLMTestCase
from deepeval.metrics import FaithfulnessMetric

def test_refund_policy_answer():
    case = LLMTestCase(
        input="Can I get a refund after 45 days?",
        actual_output=support_agent("Can I get a refund after 45 days?"),
        retrieval_context=["Refunds are available within 30 days of purchase."],
    )
    assert_test(case, [FaithfulnessMetric(threshold=0.8)])

deepeval test run test_support_agent.py

The equivalent Promptfoo check is a YAML test with an llm-rubric or a similarity assertion, run with promptfoo eval. Neither framework makes the judge model free: every LLM-as-judge metric is a model call with its own cost and latency, so a 500-case suite with four judged metrics is 2,000 extra calls per run.

What does a Braintrust eval look like in code?

Braintrust's SDK puts the data, the task under test, and the scorers in one call. This is the shape of its documented Python quickstart, trimmed to two cases:

from braintrust import Eval
from autoevals import ExactMatch

def task(input):
    return identify_movie(input)  # your model or agent call

Eval(
    "Evaluation quickstart",
    experiment_name="Movie matcher (Python)",
    data=[
        {"input": "A detective investigates murders based on the seven deadly sins.", "expected": "Se7en"},
        {"input": "A thief plants an idea into a CEO's mind through shared dreams.", "expected": "Inception"},
    ],
    task=task,
    scores=[ExactMatch],
)

bt eval movie_matcher.py

Each run lands as an experiment you can diff against the last one. Langfuse covers similar ground with datasets and experiment runs in its SDK. The practical difference is where the comparison view lives and who hosts it.

What are the best LangSmith alternatives for LLM evals?

If you are leaving LangSmith specifically for evaluation, the question is usually one of seat pricing, LangChain coupling, or self-hosting being gated to Enterprise (see our LangSmith hosted vs self-hosted breakdown). Map the reason to the replacement:

"We need to self-host without an Enterprise contract." Use Langfuse (MIT core, free to self-host) or Opik (Apache 2.0, free to self-host the full platform).
"We want a managed experiment UI and per-case diffs, not a trace explorer." Use Braintrust.
"We only used LangSmith to gate releases." You may not need a platform at all. Promptfoo or DeepEval in CI, with results archived as build artifacts, covers a release gate.
"We need adversarial and security testing." Use Promptfoo, alongside whatever platform you keep.

What are the best Opik alternatives, and where does Confident AI fit?

Opik, from Comet, is the closest open-source peer to Langfuse in this space. It is Apache 2.0 and free to self-host via a script or a Helm chart, and it combines tracing, datasets, LLM-as-judge metrics, online evaluation rules, guardrails, an agent optimizer SDK, and a pytest integration. Teams searching for Opik alternatives usually want one of three things. Choose Langfuse for a larger self-hosted ecosystem and prompt management. Choose Braintrust if you would rather not operate anything. Choose DeepEval if the platform is more than you need and the metrics are what you want.

Confident AI is the hosted platform behind DeepEval. Its pricing page lists Free (2 seats, 1 project, 1 GB-month of trace spans), Starter at $200/month and Team at $2,000/month, both per organization with unlimited seats. Dedicated on-prem deployment is listed only on Enterprise. The open-source alternative to Confident AI is mostly DeepEval itself. The metrics, test runner, and local results are Apache 2.0 and need no account. If you also want a self-hosted UI for history and production traces, DeepEval does not provide one, so you would pair it with Langfuse or Opik and push scores there.

How should you combine an eval platform and an eval framework?

The common production pattern is one framework for the pre-merge gate and one platform for history and production. A few rules keep the pair from drifting apart:

Concern	What to standardise
Dataset identity	Version the case set in Git; record its commit hash in every run, whichever tool ran it
Rubric identity	Store judge prompts and thresholds as files with versions, not inline strings in two tools
Judge configuration	Pin judge model, temperature, and provider; a judge upgrade is a breaking change to every score
Score export	Push framework scores into the platform (Langfuse accepts scores via its API; DeepEval sends runs to Confident AI after `deepeval login`) so a regression is visible in one place
Failure feedback	When a production trace fails, add it to the Git case set, not just to the platform's dataset

Before you rely on any score, check a small fixture of 20 cases by hand. Agree up front on what a human reviewer would call pass or fail, run the judge, and count the disagreements. Neither a vendor nor a metric name can tell you whether a judge is calibrated for your task. No comparative accuracy benchmark across these four tools was run for this article.

Which tool should you pick for your team?

Solo developer or small team, no security requirements: Start with DeepEval (Python) or Promptfoo (anything else) in CI. Add a platform once you are comparing more than a handful of runs a week.
Startup shipping an agent to customers: Use Promptfoo for red teaming before launch, plus Langfuse Cloud or self-hosted for production traces and online evals.
Team that iterates heavily on prompts and wants the best experiment UI without running infrastructure: Braintrust. Check that 14-day Starter retention or the 30-day Pro default matches how far back you compare.
Regulated enterprise with strict data residency: Langfuse or Opik self-hosted, or Braintrust's hybrid data plane if a vendor-run control plane holding only metadata is acceptable to your reviewers.
Team already on DeepEval that wants hosted history: Confident AI is the shortest path. Langfuse plus score export is the open-source path.

For related tooling decisions, see our LLM gateway comparison.

Frequently Asked Questions

Braintrust vs Langfuse: what is the main difference?

Braintrust is a commercial, managed platform centred on experiments. Its control plane is always vendor-operated, and data can live in your cloud on Enterprise. Langfuse is MIT-licensed at its core and centred on traces, and the full platform is free to self-host. Both run datasets, LLM-as-judge scoring, and production tracing.

Is Promptfoo still open source after joining OpenAI?

Yes, according to the project. The March 9, 2026 announcement says Promptfoo will remain open source and keep supporting a range of providers and models, and the GitHub repository still states it is MIT-licensed. If vendor neutrality matters to your procurement, watch the project's provider support and governance over time.

What is the open-source alternative to Confident AI?

DeepEval, the Apache 2.0 framework Confident AI is built around, runs its metrics and test runner locally with no account. For a self-hosted dashboard on top, pair DeepEval with Langfuse or Opik and send scores to that platform.

Can Langfuse replace Promptfoo or DeepEval?

Partly. Langfuse runs dataset experiments and LLM-as-judge evaluators, so it can host offline evals. It does not generate adversarial red-team attacks the way Promptfoo does, and it is not a pytest-native test runner like DeepEval. Teams that need a CI gate usually keep a framework in the pipeline.

Which LLM evaluation tool is cheapest?

The frameworks themselves, Promptfoo and DeepEval, are free. Your real cost is judge-model calls and CI time. Among platforms, Langfuse self-hosted has no license fee but carries infrastructure and operator cost. On the hosted plans, Langfuse Core starts at $29/month, Braintrust Pro is $249/month, and Confident AI Starter is $200/month. Each meters differently, so price your own trace volume.

Do I need a platform if I already run evals in CI?

Not at first. A CI gate with results archived as build artifacts answers "did this change regress?" You need a platform when you have to compare many runs over time, review individual failures with teammates, or score production traffic that never passes through CI.

Sources

Originally published at fp8.co. Subscribe for weekly AI engineering analysis at fp8.co/newsletters.

DEV Community