DEV Community

Cover image for Waxell vs. Braintrust: When Evaluation Isn't Enough
Waxell profile image Logan
Logan for Waxell

Posted on • Edited on • Originally published at waxell.ai

Waxell vs. Braintrust: When Evaluation Isn't Enough

#ai #agents #devtools #architecture

Consider a team running a tight eval suite. Every Friday, they run 500 real production transcripts through Braintrust scorers, iterate on prompts with Loop, and ship only when quality hits above 8.5/10. Their evals are genuinely good — not the performative kind.

Then one of their agents starts routing customer support tickets through an external summarization API. PII goes with them. The eval score? Still 8.7/10. The summarization is excellent. The governance isn't.

The problem wasn't Braintrust. Braintrust was doing exactly what it's designed to do: measure and optimize quality. The problem was that "quality" and "safe to run in production" are different questions, and the team was using one tool to answer both.

Braintrust is a developer-centric AI evaluation and observability platform: score outputs, tune prompts, track quality regressions, and use its in-product AI agent (Loop) to analyze logs, build datasets, and optimize prompts. It also ships a production-facing AI gateway (caching, load balancing, prompt environments) for teams shipping agents. Waxell is a runtime governance control plane: enforce policies at execution time, control tool access, filter outputs, and produce compliance audit trails for what agents do in production. Braintrust answers "is this agent producing good outputs, and how do I ship it reliably?" Waxell answers "is this agent allowed to do what it's doing right now?" You need both questions answered. They're not the same question.

What is Braintrust built for?

Braintrust is built around the evaluation workflow: you have agents producing outputs, you want to know if those outputs are good, and you want a systematic way to improve them. The product is built around that loop — literally, in Braintrust's own naming, since its documented workflow is Instrument → Observe → Annotate → Evaluate → Deploy.

The core capability is browser-based evaluation. You write scorers — functions that assess output quality on whatever dimensions matter to you — run them against versioned datasets of examples, and see results immediately. No infrastructure to manage, no Python scripts to maintain.

Loop is no longer just a prompt-optimization feature — it's Braintrust's general-purpose in-product agent. Loop is available throughout the platform (logs, traces, playgrounds, project pages) and, via natural language, can analyze production logs for patterns, generate SQL filters, find semantically similar traces, build datasets and scorers from what it finds, manage dataset versions and environments, search Braintrust's documentation, and file support tickets — in addition to its original job of suggesting prompt improvements based on your annotations. It's a meaningfully bigger surface than "AI-powered prompt optimization" alone.

Braintrust also connects dev to production, and has expanded further into that seam. Quality metrics from production traces flow back into the eval workflow, so you can see whether the configuration that tested at 8.5/10 is actually delivering 8.5/10 in the wild. The platform now includes a "Deploy" stage of its own: an AI gateway (unified multi-provider proxy with caching, load balancing, and reasoning-model support), versioned prompt environments (dev/staging/production), and deployment monitoring dashboards. None of this is policy enforcement — it's routing, caching, and observability for the calls an agent makes — but it means Braintrust's footprint now extends past pure evaluation into how teams ship and run agents day to day.

The free tier — 1 GB processed data/month and 10K evaluation scores, 14-day retention — is generous enough for serious development use, and the $249/month Pro plan includes 5 GB processed data/month (then $3/GB), 50K scores/month (then $1.50/1K), 30-day retention, RBAC, and priority support.

It's a well-designed tool for a specific job: help development teams ship higher-quality agents, and increasingly, run the plumbing underneath them.

Where does Braintrust stop?

The limits of Braintrust aren't flaws — they're scope decisions. Even with its expanded Deploy tooling, the platform doesn't do runtime policy enforcement. When you move to production with agents that touch sensitive systems, access external APIs, or operate in regulated environments, you're in territory Braintrust doesn't address.

No runtime policy enforcement. Braintrust can tell you that your agent produced a low-quality output. It cannot prevent the agent from making that API call, accessing that file, or routing that PII before the output ever gets evaluated. Braintrust's own "Automations" — the closest thing it has to production rules — are alerts (Slack/webhook notifications), cloud storage exports, and time-based data retention policies. None of them intercept or block an agent action before it executes. Observation happens after the fact; enforcement has to happen before it.

No tool access control. If you're running agents with MCP tools, database access, or external API integrations, Braintrust has no mechanism to restrict which tools an agent can invoke, under what conditions, or with what parameters. An agent that passes your eval suite can still call any tool it has access to in production.

MCP support exists, but it points the other direction. Braintrust does ship an MCP server ("Braintrust MCP"), but it lets AI coding tools — Claude Code, Cursor, VS Code, Windsurf, Claude Desktop — query Braintrust's own data (experiments, logs, docs) from inside your IDE. It's a developer-productivity integration, not a mechanism for governing the MCP tool calls your agents make in production. Waxell's MCP-native support is the latter: policy enforcement on the agent's own MCP tool calls, not IDE access to observability data.

No compliance enforcement record. Braintrust does offer real compliance infrastructure on paid tiers — SOC 2 Type II compliance, SAML SSO and RBAC on Enterprise, a data processing agreement (click-through on Pro, custom on Enterprise), and a business associate agreement on Enterprise for HIPAA-relevant work. What none of that provides is an enforcement record: documented evidence that a specific policy was evaluated and applied before a specific action executed. Access controls and compliance paperwork answer "who could touch this system, and under what agreement." They don't answer "was Policy X enforced before this action ran." For a compliance audit that needs the second answer, the distinction matters.

No rate limiting or cost controls at the session level. A session that loops unexpectedly in production will run until it hits an external limit or you intervene manually. Braintrust will show you it happened; it won't stop it.

None of this is a knock on Braintrust. These are production governance problems, and Braintrust — even with its newer gateway and deployment tooling — is fundamentally an evaluation and observability platform, not a policy-enforcement one. The issue arises when teams assume that passing evals is equivalent to production readiness — which the PII scenario above illustrates clearly.

What Waxell adds

Waxell instruments agent executions across any framework — LangChain, CrewAI, LlamaIndex, custom Python — with execution tracing that captures spans, tool calls, token usage, and timing. That's the observability layer.

On top of it, runtime governance policies, managed in Waxell's governance plane, evaluate before each tool call and output: what tools is this agent allowed to use? What data is it allowed to handle? What does it cost per session? What output content is blocked? If a policy trips, the action is intercepted before execution — not logged after.

The important architectural detail: policies sit above the agent code. They're not baked into prompt instructions, not embedded in the agent's logic, not dependent on the agent "behaving" correctly. They operate at the infrastructure layer, which means they enforce even when the agent's reasoning would lead somewhere else, and they can be updated independently of the agent without a deployment.

For the PII scenario: a content policy checking outbound requests for sensitive data fields would have caught the routing before the external API call completed. The eval score wouldn't have caught it, because the summarization was high quality. The governance layer would have caught it, because the data handling violated policy.

Feature comparison

Capability Waxell Braintrust
Observability
Execution tracing ✅ ✅
Tool call logging ✅ ✅
Production monitoring ✅ ✅ (excellent, includes gateway + deployment monitoring)
Evaluation & Quality
Browser-based eval suite ❌ ✅ (excellent)
In-product AI agent (log analysis, dataset/scorer generation, prompt optimization) ❌ ✅ (Loop)
Dataset versioning ⚠️ Manual ✅ Built-in
Regression detection ⚠️ Via alerts ✅
Custom scorers ✅ ✅
Governance & Runtime Control
Runtime policy enforcement ✅ Core ❌
Tool access control ✅ ❌
Output filtering / content guardrails ✅ ❌
Rate limiting (per session) ✅ ❌
Compliance enforcement record ✅ ❌ (has SOC 2 Type II, SSO/RBAC, DPA/BAA — access controls, not enforcement records)
Human-in-the-loop escalation gates ✅ ❌
Data residency enforcement ✅ ⚠️ EU data plane available; Enterprise custom retention/export
Framework & Protocol Support
Framework-agnostic ✅ ✅
MCP support ✅ Governs agent MCP tool calls ⚠️ MCP server for IDE access to Braintrust's own data (not agent governance)
Deployment
Cloud SaaS ✅ ✅
Self-hosted ✅ ✅ Enterprise
Pricing
Free tier ✅ ✅ 1 GB/mo + 10K scores, 14-day retention
Pro plan Flexible $249/month (5 GB/mo, 50K scores/mo, 30-day retention, RBAC)

Three production scenarios where evaluation isn't sufficient

Scenario 1: The PII leak. Your customer service agent has a 9/10 eval score. In production, a ticket contains a patient's medical history. The agent routes it to an external summarization API as part of its workflow. Your eval scored the summary quality — not the data handling. A Waxell content policy on outbound requests would intercept before the external call. Braintrust shows you the trace afterward, and Loop can help you find every other session with the same pattern — but neither stops the call from happening.

Scenario 2: The loop. An agent encounters an edge case your eval dataset didn't cover, enters a retry loop, and burns 200x its normal session cost before timing out. Your Braintrust dashboard shows a quality anomaly, and an alert automation can notify you via Slack. A Waxell cost policy on per-session token budgets terminates the session automatically when it exceeds threshold, before you'd even see the alert. Other sessions are unaffected.

Scenario 3: The compliance audit. Your security team needs to demonstrate that your agents operated within defined data handling constraints during Q1. Braintrust produces logs showing what your agents did, plus SOC 2 Type II and BAA paperwork proving who could access the system and under what terms. Waxell produces enforcement records: each policy evaluation, what triggered, what action was taken. The difference between evidence of access and evidence of controls.

When to use Braintrust

Braintrust is the right choice when evaluation and quality optimization are the primary workflow. If you're iterating on prompts, comparing model performance, tracking quality regressions over time, or using Loop to automate prompt tuning, analyze logs, or build datasets — Braintrust is the best tool in the category for this. The browser-based eval workflow, the free tier's generosity, and the newer gateway/deployment tooling make it accessible to teams at any scale who want one platform across dev and shipping.

If your agents are internal-only, don't touch sensitive systems, and don't have compliance requirements, Braintrust may be sufficient on its own for production as well. But the moment tool access, external APIs, PII, or regulatory requirements enter the picture, you're asking Braintrust to answer a question it wasn't built to answer.

When to use Waxell

Waxell is the right choice when production governance is required: policy enforcement, tool access control, compliance audit trails, human-in-the-loop gates. These aren't capabilities Braintrust approximates — even its SOC 2 Type II certification and Enterprise access controls answer a different question than runtime policy enforcement does.

For teams in regulated industries — healthcare, financial services, legal — Waxell provides the documented enforcement record that compliance audits require, on top of whatever access-control paperwork Braintrust or another platform already provides. For teams deploying agents with broad tool access, Waxell provides the control layer that evals can't substitute for.

The stack most production teams end up running

The honest answer is that these tools aren't really competitors — they address different phases of the same workflow:

Development: Braintrust for evaluation and prompt optimization. Run evals constantly, use Loop to improve quality and analyze logs, track regressions before they ship.

Production: Waxell for runtime governance. Enforce policies at execution time, control tool access, produce audit trails, handle the edge cases your evals didn't cover.

The PII scenario above isn't an argument against using Braintrust — it's an argument for adding Waxell to the production layer. Both tools running together give you quality optimization in development and safety enforcement in production. Running only one of them leaves either your quality or your governance unaddressed.

How Waxell handles this: Waxell's runtime governance policies operate at the infrastructure layer — above agent code, evaluated before each tool call and output. A content policy catches PII before it leaves your system. A cost policy terminates runaway sessions automatically. A compliance policy produces enforcement records alongside execution traces, giving you the audit documentation that observability data alone doesn't provide. Three lines of SDK code to instrument; policies defined once and enforced across every agent regardless of framework.

Building your prototype today and want to see what production governance looks like once you're ready to ship? Get started with Waxell.

Frequently Asked Questions

What is the difference between Braintrust and Waxell?
Braintrust is an evaluation and observability platform for AI agents, with an in-product AI agent (Loop) for analyzing logs, running evals, and optimizing prompts, plus a gateway for routing production LLM traffic. Waxell is a runtime governance control plane for production agents. It enforces policies before tool calls execute, controls what agents are allowed to do, and produces compliance audit trails. Braintrust tells you whether your agent is producing good outputs and helps you ship it; Waxell controls what your agent is allowed to do while producing them.

Can Braintrust replace Waxell for production governance?
No. Braintrust monitors, evaluates, and now routes production traffic through its gateway — but it doesn't enforce runtime policies, control tool access, block output content, or produce enforcement records. These are governance capabilities that require infrastructure-layer enforcement rather than evaluation, routing, or access-control paperwork. For production agents with compliance requirements, tool access, or sensitive data handling, Braintrust and Waxell serve different and complementary functions.

Should I use Braintrust and Waxell together?
For most production teams, yes. The typical stack is Braintrust in the development and evaluation phase (evals, Loop, regression tracking, and increasingly the AI gateway for shipping) and Waxell in the production governance phase (policy enforcement, audit trail, runtime controls). They're designed for different parts of the agent lifecycle and don't duplicate each other's core capabilities.

Is Waxell a Braintrust alternative?
Only partially. For observability and production monitoring, Waxell is a full replacement. For Braintrust's evaluation workflow — browser-based evals, the Loop agent, dataset versioning, the AI gateway — Waxell doesn't replicate those features. Teams that replace Braintrust with Waxell typically add a separate evaluation tool or maintain Braintrust alongside Waxell for the dev/eval/shipping phase.

What is Braintrust's Loop feature, and does Waxell have something equivalent?
Loop started as an AI-powered prompt optimization feature and has grown into Braintrust's general-purpose in-product agent: it analyzes logs and traces, generates SQL filters and datasets, builds scorers, manages dataset versions and environments, searches documentation, and still optimizes prompts based on your annotations. Waxell doesn't have an equivalent feature — Loop is one of Braintrust's most distinctive capabilities and has no direct counterpart in the governance category. If an in-product AI assistant for evaluation workflows is a core requirement, Braintrust is the right tool for it.

Does Braintrust support MCP?
Yes, but not in the governance sense. Braintrust ships an MCP server that lets AI coding tools (Claude Code, Cursor, VS Code, Windsurf, Claude Desktop) query Braintrust's own experiments, logs, and documentation from your IDE. It does not govern or restrict the MCP tool calls your production agents make. Waxell's MCP support works in the opposite direction: it's a governed surface for an agent's own MCP tool calls, evaluated against policy before execution.

Sources

Top comments (0)