DEV Community

Cover image for Waxell vs. Arize Phoenix: The Iteration Tool vs. the Production Control Plane
Waxell profile image Logan
Logan for Waxell

Posted on • Edited on • Originally published at waxell.ai

Waxell vs. Arize Phoenix: The Iteration Tool vs. the Production Control Plane

#ai #agents #mlops #devtools

There's a gap that opens up between testing and production that most ML teams discover the hard way.

The scenario: an engineering team spent three weeks tuning their agent in Arize Phoenix. Prompt versions, model comparisons, LLM-based scoring on 200 historical examples. The eval results were clean — 91st percentile quality scores, no regressions, consistent behavior across edge cases. They shipped.

In production, the agent had write access to the customer database. The eval suite had covered every question in the prompt quality dimension. It hadn't covered what happens when a tool orchestration bug causes the agent to issue an UPDATE query without a WHERE clause. Phoenix showed them exactly what happened, span by span. There was no layer that required approval before the destructive query ran.

This isn't a failure of Arize Phoenix. It's a failure to recognize that Phoenix answers a different question than the one production safety requires.

Arize Phoenix is an open-source AI observability and experimentation platform: version prompts, run experiments, evaluate output quality, detect regressions, and increasingly, debug and iterate with its own built-in AI agent — the research and iteration workflow before you ship. Arize's managed offering, Arize AX, extends further into continual improvement of agents already in production. Waxell is a production control plane: enforce runtime policies, control tool access, filter outputs, and govern what agents can do once you ship. Phoenix answers "was that output good, and how do I make it better?" Waxell answers "is that action allowed?" These are different questions. Both matter. Neither replaces the other.

What is Arize Phoenix built for?

Phoenix is designed for the iteration phase of AI development — the work that happens between "we have an agent" and "we have an agent we trust enough to ship."

The core workflow is experiment tracking. You take a version of your prompt, model, and parameters; run them against a dataset of representative examples; score outputs with custom metrics or LLM-based evaluation; and compare the results to a baseline. When you swap models or change your system prompt, Phoenix shows you whether quality went up or down and where specifically it regressed. The prompt playground lets you iterate visually rather than in code. Tracing captures model calls, retrieval, tool use, and custom logic — Phoenix's execution graph is not limited to LLM calls in isolation.

Phoenix now ships PXI (Phoenix Intelligence), a built-in AI engineering agent (currently in beta) that investigates failing traces, proposes root causes, iterates on prompts as reviewable diffs, authors evaluators, and annotates spans in bulk — all scoped to whatever you're already looking at in the product. It's the same category of capability as similar in-product agents that have shown up across this space recently, and it's opt-in with an explicit approval gate for any state-changing action.

Phoenix is fully open-source with no feature gates. Self-hosted, free to run, no per-trace fees. For ML teams that evaluate constantly, this is a meaningful cost advantage — especially compared to managed platforms that charge at volume. The managed Arize AX offering has grown well beyond "Phoenix plus support": it's now positioned as a "continual learning platform for agents," adding its own in-product engineering agent (Alyx), a dedicated trace datastore (ADB) that connects to BigQuery/Databricks/Snowflake, and native workflows for coding agents like Cursor, Claude Code, and OpenCode.

The thesis is sound: ship agents that you've actually measured. Know what changed between versions. Catch quality regressions before users do — and increasingly, let an agent help you find and fix them.

Where does Arize Phoenix stop?

The scope decision Phoenix and Arize AX make is to focus on evaluation, iteration, and continual improvement. Once an agent is in production, Phoenix and Arize AX give you visibility into what happened, and now some agent-assisted help interpreting it. Neither gives you control over what happens next, in the sense of stopping an action before it executes.

No runtime policy enforcement. Phoenix can tell you that an agent made a bad call, and PXI can help you figure out why. Neither can intercept that call before it executes. The database query runs, the external API gets hit, the PII leaves your system — and Phoenix records it accurately. That's not a flaw; it's the architecture of an evaluation and iteration tool, even one with an AI agent built in.

No tool access control. There's no mechanism in Phoenix or Arize AX to restrict which tools an agent can invoke, under what conditions, or with what parameters. An agent that scored 91st percentile in your eval suite can still call any tool it has access to in production.

MCP support exists, but in the tooling direction, not the governance direction. Phoenix ships a dedicated MCP server that lets AI coding assistants query your Phoenix instance directly, and Phoenix's tracing integrations include MCP as a traceable protocol (so MCP tool calls inside an instrumented app show up in traces). Neither is a policy or enforcement layer — Phoenix can show you an MCP tool call happened; it has no mechanism to scope, block, or rate-limit one before it executes.

Complex setup relative to its scope. Phoenix requires real infrastructure even in its self-hosted form. For teams that want to go from agent code to production-governed deployment quickly, the infrastructure overhead is a friction point.

These aren't criticisms — they're scope. Phoenix and Arize AX are built for the iteration-and-improvement phase of the workflow, however AI-assisted that phase has become. The problem emerges when teams treat evaluation coverage as a substitute for production controls.

What Waxell adds

Waxell picks up where evaluation leaves off. Three lines of SDK code instrument your agent — any framework, any architecture — with execution tracing that captures the full graph: LLM calls, tool invocations, external API calls, timing, token usage, costs. That's the observability layer.

On top of it, runtime governance policies, managed in Waxell's governance plane, operate before each tool call and output. A policy can require approval before any destructive database operation. A policy can block outbound requests containing PII. A policy can terminate a session that exceeds a per-session cost threshold. A policy can escalate to a human reviewer when confidence falls below a defined threshold.

The architectural point that matters: policies are infrastructure, not instructions. They don't live in the prompt, they don't depend on the model following directions, and they can be updated without a deployment. An agent that was 91st percentile in Phoenix evals and then hits a novel edge case in production is still subject to the governance policies — because those policies operate at the execution layer, not the reasoning layer.

For the database scenario: a tool access policy requiring human approval before any write operation would have intercepted the UPDATE query before it ran. The agent's eval score would have been unchanged. The production incident wouldn't have happened.

Feature comparison

Capability Waxell Arize Phoenix
Observability
Execution tracing ✅ ✅
LLM call logging ✅ ✅ (core)
Full agent workflow tracing (tool use, retrieval, custom logic) ✅ ✅
Experimentation & Iteration
Prompt versioning ❌ ✅ (core)
Experiment tracking ❌ ✅ (excellent)
Built-in AI agent for debugging/iteration ❌ ✅ (PXI, beta)
Model comparison ⚠️ Via traces ✅ (excellent)
Prompt playground ❌ ✅
Evaluation
LLM-based output scoring ⚠️ ✅ (built-in)
Regression detection ⚠️ Via alerts ✅ (automated)
Dataset versioning ❌ ✅
Governance & Runtime Control
Runtime policy enforcement ✅ Core ❌
Tool access control ✅ ❌
Output filtering / content controls ✅ ❌
Rate limiting (per session) ✅ ❌
Human-in-the-loop escalation ✅ ❌
Compliance audit trail (enforcement record) ✅ ❌
Framework & Protocol Support
Framework-agnostic ✅ ✅
MCP support ✅ Governs agent MCP tool calls ⚠️ MCP server for IDE access + MCP trace instrumentation (not agent governance)
Deployment
Cloud SaaS ✅ ✅ (Arize AX)
Self-hosted ✅ ✅ (fully open-source; Arize AX Enterprise also self-hostable)
Setup complexity Low (3 lines) Moderate (infrastructure)
Pricing
Free tier ✅ ✅ Phoenix fully free (self-hosted); Arize AX Free: 25k spans/mo, 1GB, 15-day retention
Managed cloud Flexible Arize AX Pro $50/month (50k spans, 10GB, 30-day retention); Enterprise custom

Two workflows, one agent lifecycle

The cleanest way to think about Phoenix/Arize AX and Waxell is as sequential (and increasingly overlapping) layers of the same agent deployment lifecycle.

Phase 1 — Iteration and continual improvement (Phoenix / Arize AX): You're building, improving, and now often using an in-product agent to help you do both faster. You want to know if your prompt changes improved quality. You want to catch regressions before they reach users. You want to compare models and track which configuration performs best. Phoenix and Arize AX do this well. It's the tool for the question: "Should I ship this, and how do I make it better?"

Phase 2 — Production (Waxell): You've shipped. Your agents are running against real users, real data, and real external systems. You need to know what they're doing, stop them from doing things they shouldn't, and produce documentation that they operated within defined constraints. Waxell does this. It's the tool for the question: "Can I prove this is safe?"

Most teams need both phases covered. The gap the database scenario exposed isn't unusual — it's what happens when teams invest heavily in Phase 1 tooling and skip Phase 2 tooling, and no amount of AI-assisted debugging in Phase 1 substitutes for runtime enforcement in Phase 2.

When to use Arize Phoenix

Phoenix is the right choice when prompt engineering and model optimization are your primary workflow. If you're iterating on configurations, comparing models, or tracking quality regressions across versions, Phoenix's experiment and evaluation capabilities are among the best available — and the fully open-source, self-hosted model makes it accessible at any budget. If you want an in-product agent (PXI) to help investigate failures and iterate on prompts, that's a genuine differentiator worth factoring in.

For ML-ops and research teams that evaluate constantly, Phoenix's free tier with no per-trace fees is a significant cost advantage. It's also the more complete tool for teams whose agents are still in the pre-production iteration phase, where the primary question is output quality rather than production safety — and Arize AX extends that same strength into ongoing production improvement, not production governance.

When to use Waxell

Waxell is the right choice once agents are in production and interacting with real systems. Policy enforcement, tool access control, compliance audit trails, and human-in-the-loop gates aren't capabilities Phoenix or Arize AX approximate — they're out of scope for an evaluation and improvement platform by design, regardless of how much AI assistance is layered into that platform.

For teams in regulated industries, teams deploying agents with broad tool access, or teams that need to demonstrate to auditors that their agents operated within defined constraints, Waxell provides what evaluation tooling cannot: evidence of enforcement, not just evidence of behavior.

How Waxell handles this: Waxell's runtime governance policies sit at the infrastructure layer — above agent code, evaluated before each tool call and output. A policy requiring human approval before write operations would have intercepted the database query before execution. Waxell's execution tracing captures the full agent workflow graph — not just LLM calls, but every tool invocation, external request, and session event — alongside the enforcement records showing what governance policies evaluated and what they allowed or blocked. Three lines of SDK to instrument; policies updated without a deployment.

Iterating in Phoenix today and want to see what production governance looks like once you're ready to ship? Get started with Waxell.

Frequently Asked Questions

What is the difference between Waxell and Arize Phoenix?
Arize Phoenix is an open-source AI observability and experimentation platform designed for the development and iteration phase: prompt versioning, experiment tracking, output evaluation, regression detection, and (via its built-in PXI agent) AI-assisted debugging and prompt iteration. Waxell is a production control plane designed for the deployment phase: runtime policy enforcement, tool access control, compliance audit trails, and governance of agent behavior in production. Phoenix helps you decide whether to ship an agent, and helps you improve it; Waxell governs what that agent can do once you ship it.

Can Arize Phoenix enforce runtime policies for AI agents?
No. Arize Phoenix (and the managed Arize AX platform) are built for evaluation, observability, and continual improvement — they record and analyze what agents did, and can even help you debug and fix issues via PXI or Alyx, but neither intercepts or controls agent behavior at runtime. There's no mechanism to block tool calls, filter outputs, enforce cost limits, or require human approval before actions execute. These are production governance capabilities that require a separate control-plane layer.

Is Arize Phoenix free?
Arize Phoenix is fully open-source and free to self-host with no feature gates and no per-trace fees. The managed Arize AX platform has a free tier (25k spans/month, 1GB ingestion, 15-day retention), a Pro tier at $50/month (50k spans, 10GB, 30-day retention), and custom Enterprise pricing. For teams comfortable managing their own infrastructure, self-hosted Phoenix has no cost ceiling.

Should I use Arize Phoenix and Waxell together?
For teams moving agents from development into production, yes — they serve different phases. Phoenix/Arize AX cover the iteration, evaluation, and continual-improvement phase (should I ship this, and how do I make it better?). Waxell covers the production governance phase (can I prove this is safe?). They don't duplicate each other's capabilities, and using both gives you quality measurement and improvement in development and safety enforcement in production.

Does Arize Phoenix support MCP (Model Context Protocol)?
Yes, but not as governance. Phoenix ships a Phoenix MCP Server that lets AI coding assistants query your Phoenix instance, and Phoenix can trace MCP tool calls made by an instrumented application as part of its normal tracing integrations. Neither gives Phoenix a way to scope, approve, or block an MCP tool call before it executes. Waxell's MCP support is a governance layer: every MCP tool call an agent makes is evaluated against policy before it reaches the model or the downstream server.

Sources

Top comments (0)