The best LLM observability tool in 2026 depends on what you need to see. For an all-round open-source platform with tracing, evals, cost tracking, and prompt management, pick Langfuse. For LangChain and LangGraph apps, pick LangSmith. For experiment-driven evals, pick Braintrust. If your services already run on Datadog, use Datadog. For the cheapest managed plan at volume or a fully free self-hosted platform, pick Opik. For a free self-hosted workbench with a managed upgrade, pick Arize Phoenix, then Arize AX. To judge whole production conversations and turn failures into pull requests, add Agnost AI alongside one of these.
This page is updated for 2026. A lot changed since 2025: Helicone went into maintenance mode after Mintlify acquired it, Traceloop is joining ServiceNow, Datadog renamed its product to Agent Observability, and most tools added agent and session views. Below we compare eleven tools on tracing, evals, cost tracking, prompts, licence, and price.
If your question is narrower, "how do I know my agent is working for users?", read Best AI agent monitoring tools (2026) instead. That page sorts tools by the monitoring job. This one is the broad LLM observability buyer's guide.
Disclosure: We make Agnost AI. Facts about every other tool come from that vendor's own docs and pricing pages, checked on November 18, 2026. This is a documentation comparison, not a hands-on benchmark. Confirm plan details with each vendor.
What LLM observability covers
An LLM observability tool records what your model and agent did and helps you judge whether it was good. Buyers usually need five things:
- Tracing. Every model call, tool call, and retrieval step for a request, with inputs, outputs, latency, and errors.
- Evals. Scores on outputs, either offline against a dataset before release or online on production traffic.
- Cost and token tracking. Spend by model, user, feature, or agent run.
- Prompt management. Versioned prompts you can change without a deploy, plus a playground.
- Monitoring and alerts. Dashboards and alerts when quality, cost, or error rates move.
No tool is best at all five. Pick by the one that hurts most today.
Quick comparison
| Tool | Best for | Open source | Self-host | Cost tracking | Prompt mgmt | Free tier | Entry paid plan | Billing unit |
|---|---|---|---|---|---|---|---|---|
| Langfuse | All-round open source | MIT (except ee) |
Free | Yes | Yes | 50K units | Core $29/mo | Units |
| LangSmith | LangChain and LangGraph | No | Enterprise | Not stated | Not stated | 5K traces, 1 seat | Plus $39/seat/mo | Seats plus traces |
| Arize Phoenix | Free self-hosted workbench | ELv2 (source-available) | Free | Not stated | Yes | Free | — | — |
| Arize AX | Managed evals and issue discovery | No | Enterprise | Yes | Not stated | 25K spans | Pro $50/mo | Spans and GB |
| Braintrust | Experiments and scorers | No | Enterprise | Yes | Prompt versions | 1 GB, 10K scores | Pro $249/mo | GB plus scores |
| Datadog | Teams already on Datadog | No | No | Yes | Not stated | 40K LLM spans | Pro from $160/mo (annual) | LLM spans |
| Opik | Cheapest managed, free self-host | Apache-2.0 | Free, full platform | Yes | Yes | 25K spans | Pro $19/mo | Spans |
| W&B Weave | Teams on Weights & Biases | SDK Apache-2.0 | Enterprise | Not stated | Not stated | 1 GB | Pro from $60/mo | MB ingested |
| Traceloop / OpenLLMetry | OTel instrumentation for any backend | OpenLLMetry Apache-2.0 | Enterprise on-prem | Not stated | Yes | 50K spans, 24 h | Enterprise only | Spans |
| Laminar | Open-source tracing, CI evals, Signals | Apache-2.0 | Yes (Docker Compose) | Not stated | Not stated | 1 GB, 1 seat | Starter $30/mo | GB plus Signal tokens |
| Helicone | Not for new projects (maintenance mode) | Apache-2.0 | Yes | Not stated | Yes | 10K requests | Pro $79/mo | Requests and storage |
| Agnost AI | Whole-conversation failures turned into PRs | No | Enterprise (VPC) | No | No | 1K events | Starter $49/mo | Events |
"Not stated" means the vendor pages we checked didn't say, not that the feature is missing.
Langfuse
What it is. An open-source LLM engineering platform. It covers tracing of LLM and non-LLM steps, token and cost tracking per user, prompt management with versioned labels, a playground, LLM-as-a-judge and code evaluators, datasets with experiments, and dashboards that alert to Slack, GitHub Actions, or webhooks. It accepts OpenTelemetry on an OTLP endpoint and has typed Python and JS/TS SDKs plus a drop-in OpenAI wrapper.
Best for. Teams that want one open-source tool for the whole LLM observability list, with the option to self-host.
Limits. Self-hosting is free but not small: you run web and worker containers on Postgres, ClickHouse, Redis or Valkey, and S3-compatible storage. A few add-ons, such as the Instance Management API, need an enterprise licence key. LLM-as-a-judge evaluators now target observations or experiments: trace-level evaluators are deprecated and stopped producing results on Langfuse Cloud after November 16, 2026. See what to do about them.
Pricing. Self-hosting is free. Hobby is free for 50,000 units and 2 users. Core is $29 a month and Pro $199 a month, each with 100,000 units, then $8 per 100,000 up to 1 million and $7 per 100,000 from 1 to 10 million. A unit is any trace, observation, or score.
LangSmith
What it is. LangChain's platform for tracing, datasets, online evaluations, dashboards and alerts, annotation queues, and Engine, which detects recurring issues in traces and helps resolve them. It accepts OpenTelemetry through an /otel endpoint.
Best for. Teams building on LangChain and LangGraph.
Limits. Plus is priced per seat. Base traces are kept 14 days, and extended retention (180 days) costs extra. The public pricing page doesn't print a per-trace overage rate; usage is metered in LangChain Standard Units at $1 each through a calculator. Self-hosting is Enterprise only.
Pricing. Developer is free for 1 seat and 5,000 base traces a month. Plus is $39 per seat per month with 10,000 base traces, then pay as you go. Enterprise is custom.
See also LangSmith alternatives.
Arize Phoenix and Arize AX
What they are. Phoenix is Arize's self-hostable tool built on OpenTelemetry and OpenInference. It covers tracing, LLM and code evaluators, a prompt playground with span replay, and datasets with experiments. AX is the managed platform. It records cost on each traced run, runs online evals continuously, and its Signal feature groups recurring failures into ranked issues with a proposed fix.
Best for. Phoenix fits teams that want a free self-hosted workbench with no feature gates. AX fits teams that want managed evals and issue discovery.
Limits. Phoenix is under the Elastic License 2.0, which is source-available, not OSI open source. Arize's pricing page lists no overage rates for AX, so price anything above the Pro allowance with sales.
Pricing. Phoenix is free to self-host. AX Free covers 25,000 spans and 1 GB a month with 15-day retention. AX Pro is $50 a month for 50,000 spans and 10 GB with 30-day retention. Users are unlimited.
Braintrust
What it is. An evaluation and observability platform built around experiments, datasets, and scorers, with tracing, a Discover Patterns feature that finds recurring trace problems, and an AI assistant called Loop. Its SDK wrappers attach token counts and estimated cost to each LLM span. It accepts OpenTelemetry.
Best for. Teams whose releases run on experiments and who want production logs to feed datasets and scorers.
Limits. Each score is metered, so scoring all production traffic gets expensive. Self-hosting is Enterprise only.
Pricing. Starter is free with 1 GB and 10,000 scores, then $4 per GB and $2.50 per 1,000 scores, with 14-day retention. Pro is $249 a month with 5 GB and 50,000 scores, then $3 per GB and $1.50 per 1,000 scores. Users are unlimited.
Datadog LLM Observability (Agent Observability)
What it is. Datadog's LLM tracing product, now sold as Agent Observability. It covers traces, cost and token dashboards, managed and custom evaluations, Patterns topic clustering, and sensitive-data and prompt-injection scanning. You can instrument with its SDK or send OpenTelemetry spans that follow the GenAI conventions (1.37+) or OpenInference, without a Datadog Agent.
Best for. Teams whose services already run on Datadog and want LLM traces linked to APM.
Limits. No self-hosted option is described. OTel traces can take 3 to 5 minutes to appear. Retention is 15 days unless you pay for an add-on.
Pricing. Free covers 40,000 LLM spans a month. Pro starts at $160 a month on an annual commitment ($200 month-to-month, $240 on-demand) for 100,000 LLM spans, then $3.50 per 10,000 on annual billing. Only calls to a model provider count; tool and retrieval spans are not billed.
See also Datadog LLM Observability alternatives.
Opik
What it is. Comet's Apache-2.0 platform for tracing, cost tracking, LLM-as-a-judge and heuristic evals, prompt management with a playground, and automatic prompt optimization. Online rules score spans, traces, or whole threads, with a sampling rate up to 100%. Thread rules wait for a 15-minute cooldown by default. Score alerts go to Slack, PagerDuty, or a webhook. It accepts OpenTelemetry.
Best for. Teams that want the full platform free to self-host, or the lowest managed price at volume.
Limits. Alerts fire when an average score crosses a threshold. Self-hosting runs locally for testing or on Kubernetes for production.
Pricing. Open source is free. Cloud Free covers 25,000 spans a month. Pro is $19 a month for 100,000 spans, then $5 per 100,000.
W&B Weave
What it is. Weights & Biases' toolkit for tracing and evaluating LLM apps and agents, with LLM judges, custom scorers, and an Agents view of sessions, turns, LLM calls, and tool calls. It accepts OpenTelemetry and reads GenAI, OpenInference, Vercel AI SDK, Traceloop, and other conventions.
Best for. ML teams that already use Weights & Biases.
Limits. The GitHub repo is the Apache-2.0 client SDK; the trace server lives in W&B's core platform, and private hosting is an Enterprise option. Billing is by ingested data, so verbose traces cost more. Pricing now lives on CoreWeave's site.
Pricing. Free includes 1 GB of Weave ingestion a month. Pro starts at $60 a month with 1.5 GB, then $0.10 per MB.
Traceloop and OpenLLMetry
What they are. OpenLLMetry is an Apache-2.0 set of OpenTelemetry extensions for LLM apps. You call Traceloop.init() and it exports to Traceloop or to 24 tested backends, including Datadog, Grafana, Honeycomb, Laminar, New Relic, Sentry, and any OpenTelemetry Collector. Traceloop is the hosted platform on top, with monitoring, evaluators, prompt management, and a proxy called Traceloop Hub (beta).
Best for. Teams that want vendor-neutral instrumentation they can point at any backend.
Limits. Traceloop announced in March 2026 that it is joining ServiceNow. It says it will keep selling Traceloop and that OpenLLMetry will remain open source, with roadmap changes announced in advance. The free plan keeps data for 24 hours.
Pricing. Free Forever covers up to 50,000 spans a month with 24-hour retention. Above that is Enterprise, priced by quote, with an on-prem option.
Laminar
What it is. Apache-2.0, OpenTelemetry-native agent observability written in Rust. It has one-line auto-tracing for the Vercel AI SDK, OpenAI, Anthropic, and LangChain, an evals SDK and CLI that run locally or in CI, SQL access to traces, and Signals: plain-English checks an LLM runs on each finished trace, with Slack or email alerts.
Best for. Engineering teams that want open-source tracing with CI evals.
Limits. Signals aren't on the free plan and are billed by tokens. The free plan has one seat and 7-day retention.
Pricing. Free with 1 GB. Starter is $30 a month for 3 GB, then $2 per GB. Pro is $150 a month for 10 GB, then $1.50 per GB, with 6-month retention.
Helicone
What it is. An Apache-2.0 AI gateway with logging. You point an OpenAI-compatible client at Helicone's gateway URL and get routing across 100+ models, sessions, prompts, datasets, and a playground.
Status. Mintlify acquired Helicone on March 3, 2026 and said it "will continue operating in maintenance mode," with security updates, bug fixes, and new models, and that it will help customers migrate. Helicone's site now says it "is now part of Experiential Labs. Details to come soon."
Our take. Don't start a new project on it. If you run it today, keep it patched and plan a move.
Pricing. Hobby is free for 10,000 requests. Pro is $79 a month and Team $799 a month, plus usage-based charges.
Agnost AI
What it is. Agent monitoring for AI agents in production: chat agents, voice agents (through transcripts), copilots, and MCP servers. It is OpenTelemetry-native, with Python and TypeScript SDKs, and works with the OpenAI Agents SDK, Vercel AI SDK, LangChain and LangGraph, Mastra, Pydantic AI, Google ADK, and Agno.
How it differs. Most tools above judge single requests or traces. Agnost groups traffic by user and session and judges whole conversations. It detects failures, user frustration, intents, tool-call errors, loops, and silent wrong answers. Its online evals are plain-language rules: once a conversation has been idle for 15 minutes, an LLM marks each assistant message pass or fail against every rule, in batches, with no sampling. Alerts go to Slack or email. Auto Improve opens a pull request with a fix tested against past runs, with the replay results in the PR, for a person to review.
Best for. Teams whose failures only show up across a conversation, such as a wrong answer three turns in or a user who gives up.
Limits. Agnost is not a full LLM observability platform. It has no cost tracking, no prompt management, no offline datasets or experiments, and no CI gating. Verdicts are pass or fail, not scores, and judging runs in batches, not in real time. It is not open source. Most teams run it next to Langfuse, LangSmith, Braintrust, or Datadog.
Pricing. Free covers 1,000 events a month with 7-day retention. Starter is $49 a month for 10,000 events. Pro is $499 a month for 1,000,000 events. Enterprise is custom and includes self-hosted VPC deployment.
How to choose
| If your priority is… | Pick |
|---|---|
| One open-source tool for tracing, evals, cost, and prompts | Langfuse |
| The full platform free to self-host, or the cheapest managed plan at volume | Opik |
| LangChain or LangGraph | LangSmith |
| Experiments, datasets, and scorers gating releases | Braintrust |
| LLM traces next to your existing APM | Datadog |
| A free self-hosted workbench now, managed later | Arize Phoenix, then Arize AX |
| You already train models in Weights & Biases | W&B Weave |
| Vendor-neutral OpenTelemetry instrumentation | OpenLLMetry, exporting to any backend above |
| Open-source tracing plus evals in CI | Laminar |
| Knowing whether whole conversations worked for users, with fixes as PRs | Agnost AI, alongside one of the above |
Three checks before you commit:
- Instrument with OpenTelemetry if you can. Every tool above accepts OTel in some form, so you can switch or send to two backends without re-instrumenting. Our OpenTelemetry guide shows the spans to emit.
- Price your own volume. Units, traces, spans, LLM spans, GB, MB, scores, and events don't convert one-to-one. We worked the numbers at 10K, 100K, and 1M traces a month in the easiest and best-value monitoring tools.
- Decide on self-hosting early. If data can't leave your network, see free and open-source agent monitoring tools for what each licence allows.
Related reading
- Best AI agent monitoring tools (2026): pick by the job
- Best free and open-source AI agent monitoring tools (2026)
- Easiest AI agent monitoring tools to set up, and the best value
- LangSmith alternatives for agent evals and monitoring
- Datadog LLM Observability alternatives
- Best online evaluation tools for AI agents
Frequently asked questions
What is the best LLM observability tool in 2026?
There is no single best tool; it depends on the job. Langfuse is the best all-round open-source pick, with tracing, evals, cost tracking and prompt management, free to self-host. LangSmith is best for LangChain and LangGraph teams. Braintrust is best when releases run on experiments and scorers. Datadog is best if your services already run on Datadog. Opik is the cheapest managed plan at volume and fully free to self-host. Arize Phoenix is a free self-hosted workbench with Arize AX as the managed upgrade. Agnost AI is best for judging whole production conversations and turning failures into pull requests, alongside one of the others.
What is the best open-source LLM observability tool?
Langfuse (MIT, except its ee folders) and Opik (Apache-2.0, full platform free to self-host) are the most complete. Laminar is Apache-2.0 with tracing, evals and Signals. OpenLLMetry is an Apache-2.0 OpenTelemetry instrumentation library, not a backend. Arize Phoenix is free to self-host with no feature gates, but under the Elastic License 2.0, which is source-available rather than OSI open source. W&B Weave's SDK is Apache-2.0, but its trace server is not in that repo.
What is the difference between LLM observability and AI agent monitoring?
LLM observability records and inspects model calls: traces, token usage, cost, latency, prompts and eval scores on individual requests. AI agent monitoring asks whether the agent did its job for the user across a whole session or conversation: failed tasks, tool-call errors, loops, frustration and wrong answers that never throw an error. Most teams need both, and most tools lean one way.
Which LLM observability tools track cost?
Langfuse, Opik, Datadog, Arize AX and Braintrust all record token usage and estimated cost on traced LLM calls, according to their docs. Agnost AI does not track cost, so pair it with one of these if spend matters.
Is Helicone still maintained?
Mintlify acquired Helicone on March 3, 2026 and said Helicone will continue operating in maintenance mode, with security updates, bug fixes and new models, and that it will help customers migrate to another platform. Helicone's own site now says it is part of Experiential Labs, with details to come. We would not pick it for a new project in November 2026.
Which LLM observability tool is cheapest?
Self-hosting Langfuse, Opik or Arize Phoenix costs no licence fee, only your infrastructure. Of the managed plans, Opik Pro is $19 a month for 100,000 spans plus $5 per 100,000 more, which makes it the cheapest at high volume. Langfuse Core is $29 a month for 100,000 units. See our setup and pricing comparison for worked costs at 10K, 100K and 1M traces a month.
Does OpenTelemetry work with LLM observability tools?
Yes, almost all of them. Langfuse, LangSmith, Braintrust, Arize Phoenix, Opik, W&B Weave, Laminar and Agnost AI accept OTLP traces, and Datadog ingests spans that follow the OpenTelemetry GenAI semantic conventions (1.37+) or OpenInference without a Datadog Agent. OpenLLMetry instruments your app with OpenTelemetry and exports to 24 tested backends.
What changed in LLM observability since 2025?
Consolidation. Mintlify acquired Helicone in March 2026 and put it in maintenance mode. Traceloop announced it is joining ServiceNow in March 2026 and said OpenLLMetry will stay open source. Datadog now sells LLM Observability as Agent Observability. W&B pricing moved to CoreWeave's site. LangSmith meters usage in LangChain Standard Units at $1 each. And most tools added agent features: session views, issue clustering and automatic failure detection.
Originally published on agnost.ai.
Top comments (0)