DEV Community

Shubham Palriwala
Shubham Palriwala

Posted on Originally published at agnost.ai

Best Datadog LLM Observability alternatives (2026)

The best Datadog LLM Observability alternatives are Langfuse, LangSmith, Arize Phoenix and AX, Braintrust, W&B Weave, New Relic AI monitoring, and Agnost AI. Langfuse is the open-source pick you can self-host for free. LangSmith fits LangChain and LangGraph teams. Arize covers free self-hosting (Phoenix) and managed session-level evals (AX). Braintrust fits teams that run releases on experiments and scorers. W&B Weave fits teams already on Weights & Biases. New Relic fits teams whose APM is New Relic. Agnost AI judges whole production conversations against plain-language rules and turns failures into tested pull requests, and it runs alongside Datadog rather than replacing it. Helicone has been in maintenance mode since March 2026, so skip it for new projects.

Datadog LLM Observability, now sold as Agent Observability, is a capable product. It includes evaluations, datasets, and experiments on every tier, and it is the obvious choice if your services already run on Datadog. This guide covers the reasons teams look elsewhere that hold up against Datadog's own pricing and docs, then compares each alternative on what it does, who it fits, its limits, and its price.

Disclosure: We make Agnost AI. Facts about every other tool come from that vendor's own docs and pricing pages, checked on November 18, 2026. This is a documentation comparison, not a hands-on benchmark. Confirm plan details with each vendor.

Why teams look for Datadog LLM Observability alternatives

These are the reasons that hold up against Datadog's own pricing and docs:

  • Cost grows with every model call. Datadog bills per LLM span, and an LLM span is a single call to a model provider. One agent run that plans, calls tools, and answers produces several. Pro starts at $160 a month for 100,000 LLM spans on an annual commitment ($200 month-to-month, $240 on-demand). Each extra 10,000 costs $3.50 annual, $4.20 month-to-month, or $5 on-demand. At 1 million LLM spans a month, that is $475 a month on annual billing or $690 on-demand.
  • Short default retention, and paying for more costs more than the spans. Traces are kept for 15 days. Keeping them 90 days is an add-on at $9 per 10,000 LLM spans a month, which is more than the $3.50 base overage rate. Retaining 1 million LLM spans for 90 days adds $900 a month on top of the $475.
  • No self-hosting. Datadog's docs describe Agent Observability only on Datadog's hosted sites and describe no self-hosted option. It is also not available on Datadog's US government sites. Teams with data-residency rules often look at Langfuse or Arize Phoenix, which are free to self-host.
  • Not open source. The platform is a commercial SaaS. If you want to read, fork, or run the code, Langfuse, Weave, and Phoenix are options.
  • LLM judges run on your own model account. Template and custom LLM-as-a-judge evaluators use an LLM provider account you connect with your own key. Datadog's evaluations are unlimited, but the judge's model calls go through that account, so plan for that usage on your provider's side. Datadog's docs don't say whether judge calls also count as billable LLM spans.
  • Session-level judging needs explicit session boundaries. Datadog's Goal Completeness evaluator "runs only on sessions marked as completed," and session-scoped judges need spans tagged with a session_id. Its docs don't say how a session is marked complete, so chat and voice teams should confirm how that fits their conversations.

Two complaints that don't hold up. You don't need the rest of Datadog: Datadog says Agent Observability is a standalone product you can buy without any other Datadog subscription. And you don't pay for every tool call: tool, retrieval, and workflow spans are not billed, only LLM spans. If you do run Datadog APM, Infrastructure, or RUM, correlating agent traces with them on one trace ID is a real advantage that no tool below fully matches.

Quick comparison

Tool Best for Open source Self-host OpenTelemetry Entry paid plan Billing unit
Datadog (baseline) Teams already on Datadog APM No No GenAI semconv 1.37+ and OpenInference Pro from $160/mo (annual) LLM spans
Langfuse Open-source tracing, prompts, evals MIT (except ee folders) Free Yes, OTLP endpoint Core $29/mo Units
LangSmith LangChain and LangGraph teams No Enterprise Yes Plus $39/seat/mo Seats plus traces
Arize Phoenix Free self-hosted workbench ELv2 (source-available) Free Yes, plus OpenInference Free Not applicable
Arize AX Managed evals at span, trace, session No Enterprise Yes Pro $50/mo Spans and GB
Braintrust Experiments and scorers No Enterprise Yes Pro $249/mo GB plus scores
W&B Weave Teams on Weights & Biases Apache-2.0 repo Self-Managed endpoint documented GenAI and OpenInference Pro from $60/mo Ingested data (MB)
New Relic AI monitoring Teams whose APM is New Relic No Not documented Yes Free 100 GB, then $0.40/GB plus users GB ingested plus users
Helicone Not recommended for new projects Apache-2.0 Docker or Helm — Pro $79/mo Requests and storage
Agnost AI Whole-conversation failures turned into PRs No Enterprise (VPC) Yes, reads OTel traces Starter $49/mo Events

Langfuse

What it is. An open-source LLM engineering platform for tracing, prompt management, evaluations, and datasets with experiments. It is built on OpenTelemetry and accepts OTLP traces directly. The repo is MIT-licensed except for its ee folders, and you can self-host with Docker Compose, Helm, or Terraform.

Best for. Teams leaving Datadog for open source and free self-hosting, who want prompts and evals in the same tool.

Limits. LLM-as-a-judge evaluators target observations or experiments, with sampling. Trace-level evaluators are deprecated and stopped producing results on Langfuse Cloud after November 16, 2026, and sessions are not an evaluation target. See what to do about Langfuse's trace-level evaluators. There is no infrastructure APM to correlate with.

Pricing. Self-hosting is free. Cloud Hobby is free for 50,000 units and 2 users with 30 days of data access. Core is $29 a month and Pro $199 a month, each with 100,000 units, then $8 per 100,000, and unlimited users. Enterprise is $2,499 a month. A unit is any tracing data point (trace, observation, or score), so it doesn't map one-to-one to Datadog's LLM spans.

LangSmith

What it is. LangChain's tracing, evaluation, and monitoring platform. It traces LangChain and LangGraph natively and accepts OpenTelemetry traces from any OTel-compatible agent.

Best for. Teams building on LangChain and LangGraph.

Limits. Plus is priced per seat, so cost grows with the number of people who need access. Base traces are kept 14 days, and extended traces (180 days) cost extra. Self-hosting is an Enterprise option. Like Datadog, it is not open source.

Pricing. Developer is free for 1 seat and 5,000 base traces a month. Plus is $39 per seat per month with 10,000 base traces, then pay as you go. Enterprise is custom.

For more, see LangSmith alternatives.

Arize Phoenix and Arize AX

What they are. Phoenix is Arize's self-hostable tracing, evaluation, prompt, and datasets-with-experiments tool, built on OpenTelemetry and OpenInference. AX is Arize's managed platform. Its online evals run an LLM judge, code, agent-as-a-judge, or remote evaluator at span, trace, or session level, with sampling.

Best for. Phoenix fits teams that want free self-hosting with no feature gates. AX fits teams that want managed evals, including session-level judging, without running infrastructure.

Limits. Phoenix uses the Elastic License 2.0, which is source-available rather than OSI open source. AX Free and Pro are SaaS only. Self-hosted AX is an Enterprise option.

Pricing. Phoenix is free to self-host. AX Free covers 25,000 spans and 1 GB a month with 15-day retention. AX Pro is $50 a month for 50,000 spans and 10 GB with 30-day retention. Users and evals are unlimited on every AX tier.

For more, see Arize alternatives.

Braintrust

What it is. An evaluation and observability platform built around experiments, datasets, and scorers. It accepts OpenTelemetry spans. Online scoring rules run at span, trace, or group scope, with sampling.

Best for. Teams whose releases run on experiments, who want production logs to feed datasets and scorers.

Limits. Each score is metered, so scoring all production traffic gets expensive. Braintrust's own docs suggest sampling 1-10% for high-volume apps. Trace and group scorers wait for a 30-second idle timeout by default, which is short for chat where users pause between turns. Self-hosting is Enterprise only.

Pricing. Starter is free with 1 GB, 10,000 scores, and 14-day retention. Pro is $249 a month with 5 GB and 50,000 scores, 30-day retention, then $3 per GB and $1.50 per 1,000 scores. Users are unlimited.

For more, see Braintrust alternatives.

W&B Weave

What it is. Weights & Biases' toolkit for tracing and evaluating AI agents. The repo is Apache-2.0. It traces sessions, turns, LLM calls, and tool calls in an Agents view, and runs evaluations with LLM judges and custom scorers. It accepts OpenTelemetry on a dedicated endpoint, parsing both GenAI and OpenInference conventions. Signals score agent turns automatically.

Best for. ML teams that already use Weights & Biases for training and experiments.

Limits. Billing is by ingested data, so verbose traces cost more. Pro includes 1.5 GB a month, then $0.10 per MB. Signals only support turn scope. OpenInference spans show in the Traces view but not the Agents view, which needs GenAI conventions. W&B pricing now lives on CoreWeave's site.

Pricing. Free includes 1 GB a month of Weave ingestion. Pro starts at $60 a month with 1.5 GB of Weave ingestion and up to 10 model seats. Enterprise is custom.

New Relic AI monitoring

What it is. New Relic's APM for AI apps. Its APM agents capture traces, prompt and completion tokens, errors, and end-user feedback for supported models. The Python agent covers LangChain, LangGraph, CrewAI, Google ADK, AutoGen, Strands Agents, Microsoft Agent Framework, and MCP servers built with the MCP Python SDK or FastMCP. Go, Java, .NET, Node.js, and Ruby agents cover provider SDKs such as OpenAI and Bedrock.

Best for. Teams whose APM is New Relic and who want agent traces next to it, the same reason Datadog customers choose Datadog.

Limits. It is APM-first. Its AI monitoring docs center on performance, tokens, errors, and user feedback, and we found no documented LLM-as-a-judge evaluator setup like Datadog's. AI data isn't captured when distributed tracing is off or high security mode is on. Agent-framework and MCP coverage is Python-only.

Pricing. New Relic charges for data ingest and users. Every account gets 100 GB of ingest a month free and one full platform user, then $0.40 per GB. Full platform users are $10 for the first and $99 for each additional on Standard, or $349 a user per month on Pro with an annual commitment. The pricing page lists no separate charge for AI monitoring.

Helicone

What it is. An Apache-2.0 AI gateway with built-in logging, cost, and latency analytics.

Status. Mintlify acquired Helicone on March 3, 2026. Mintlify says Helicone "will continue operating in maintenance mode" with security updates, bug fixes, and new models, and that it will help every customer migrate to another platform. Helicone's site now says it "is now part of Experiential Labs," with details to come.

Our take. Don't pick it as a Datadog replacement for new work. If you already run it, plan a migration.

Pricing. Hobby is free for 10,000 requests. Pro is $79 a month and Team $799 a month, with usage-based pricing above that.

Agnost AI

What it is. Agent monitoring for AI agents in production: chat agents, voice agents (through transcripts), copilots, and MCP servers. It is OpenTelemetry-native and reads your production conversations and OTel traces. It works with the OpenAI Agents SDK, Vercel AI SDK, LangChain and LangGraph, Mastra, Pydantic AI, Google ADK, and Agno, and groups activity by user and session.

How it differs from Datadog. Agnost judges whole conversations, not single spans or traces. It detects failures, user frustration, intents, tool-call errors, and silent wrong answers. Its online evals are plain-language rules: once a conversation has been idle for 15 minutes, an LLM judges it in a batch and marks each assistant message pass or fail against every rule. There is no sampling, and conversations don't need to be marked complete. Alerts go to Slack or email. Auto Improve turns a detected failure into a pull request with a fix tested against past runs, with the replay results shown, for a person to review. You can also query Agnost from Claude or Codex through MCP, or ask it in Slack.

Best for. Teams whose failures only show up across a whole conversation, such as a wrong answer three turns in or a user who gives up. Use it next to Datadog, or instead of Datadog if agent conversations are all you need to monitor.

Limits. Agnost is not a replacement for Datadog's broader observability. It has no infrastructure APM and no cost tracking, so keep Datadog (or another APM) for latency, infrastructure, and spend. Verdicts are pass or fail, not numeric scores, and judging runs in batches rather than in real time. It is not open source.

Pricing. Free covers 1,000 events a month with 7-day retention. Starter is $49 a month (10,000 events). Pro is $499 a month (1,000,000 events). Enterprise is custom and includes self-hosted VPC deployment, audit logs, and SLAs. Events and Datadog's LLM spans are counted differently, so price your own traffic.

When to stay on Datadog

Stay on Datadog if:

  • Your services, infrastructure, and front end already run on Datadog. Linking an agent trace to the backend service, host, and user session on one trace ID is Datadog's strongest feature. Moving agent traces out breaks that.
  • Your volume is moderate. Only LLM spans are billed, evaluations are unlimited, and datasets, experiments, and the playground come with every tier. At a few hundred thousand LLM spans a month, Pro is a few hundred dollars.
  • You want one vendor and one contract. Security, procurement, and on-call already know Datadog.
  • Your judges fit spans and traces. Template evaluators already cover failure to answer, hallucination, prompt injection, toxicity, tool selection, and tool arguments.

If your gap is understanding what users experienced across whole conversations, you don't have to leave. Send the same OpenTelemetry traces to Datadog and to a conversation-level tool like Agnost AI.

How to choose

If you are looking past Datadog because… Pick
You want open source and free self-hosting Langfuse, or Arize Phoenix
You build on LangChain or LangGraph LangSmith
You want managed session-level evals Arize AX
Your releases run on experiments and datasets Braintrust
Your ML team already lives in Weights & Biases W&B Weave
Your APM is New Relic, not Datadog New Relic AI monitoring
You need longer retention without per-span add-ons Langfuse Core (90 days) or Pro (3 years)
Your failures only show up across whole conversations and you want them turned into PRs Agnost AI, next to Datadog
You correlate agents with Datadog APM, infrastructure, and RUM Stay on Datadog

Before you switch, three checks:

  1. Look at your instrumentation. If you emit OpenTelemetry with GenAI or OpenInference conventions, most tools above take it with an endpoint and header change. Our OpenTelemetry instrumentation guide shows the setup. If you use Datadog's ddtrace LLM Observability SDK, plan to replace those calls.
  2. Price your real volume. LLM spans, units, traces, spans, GB, MB, scores, and events don't convert one-to-one. The AI agent monitoring pricing guide has a worksheet.
  3. Run both for a week. Send the same traffic to Datadog and the candidate, and compare what each catches. The agent monitoring pilot checklist lists what to score.

Related reading

Frequently asked questions

What are the best Datadog LLM Observability alternatives?

Langfuse is the open-source pick you can self-host for free. LangSmith fits LangChain and LangGraph teams. Arize Phoenix is free to self-host, and Arize AX adds managed evals at span, trace and session level. Braintrust fits teams that run releases on experiments and scorers. W&B Weave fits teams already on Weights & Biases. New Relic AI monitoring fits teams whose APM is New Relic rather than Datadog. Agnost AI judges whole production conversations against plain-language rules and turns failures into pull requests, and can run next to Datadog. Helicone is in maintenance mode, so avoid it for new projects.

How much does Datadog LLM Observability cost?

Datadog bills only LLM spans, meaning calls to a model provider, not tool, retrieval or workflow spans. The free tier covers 40,000 LLM spans a month with 15-day trace retention. Pro starts at $160 a month on an annual commitment for 100,000 LLM spans ($200 month-to-month, $240 on-demand), then $3.50 per 10,000 extra LLM spans on annual billing. Longer retention is an add-on: 90 days costs $9 per 10,000 LLM spans a month.

Is there an open-source alternative to Datadog LLM Observability?

Yes. Langfuse is MIT-licensed except its ee folders and is free to self-host. W&B Weave's repository is Apache-2.0. Arize Phoenix is free to self-host with no feature gates under the Elastic License 2.0, which is source-available rather than OSI open source. Helicone is Apache-2.0 but has been in maintenance mode since its March 2026 acquisition by Mintlify.

Can I self-host Datadog LLM Observability?

Datadog's docs describe Agent Observability only on Datadog's own hosted sites and do not describe a self-hosted option. It is also not available on Datadog's US government sites. If you need self-hosting, Langfuse and Arize Phoenix are free to self-host, while LangSmith, Arize AX, Braintrust and Agnost AI offer self-hosting on their enterprise plans.

Do I need Datadog APM to use Datadog LLM Observability?

No. Datadog says Agent Observability, the current name for LLM Observability, is a standalone product you can buy without any other Datadog subscription. Correlation with APM, infrastructure and Real User Monitoring only works if you also run those products in Datadog.

Can I switch from Datadog LLM Observability without re-instrumenting?

If you send OpenTelemetry, mostly yes. Datadog ingests OTel spans that follow the GenAI semantic conventions or OpenInference, and Langfuse, LangSmith, Arize Phoenix, Braintrust, W&B Weave and Agnost AI all accept OTel too, so you change the exporter endpoint and headers. If you instrumented with Datadog's ddtrace SDK and its LLM Observability decorators, plan to replace those calls.


Originally published on agnost.ai.

Top comments (0)