DEV Community

Shubham Palriwala
Shubham Palriwala

Posted on Originally published at agnost.ai

Best tools to evaluate multi-turn conversations with LLM agents (2026)

The best tools to evaluate multi-turn conversations with LLM agents are DeepEval (with Confident AI), Opik, LangSmith, Promptfoo, Ragas, Braintrust, Arize AX, and Langfuse for offline and pre-release testing, and Agnost AI for judging whole production conversations. DeepEval and Opik are the most complete open-source options: both score whole conversations with built-in conversational metrics and both can simulate users. LangSmith does both halves well if you're on LangChain. Braintrust, Arize AX, and Opik also judge grouped production sessions after an idle timeout. Agnost AI covers only the production half: it judges every real conversation once it goes idle, with no sampling, but it has no datasets or simulated users.

Updated for 2026. A lot has changed since 2025: most eval platforms now have a whole-conversation (thread, group, or session) evaluator for production traffic, simulated users moved into the main frameworks, Langfuse deprecated trace-level evaluators, and Promptfoo became part of OpenAI.

Disclosure: We make Agnost AI. Every fact about another tool comes from that vendor's own docs, blog, or pricing page, checked on November 18, 2026. This is a documentation comparison, not a hands-on benchmark.

Why single-turn and trace-level evals miss conversation failures

A trace-level eval asks: was this one answer good? Many conversation failures look fine one answer at a time:

  • Forgetting. The user gave their order number in turn 2. In turn 6 the agent asks for it again. Each reply is polite and relevant.
  • Looping. The user rephrases the same request four times and gets four different, plausible, wrong answers.
  • Drift. The agent slowly leaves its role or policy over a long exchange. No single message is clearly wrong.
  • False success. A tool call fails in turn 3, and in turn 5 the agent says "done."
  • Abandonment. The user stops replying after an answer that looked correct to a judge but didn't solve their problem.

To catch these, the judge needs the whole transcript, and the tool needs to know which traces belong to one conversation. That means a shared ID on every turn (session_id, thread_id, or session.id, depending on the tool) and a rule for when the conversation is finished. For more detail on choosing the unit, see conversation-level vs trace-level online evals.

There are two halves to multi-turn evaluation:

  • Offline. You build conversations from a dataset, or let a simulated user talk to your agent, then score them before release. This is where you catch regressions and test new prompts.
  • Online. Real users create the conversations. The tool waits until a conversation goes idle, then judges it. This is where you find the failures nobody thought to test. What are online evals? covers the basics, and LLM evals in CI vs production covers which failures each half catches.

Quick comparison

Tool Offline multi-turn Online (production) Session-level scoring Simulated users Open source Pricing
DeepEval / Confident AI Yes: ConversationalTestCase Confident AI thread rules, 300 s idle default Knowledge Retention, Role Adherence, Conversation Completeness, more ConversationSimulator Apache-2.0 (DeepEval) Free; Confident AI Starter $200/mo
Ragas Yes: MultiTurnSample No AspectCritic, agent goal and topic metrics Not documented Apache-2.0 Free
Promptfoo Yes: conversation tests No Per-test assertions; no session metric documented promptfoo:simulated-user MIT Free; Enterprise custom
LangSmith Yes: simulation inside evaluate() Thread evaluators, 10 min idle default Yes, whole thread openevals No Free seat; Plus $39/seat/mo
Braintrust Yes: experiments on chat datasets Group scope, 30 s idle default Yes, up to 64 traces per group Not documented No Free; Pro $249/mo
Opik Yes: evaluate_threads Thread rules, 15 min cooldown default 12 thread metrics incl. UserFrustrationMetric SimulatedUser Apache-2.0 Free; Pro $19/mo
Arize AX Historical runs over logged sessions Session evals over session.id Yes, {conversation} transcript Not documented No (Phoenix is ELv2) Free; Pro $50/mo
Langfuse N+1 datasets, experiments Observation-level only Manual or API session scores Via external OpenEvals MIT (except ee) Free; Core $29/mo
Agnost AI No Every conversation, 15 min idle, no sampling Pass/fail per assistant message, judged with full context No No Free; $49, $499/mo

DeepEval and Confident AI

What it is. DeepEval is an Apache-2.0 Python eval framework. Confident AI is the hosted platform from the same team.

Multi-turn support. A ConversationalTestCase holds a list of Turn objects (role user or assistant, content, and optional tools_called and retrieval_context on assistant turns), plus optional scenario, expected_outcome, expectations, and chatbot_role. Built-in multi-turn metrics include Knowledge Retention, Role Adherence, Conversation Completeness, Conversation Relevancy, and Turn Relevancy. ConversationalGEval lets you write your own. Conversation Completeness, for example, extracts the user's intentions and scores the share the assistant satisfied.

Simulated users. The ConversationSimulator uses an LLM to role-play the user from a ConversationalGolden (scenario, expected outcome, and persona). It calls your agent through a model_callback until it reaches the expected outcome or max_user_simulations (default 10), then returns ConversationalTestCases you can score.

Production. On Confident AI, a Thread Evaluation Rule runs multi-turn metrics once a thread has been idle for 300 seconds by default. You can also trigger it manually with evaluate_thread. The docs don't describe a sampling rate for threads.

Pricing. DeepEval is free. Confident AI Free is $0 for 2 seats. Starter is $200 a month per organization and is the first plan with chat simulations and online evals. Team is $2,000 a month.

Ragas

What it is. An Apache-2.0 Python library for LLM evaluation.

Multi-turn support. A conversation is a MultiTurnSample of HumanMessage and AIMessage objects. AspectCritic judges it against a plain-language definition and returns 1 or 0, so you can write checks like "the AI completes every request without the user having to ask again." Agent metrics include TopicAdherence, ToolCallAccuracy, ToolCallF1, and AgentGoalAccuracy.

Limits. It is a library, not a platform. There is no production monitoring and no documented user simulator. The legacy metric API that takes MultiTurnSample is marked deprecated and "will be removed in v1.0," so check the current metric API before building on it.

Pricing. Free.

Promptfoo

What it is. An MIT-licensed CLI and library for evals and red teaming, run on your machine or in CI. Promptfoo is now part of OpenAI and says it stays open source and MIT-licensed.

Multi-turn support. The built-in _conversation variable carries earlier turns into each prompt. Tests that share a conversationId in their metadata share one history. Using _conversation forces single-threaded runs.

Simulated users. The promptfoo:simulated-user provider, inspired by Tau-bench, plays a user from your instructions template and talks to your agent for up to maxTurns (default 10) new turns, or until it emits ###STOP###.

Limits. No production monitoring.

Pricing. Community is free. Enterprise is custom.

LangSmith

What it is. LangChain's tracing and eval platform. It works with or without LangChain.

Offline. LangChain's open-source openevals package has create_llm_simulated_user and run_multiturn_simulation. You run the simulation inside your evaluate() target and score the returned trajectory with an LLM-as-judge evaluator. Each dataset example can carry its own simulated-user prompt.

Online. Traces that share a session_id or thread_id metadata key form a thread. Multi-turn online evaluators run once per completed thread, after an idle period that defaults to 10 minutes (minimum 2). You choose what the judge sees: all messages, only human and AI messages, the first human and last AI message, or the full trajectory with tool calls.

Limits. At most 10 multi-turn online evaluators per workspace and 500 threads per run. Only runs from the past 7 days are eligible.

Pricing. Developer is free for 1 seat and 5,000 base traces. Plus is $39 per seat per month with 10,000 base traces. Base traces are kept 14 days.

Braintrust

What it is. An eval and observability platform built around experiments, datasets, and scorers.

Offline. You run experiments on datasets whose rows carry chat history and the next user input, and score the result. Braintrust's own guide to multi-turn tools doesn't describe a built-in user simulator, so you bring your own script or simulator.

Online. Group-scope scoring rules score related traces together, grouped by a field such as metadata.session_id. Scoring fires after an idle timeout that defaults to 30 seconds, because there is no separate "conversation ended" signal. Groups look back 24 hours by default and are truncated to the most recent 64 traces. Sampling decides whether a new trace triggers grouped scoring, but doesn't drop turns from the history the scorer sees.

Watch out for. A 30-second idle default will score many chat conversations before the user has finished. Raise it to match how long your users pause.

Pricing. Starter is free (1 GB, 10,000 scores, 14 days). Pro is $249 a month (5 GB, 50,000 scores, 30 days). All plans have unlimited users.

Opik

What it is. Comet's Apache-2.0 eval and tracing platform, free to self-host.

Offline. SimulatedUser plays a persona, and run_simulation drives a conversation with your agent for a set number of turns (max_turns defaults to 5). evaluate_threads scores existing threads. There are 12 conversation metrics, including KnowledgeRetentionMetric, ConversationDegenerationMetric, ConversationalCoherenceMetric, SessionCompletenessQuality, and UserFrustrationMetric.

Online. Thread evaluators wait for a cooldown after the last activity, 15 minutes by default, so the judge sees the whole conversation. Built-in thread templates are Task completion, Conversational coherence, and User frustration. You set a sampling rate per rule.

Pricing. Open source is free. Cloud Free covers 25,000 spans a month. Pro is $19 a month for 100,000 spans.

Arize AX and Phoenix

What it is. Arize AX is a managed eval and observability platform. Phoenix is its free, self-hostable workbench under the Elastic License 2.0.

Session evals. In AX, a session eval scores every trace that shares a session.id. The judge reads a {conversation} transcript built from each turn's input and output, or a custom {turn_data} layout. Sessions exist only if your app sets session.id on its spans. Eval tasks can run continuously on new data or once over historical data, and sampling is applied per session.

Limits. We didn't find a built-in user simulator in Arize's docs, and the pages we checked don't confirm session-scoped evaluators in Phoenix.

Pricing. Phoenix is free. AX Free covers 25,000 spans a month. AX Pro is $50 a month for 50,000 spans. Self-hosted AX is Enterprise.

Langfuse

What it is. An open-source (MIT, except ee folders) tracing, prompt, and eval platform, free to self-host.

Multi-turn support. A sessionId groups traces into a session you can replay as one conversation. You can add session scores by hand in the UI or through the SDK and API. LLM-as-a-judge evaluators run on observations and experiments, not sessions. Trace-level evaluators are deprecated in Langfuse v4 and stopped producing results on Langfuse Cloud after November 16, 2026.

Langfuse's own advice. For multi-turn evals, Langfuse recommends N+1 datasets (take a real conversation up to turn N and test turn N+1) and simulated conversations run with an external framework such as OpenEvals, scored with simple pass/fail judges.

Pricing. Self-hosting is free. Cloud Hobby is free for 50,000 units. Core is $29 a month.

Agnost AI

What it is. Agent monitoring for production chat and voice agents, copilots, and MCP servers. It is OpenTelemetry-native and has Python and TypeScript SDKs.

Multi-turn support. Agnost groups traffic by user and session and judges whole conversations, not single traces. You write online eval rules in plain language ("confirm the booking before saying it's done"). Once a conversation has been idle for 15 minutes, an LLM judge reads the full conversation in a batch and marks each assistant message pass or fail against every active rule. There is no sampling: every conversation is judged, up to your plan's event limit. It also detects user frustration, intents, tool-call errors, loops, and silent wrong answers. Alerts go to Slack or email. With Auto Improve connected, an alert opens a pull request with a fix, tested by replaying the affected past conversations, for a person to review.

Limits. Agnost covers only the production half. It has no offline datasets or experiments, no simulated users, and no CI gating. Verdicts are pass or fail, not scores. Judging runs in batches after 15 minutes of inactivity, not in real time. Most teams pair it with one of the offline tools above and turn production failures into regression cases there.

Pricing. Free (1,000 events a month, 7-day retention), Starter $49 a month (10,000 events), Pro $499 a month (1,000,000 events), Enterprise custom with self-hosted VPC.

How to choose

If you need… Start with
Open-source multi-turn tests in Python, with simulated users DeepEval or Opik
A test runner in CI with a simulated user, no Python Promptfoo
A lightweight library to score existing transcripts Ragas
Offline simulation plus production thread evals, on LangChain LangSmith
Experiments on chat datasets, plus grouped production scoring Braintrust
Managed session evals at scale Arize AX
Open-source tracing with sessions, and you'll build the judge Langfuse
Every production conversation judged as a whole, with no sampling, and failures turned into PRs Agnost AI, next to an offline tool

Three questions settle most choices:

  1. Do you need to test before release, watch production, or both? Most teams need both. Pick one offline tool and one production tool. They can be the same vendor, but they don't have to be.
  2. How do you mark the end of a conversation? Every online tool here uses an idle timeout: 30 seconds (Braintrust), 300 seconds (Confident AI), 10 minutes (LangSmith), 15 minutes (Opik and Agnost AI). Pick one that fits how long your users actually pause, or a conversation gets judged in pieces.
  3. Can you afford to judge everything? Judging whole conversations with an LLM costs more than judging single spans. Most tools let you sample. If rare failures matter, check what you lose at 10%.

For the wider landscape, including CI gates and single-turn scorers, see best AI agent evaluation tools.

Related reading

Frequently asked questions

What are the best tools to evaluate multi-turn conversations with LLM agents?

For offline tests before release, DeepEval (ConversationalTestCase, conversational metrics and a ConversationSimulator), Opik (thread metrics plus SimulatedUser), LangSmith (openevals multi-turn simulation inside evaluate()), and Promptfoo (conversation tests and a simulated-user provider) are the most complete. Ragas scores MultiTurnSample conversations with AspectCritic and agent metrics. For whole-conversation judging in production, LangSmith thread evaluators, Opik thread rules, Braintrust group-scope scoring, Arize AX session evals, Confident AI thread evaluation and Agnost AI all wait for a conversation to go idle and then judge it as a whole. Agnost AI covers only the production half: it judges every conversation with no sampling, but has no datasets or simulation.

How do I evaluate a multi-turn conversation instead of a single LLM response?

Group every turn under one conversation ID (a session_id, thread_id or session.id, depending on the tool), wait until the conversation is finished or idle, then give an LLM judge the full transcript with a question about the whole exchange: did the user's goal get met, did the agent remember earlier facts, did it stay in role. Offline, you build the conversation from a dataset or a simulated user. In production, you let real users create it and judge it after an idle timeout.

What is a DeepEval ConversationalTestCase?

It is DeepEval's test case for a whole conversation. It holds a list of Turn objects (role user or assistant, content, and optional tools_called and retrieval_context on assistant turns), plus optional scenario, expected_outcome, expectations and chatbot_role fields. You score it with multi-turn metrics such as Knowledge Retention, Role Adherence, Conversation Completeness, Conversation Relevancy, Turn Relevancy, or a custom ConversationalGEval, and DeepEval's ConversationSimulator can generate the turns from a ConversationalGolden's scenario and persona.

Which tools can simulate users for multi-turn agent testing?

DeepEval's ConversationSimulator (also sold as chat simulations on Confident AI Starter and above), Opik's SimulatedUser and run_simulation, LangChain's open-source openevals package (create_llm_simulated_user and run_multiturn_simulation, which you can run inside LangSmith evaluate()), and Promptfoo's promptfoo:simulated-user provider. Langfuse recommends using an external framework such as OpenEvals. We could not find a documented built-in user simulator for Braintrust, Arize AX, Ragas or Agnost AI.

What is the difference between session-level and trace-level evaluation?

A trace-level eval judges one request: one user message and the agent's work to answer it. A session-level (or thread-level) eval judges every trace that shares a conversation ID as one transcript. Failures such as the agent forgetting what the user said three turns ago, the user rephrasing the same request four times, or the agent claiming success after a failed tool call are only visible at the session level.

Can I evaluate multi-turn conversations in production without sampling?

Most tools let you set sampling to 100%, but they default to or recommend sampling for cost: Opik, Braintrust, Arize AX and PostHog all expose a sampling rate, and LangSmith thread evaluators process at most 500 threads per run. Agnost AI has no sampling setting: once a conversation has been idle for 15 minutes it judges the whole conversation against every active rule, up to your plan's event limit.

Does Langfuse support multi-turn or session-level evals?

Partly. Langfuse groups traces into sessions with a sessionId and lets you add session scores in the UI or through the SDK and API, but its LLM-as-a-judge evaluators target observations and experiments, not sessions. Trace-level evaluators are deprecated in Langfuse v4 and stopped producing results on Langfuse Cloud after November 16, 2026. For multi-turn evals, Langfuse recommends N+1 datasets built from real conversations and simulated conversations orchestrated with an external framework such as OpenEvals.

What changed in multi-turn evaluation tools since 2025?

Whole-conversation judging in production became standard: LangSmith thread evaluators, Opik thread rules, Braintrust group scope, Arize AX session evals and Confident AI thread evaluation rules all wait for a conversation to go idle and then judge it. Simulated users moved into the main tools (Opik SimulatedUser, DeepEval ConversationSimulator, Promptfoo simulated-user, openevals). Langfuse deprecated trace-level evaluators in v4, Ragas marked its legacy MultiTurnSample metric API for removal in v1.0, and Promptfoo became part of OpenAI while staying MIT-licensed.


Originally published on agnost.ai.

Top comments (0)