The tools that analyze AI agent conversations to find user frustration and failure patterns are Agnost AI, LangSmith, Braintrust, Langfuse, Arize (Phoenix and AX), Amplitude Agent Analytics, Raindrop, PostHog, Opik, Datadog Agent Observability, and, for voice agents, Coval. Agnost AI is best for teams running their own chat or voice agent: it judges whole conversations, labels sentiment and frustration on each user message, tracks the failure patterns you define, and can open a pull request to fix a recurring one. Amplitude, Arize, Opik, and LangSmith ship ready-made frustration or friction judges, and Raindrop has a user-frustration preset. Arize AX, Braintrust, LangSmith, Raindrop, and Coval also group recurring failures automatically. Langfuse gives you the parts to build your own judge.
Frustration rarely shows up as angry words. It shows up as behavior: the user rephrases the same request, corrects the agent, asks for a person, or leaves mid-task. This guide compares the tools that detect those patterns in production chatbot and agent conversations: what each one analyzes, how it finds recurring failures, how it is priced, and when it is the better choice. We make Agnost AI, so it is listed first. Every other tool gets the same structure, and its limitations come from its own documentation, checked on November 17 and 18, 2026.
Quick comparison
| Tool | Best for | Frustration signal | Recurring failure patterns | Open source | Self-host | Pricing |
|---|---|---|---|---|---|---|
| Agnost AI | Own chat or voice agents; fixes as PRs | Sentiment and frustration labels per user message | Intents and rules you define, counted over time; alerts; PRs | No | Enterprise (VPC) | Free; $49, $499/mo |
| Amplitude Agent Analytics | Teams already on Amplitude | User Friction signal, High frustration tag | Topic clustering, topic spikes | No | No | Free; per agent session |
| PostHog | Teams already on PostHog | Sentiment labels, User correction judge | Task outcome judge, funnels | Yes (MIT core) | Yes (unsupported) | 100k events/mo free |
| Arize Phoenix and AX | Free self-hosted evals (Phoenix); automatic grouping (AX) | User Friction evaluator | Signal groups failures into issues (AX) | Source-available (ELv2) | Phoenix free; AX Enterprise | Phoenix free; AX Free, Pro $50/mo |
| Opik | Open-source conversation judges | UserFrustrationMetric | Not documented | Yes (Apache-2.0) | Yes, free | Free; Pro $19/mo |
| Braintrust | Eval-driven teams | Sentiment facet in Topics | Topics with an Issues facet | No | Enterprise | Free; Pro $249/mo |
| Datadog Agent Observability | Teams already on Datadog | Sentiment template; session judge you write | Session-level judges | No | No (SaaS) | 40k spans free; from $160/mo |
| LangSmith | LangChain and LangGraph teams | Perceived Error, User Satisfaction thread evaluators | Insights clusters failure modes | No | Enterprise add-on | Free seat; $39/seat/mo |
| Langfuse | Build your own judge, open source | Custom LLM-as-judge scores | Build your own | Yes (MIT core) | Yes, free | Free; Core $29/mo |
| Raindrop | Recurring failures tracked as rates | User frustration preset | Issue Detection groups recurring failures | Not stated | Not stated | Free (1k events); Pro $299/mo |
| Coval | Voice agents, testing and production | Audio Sentiment metric | Failure-mode categorization, anomaly detection | No | Enterprise (VPC) | From $100/mo |
Agnost AI
What it is. Conversation analytics for production AI agents. Agnost AI reads real chat and voice conversations to show what users wanted, where they got frustrated, and where the agent broke its own rules, then turns recurring failures into pull requests your team reviews.
Best for. Teams that want frustration tied to the exact turn and tool call that caused it, and want the recurring causes fixed rather than charted.
Key features.
- An LLM judge reads each whole conversation once it has been idle for 15 minutes, in batches. It labels sentiment and frustration on individual user messages, weighing tone, pacing, repetition, and whether the user got what they wanted, so polite repetition counts and an angry message the agent then resolved does not. - Failure patterns as intents: name the patterns you care about in plain language ("asks for a human", "repeats a request the agent already answered") and Agnost matches user messages against them and tracks volume over time. It also suggests intents from your product description and observed traffic. The judge is tuned for precision: an ambiguous moment, or a request the agent then handled without pushback, is not tagged.
- Agent-side causes as rules: checks such as "offer a handoff when the user asks for a person" give a pass or fail on each assistant message, with the supporting messages attached.
- Conversation timelines that keep turns, tool calls, and errors together, plus tool-call analytics for success rate, error distribution, and latency per tool.
- Threshold alerts on any intent or rule, sent to Slack or email. A fired alert can start Auto Improve, which investigates the finding and opens a pull request for a human to review.
Limitations. Not real time: conversations are judged in batches after 15 minutes of inactivity. No pre-launch simulation of conversations (Auto Improve does test its fixes against past runs). Voice calls are analyzed as transcripts only; there is no audio analysis of tone or pauses. Not open source; self-hosting is an Enterprise option. No built-in PII redaction before ingestion (data governance).
Pricing. Free (1,000 events a month, 7-day retention), Starter $49/month (10,000 events), Pro $499/month (1,000,000 events), Enterprise custom with self-hosted VPC.
Amplitude Agent Analytics
What it is. Amplitude's product analytics plus Agent Analytics, which observes what users ask an AI agent and ties agent quality to conversion, retention, and revenue.
Best for. Teams that already standardize on Amplitude and want frustration counted in the same funnels and cohorts as the rest of the product.
Key features. Seven built-in session signals, including User Friction, Task Completed, User Intent, and Has Negative Feedback. The User Friction evaluator sorts friction into categories such as Repeated Question and Confusion, and sessions can carry a High frustration tag. Topic clustering across sessions, and Eval Agents that flag a topic whose volume jumps or quality drops. Custom LLM and code-based evaluators run on all sessions with your own LLM key, at no extra Amplitude charge. OpenTelemetry GenAI spans flow in through a bridge.
Limitations. Sentiment isn't a named signal. Conversation text is kept 14 days on Free and Plus and 30 days on Growth and Enterprise, extendable to 90 days as a paid add-on. Several sections, including Evaluators, are still labeled beta.
Pricing. Free (2M events and 5,000 agent sessions a month). Plus starts at $0 with 5,000 agent sessions and can buy more. Growth (10,000 sessions) and Enterprise (20,000) are custom. Agent Analytics is billed on agent sessions.
PostHog
What it is. PostHog's AI Observability (formerly LLM analytics) captures every LLM call as a trace built on PostHog events, so each trace carries the user, their session replay, and any exceptions.
Best for. Teams already on PostHog who want frustration signals next to session replay and funnels without another vendor.
Key features. Positive, neutral, and negative labels on user messages from a local model, with no LLM cost, once you set up a sentiment evaluation. LLM-as-judge templates including User correction (the user corrects the assistant or repeats a request) and Task outcome, run per generation, trace, or session with sampling from 0.1% to 100%. Funnels and retention on the same data.
Limitations. The sentiment labels score tone, so polite repetition needs the judge templates. LLM-judge evals need your own provider key, and each evaluation run counts as an event. Full prompt and response text is deleted after 30 days. No automatic clustering of recurring failures is documented. Self-hosting is free but officially unsupported.
Pricing. First 100,000 AI events a month free, then from $0.00035 per event, falling with volume. No per-seat charges.
Arize Phoenix and Arize AX
What it is. Phoenix is Arize's open-source tracing and evaluation platform, built on OpenTelemetry and OpenInference. Arize AX is the hosted platform that adds online evals and automatic failure grouping.
Best for. Phoenix fits teams that need free, self-hosted frustration evals, including air-gapped environments. AX fits teams that want recurring failures grouped for them.
Key features. A pre-built User Friction evaluator that detects corrections, retries after an unsuccessful response, frustration, and challenges to unrequested actions, at span, trace, or session level. Sessions group a conversation into one thread. In AX, Signal scans your traces on a schedule and groups recurring failure patterns into issues, and online evals run on every tier.
Limitations. Phoenix is licensed under the Elastic License 2.0, which is source-available rather than OSI-approved. Phoenix does not evaluate incoming production traces automatically: its server evals run on dataset experiments, so friction is an evaluator you run and log back. Signal is an AX feature, capped at 10 issues a month on AX Free and 25 on AX Pro.
Pricing. Phoenix is free to self-host. AX Free is $0 (25k spans a month, 15-day retention), AX Pro is $50/month (50k spans, 30-day retention), and AX Enterprise is custom, with a self-hosted option.
Opik
What it is. Comet's open-source observability and evaluation platform for agents, with a library of LLM-judge metrics.
Best for. Teams that want Apache-2.0, self-hosted tracing with a ready-made conversation-level frustration judge.
Key features. UserFrustrationMetric, which scores from 0 to 1 how frustrated the user became over the interaction, plus SessionCompletenessQuality and other thread metrics. Online evaluation rules at trace, thread, and span scope with sampling and filters. Thread rules wait for a cooldown (15 minutes by default) so the conversation is complete before it is judged.
Limitations. Frustration is an evaluator you configure as a rule, and rules only run on data logged after you create them. Topic or failure clustering of conversations is not documented.
Pricing. Self-hosted open source is free. Cloud Free covers 25,000 spans a month; Pro is $19/month for 100,000 spans, then $5 per 100,000. Both cloud plans keep data for 60 days.
Braintrust
What it is. An evaluation-first platform where production traces become datasets and evals. Its Topics feature classifies production traffic automatically.
Best for. Eval-driven teams that want frustration and issue clusters to feed new test cases.
Key features. Topics includes a built-in Sentiment facet (for example positive, frustrated, or neutral) and an Issues facet (for example "Tool call failed" or "Incomplete answer"), next to a Task facet and custom facets. New logs are classified against existing topics as they arrive, and labels are written back to logs. Online scoring rules run scorers on production spans, traces, or whole sessions.
Limitations. Topics needs at least 100 summaries before it clusters, and the topics themselves regenerate daily. Topics usage draws down monthly model credits ($10 on Starter, $100 on Pro). Self-hosting is Enterprise-only.
Pricing. Starter free (1 GB processed data, 10k scores, 14-day retention). Pro $249/month (5 GB, 50k scores, 30-day retention). Billed on processed data and scores, not seats.
Datadog Agent Observability
What it is. Datadog's Agent Observability, formerly LLM Observability: traces, evaluations, and experiments for AI agents, sharing trace IDs with APM and RUM.
Best for. Teams that already run Datadog and want frustration evals next to backend root-cause analysis.
Key features. A Sentiment LLM-as-judge template (positive, negative, neutral), plus Goal Completeness and Failure to Answer templates. Session-level evaluations run once a session has had no new spans for 30 minutes, with a documented example judge prompt that labels sessions no_issues, mild_frustration, high_frustration, or abandoned. Evaluators run automatically on sampled production spans, traces, or sessions.
Limitations. There is no frustration template: you write and maintain the session-level judge prompt. No automatic grouping of recurring failures is documented. Default trace retention is 15 days, with 30, 60, and 90-day add-ons.
Pricing. Free up to 40,000 LLM spans a month with 15-day retention. Pro starts at $160/month billed annually ($200 month to month) for 100,000 LLM spans. Billed on LLM spans only.
LangSmith
What it is. LangChain's tracing and evaluation platform, with thread views for multi-turn conversations.
Best for. Teams on LangChain or LangGraph that want frustration judged at the conversation level without writing the judge first.
Key features. Built-in thread-level evaluator templates including Perceived Error, a true/false check of whether user signals such as frustration or repeated corrections show the user saw the agent make a mistake, and User Satisfaction. Thread evaluators run after a conversation goes idle (10 minutes by default). Insights clusters traces into usage patterns, common agent behaviors, and failure modes on a daily or weekly schedule. A Threads chat view answers questions about sentiment, pain points, and whether issues were resolved.
Limitations. Thread evaluators cover runs from the past 7 days, up to 500 threads per run. Insights is on Plus and Enterprise only, samples up to 1,000 traces per report, and has no built-in sentiment detector. Self-hosting is an Enterprise add-on.
Pricing. Developer $0 (1 seat, 5,000 base traces). Plus $39 per seat per month (10,000 base traces). Insights runs cost about $1 to $4 per 1,000 threads.
Langfuse
What it is. An MIT-licensed tracing and evaluation platform. It ships no frustration detector, but gives you the parts to build one.
Best for. Teams that want open-source, self-hosted tracing and will write and calibrate their own judge.
Key features. Sessions that group traces into a conversation, live LLM-as-judge evaluators on production observations, and scores you can filter and chart. Its product-team guide suggests writing labels back as categorical scores.
Limitations. You define, calibrate, and maintain the frustration judge yourself. Judges cannot run on sessions, because Langfuse does not know when a session has ended; the documented workaround is to judge the final trace or attach a custom score. Trace-level judges are deprecated and stopped producing results on Langfuse Cloud on November 16, 2026, so live judges now target individual observations.
Pricing. Self-hosting is free. Cloud Hobby is free (50k units a month, 30-day retention); Core is $29/month (100k units, 90-day retention).
Coval
What it is. A testing and evaluation platform for voice AI agents that also runs evals on live production conversations.
Best for. Voice-agent teams that want caller frustration read from the audio, alongside simulated calls before launch.
Key features. An Audio Sentiment metric that reads the caller's voice; Coval's docs note that a shift toward angry or sad is a common sign of mounting frustration. Transcript Sentiment, latency, interruption rate, and other acoustic metrics. Its Observe product lists failure-mode categorization (for example message truncation, off-script, escalation, and compliance gaps), anomaly detection, and one-click monitors. Alerts go to Slack, email, or a webhook.
Limitations. There is no dedicated frustration metric for production calls; its "Handle Frustrated Callers" solution is a set of simulated callers for testing. Production conversations arrive by upload through the API, CLI, or app, or as OpenTelemetry traces. Its docs list direct connections for LiveKit and Pipecat Cloud, not Vapi or Retell. Private or VPC deployment is Enterprise only.
Pricing. Starter $100/month (100 simulation minutes, 3,000 upload minutes), Growth $500/month, Enterprise from $4,500/month. A 7-day trial needs a credit card.
Raindrop
What it is. An AI observability platform for teams shipping agents that traces production runs and looks for failures that never throw an error, such as wrong answers, forgotten context, tool loops, and frustrated users.
Best for. Teams that want recurring agent failures grouped for them and tracked as rates, with Slack alerts.
Key features. Presets for common failures, including forgetting, user frustration, and task failure. Issue Detection groups recurring agent failures across runs into issues, with the original traces as evidence. Signals let you describe a behavior in plain language and track its rate as a metric across all events, and Signal Alerts post to Slack when a signal crosses a threshold.
Limitations. The full Stumbles view of single conversations where something went wrong is on Pro. On the free Hobby plan, ingestion pauses after 1,000 events a month.
Pricing. Hobby free (1,000 events a month). Pro $299/month plus metered events ($0.003 per event up to 1M, then $0.002), with a 14-day trial. Enterprise custom.
How to choose
| If your team… | Start with |
|---|---|
| Ships its own agent and wants frustration traced to the turn that caused it, with fixes delivered as PRs | Agnost AI |
| Already runs Amplitude and wants friction in its funnels and retention charts | Amplitude Agent Analytics |
| Already runs PostHog and wants frustration next to session replay | PostHog |
| Needs a free, self-hosted frustration evaluator | Arize Phoenix or Opik |
| Wants recurring failures grouped into issues automatically | Arize AX, LangSmith Insights, or Raindrop |
| Runs an eval-driven workflow and wants frustrated sessions to become test cases | Braintrust |
| Already runs Datadog APM | Datadog Agent Observability |
| Builds on LangChain or LangGraph | LangSmith |
| Already traces with Langfuse and has time to build and calibrate a judge | Langfuse |
| Runs a voice agent and needs frustration read from the caller's voice | Coval, with Agnost AI for transcript-level patterns and fixes |
Before you buy, run every candidate against the same labeled sample of your own conversations (see validation below). The tool that wins a vendor demo is not always the one that separates polite repeated attempts from difficult-but-successful tasks on your traffic. For voice agents specifically, see best voice agent monitoring tools.
What a frustration detector should catch
Sentiment alone is not enough. Users often stay polite while repeating a failed goal, and they can use negative language while the agent performs correctly. Start with interpretable signals rather than one opaque score:
| Signal | Example definition | Why it matters |
|---|---|---|
| Repetition | Similar user goal appears again before resolution | The previous answer did not move the task forward |
| Correction | "No," "that's not what I asked," or a factual correction | Explicit evidence the agent missed |
| Execution loop | Same tool with materially the same input repeats | Agent may be stuck rather than recovering |
| Outcome mismatch | Agent says complete, the system of record says failed or pending | High-severity false success |
| Handoff request | User asks for a person or a support email | User is routing around the agent |
| Abandonment | Session ends after a failure or repeated attempt | User stopped trying |
| Cross-session retry | User returns with the same unresolved goal | Failure persists beyond one conversation |
Keep the raw evidence behind each signal. Product teams need to know whether a flagged conversation came from one explicit correction or six weak indicators. For turning these into one metric, see the Frustration Index.
Repeated failed attempts
String equality misses paraphrases. "Reset my password," "I still can't log in," and "send the reset link again" may be the same goal. A workable sequence:
- Extract a normalized user goal from each turn.
- Compare it with unresolved goals earlier in the conversation.
- Require no confirmed success between attempts.
- Separate a justified retry (after a timeout) from a duplicate call.
- Join the agent's final claim to the authoritative outcome.
- Keep the supporting turn and tool IDs.
Duplicate tool calls are the execution-side version of the same problem; see how to detect and stop duplicate tool calls.
Validate a detector before you trust it
Build a labeled test set that includes direct anger, polite repeated attempts, difficult but successful tasks, negative topics handled correctly, justified retries after transient errors, false success, quick exits after correct answers, and short or multilingual messages.
Measure precision and recall per signal and per use case. Then measure alert precision: of the alerts your team reviewed, how many led to a confirmed issue? A classifier can label single conversations well and still produce noisy alerts if it ignores rate, volume, and clustering. Alert on a meaningful change, with a minimum sample:
alert when eligible >= N
and current_rate >= baseline_rate + absolute_delta
and current_rate / baseline_rate >= relative_multiplier
Tune N, the delta, and the multiplier on past incidents and healthy periods. Compare groups (plan, release, model, language) by rate with the denominator shown, so a 50% rate from two conversations does not outrank 12% across ten thousand.
From frustration to a fix
Rank confirmed issue clusters by affected users, frustration rate, severity, and recurrence, and keep those inputs visible. A rare false refund claim can outrank a frequent wording complaint because the severity is higher. Then route each issue by fix type:
- Prompt or policy: the agent had the evidence but reasoned or communicated badly.
- Retrieval: the right context was missing or stale.
- Tool or integration: execution failed or returned a business error.
- Product UX: the underlying flow is confusing, and the agent is explaining it faithfully.
- Missing capability: the user's goal cannot be completed yet.
Evaluation checklist
- Can every detected case open to its source conversation and trace?
- Does it detect semantic repetition, not only repeated strings?
- Can it tell recovery and justified retries from loops?
- Can it join the agent's claim to the authoritative outcome?
- Can you compare rates across cohorts and releases?
- Do alerts support baselines, sample floors, and cooldowns?
- Can confirmed cases become regression tests, and can you track recurrence after a fix ships?
- Are redaction, retention, access, and exports sufficient for your data?
- Is cost predictable at your peak trace volume?
Related reading
- Best conversation analytics tools for AI agents (2026), the wider comparison these tools come from
- What tool shows where users get stuck with my AI agent?
- Frustration Index: how to quantify user friction in a conversation
- Why thumbs down feedback is not enough for AI agents
- What Claude Code's source code shows about frustration detection
- Best voice agent monitoring tools (2026)
Frequently asked questions
Which tools analyze AI agent conversations to find user frustration and failure patterns?
Agnost AI, LangSmith, Braintrust, Langfuse, Arize (Phoenix and AX), Amplitude Agent Analytics, Raindrop, PostHog, Opik, Datadog Agent Observability, and, for voice agents, Coval. Agnost AI judges whole conversations, labels sentiment and frustration on each user message, tracks the failure patterns you define, and can open a pull request to fix a recurring one. Amplitude, Arize, Opik, and LangSmith ship ready-made frustration or friction judges, and Raindrop has a user-frustration preset. Arize AX, Braintrust, LangSmith, Raindrop, and Coval also group recurring failures automatically. Langfuse has no built-in frustration judge; you write one.
Which tools find recurring failure patterns in AI agent conversations automatically?
Arize AX Signal groups recurring failure patterns from your traces into issues on a schedule. LangSmith Insights clusters traces into usage patterns and failure modes on the Plus and Enterprise plans. Braintrust Topics has an Issues facet that classifies production logs. Raindrop's Issue Detection groups recurring agent failures across runs into issues with the original traces as evidence. Amplitude clusters sessions into topics. Coval categorizes voice-agent failure modes. Agnost AI counts the failure patterns you describe as intents and rules over time, alerts on them, and can turn a recurring one into a pull request.
Which tools analyze voice agent calls for caller frustration?
Coval and Agnost AI both work on voice. Coval scores production calls with metrics such as Audio Sentiment, which reads the caller's voice, and categorizes failure modes. Agnost AI ingests calls from Retell, Vapi, LiveKit, Pipecat, and other platforms, then judges the transcripts with the same frustration labels and rules it uses for chat. Agnost AI does not analyze audio, so tone of voice that never reaches the words is invisible to it.
Which tools actually catch user frustration in chatbot conversations?
Tools that read the whole conversation and look at behavior, not only tone. Amplitude has a User Friction signal, PostHog labels message sentiment and offers a user-correction judge, Arize Phoenix and Opik ship frustration evaluators, LangSmith has a Perceived Error thread evaluator, and Agnost AI labels frustration on each user message and tracks the patterns you define. Whichever you pick, test it on a labeled sample of your own conversations before trusting its numbers.
Is sentiment analysis enough to detect frustration?
No. Sentiment scores the tone of a single message, and frustrated users often stay polite while repeating a failed request or quietly leaving. Negative words also show up in conversations the agent handled well, such as a complaint about a late order that was fixed. Use sentiment as one input next to repetition, corrections, failed tool calls, and abandonment.
How can I track abandoned tasks in my chatbot?
Define when a task starts and which event proves it succeeded, then mark conversations that end without that success event after a set time. Join each abandoned task to its conversation and tool calls so you can see the turn where the user stopped. A quiet exit after a correct answer is not abandonment, so keep those as controls.
What should I use to prioritize product fixes based on frustration?
Rank confirmed issue clusters, not raw counts: combine affected users, the frustration rate among users who reached that step, severity, and how often the issue recurs. Keep those inputs visible so the team can see why one issue outranks another. Then route each issue to its fix type: prompt, retrieval, tool, product UX, or a missing capability.
Is there an open-source tool for detecting user frustration?
Yes. Opik (Apache-2.0) ships a UserFrustrationMetric for conversation threads, and Arize Phoenix (Elastic License 2.0) ships a User Friction evaluator; both are free to self-host. Langfuse (MIT core) has no built-in frustration metric, but you can write an LLM-as-judge evaluator for it and store the result as a score.
Originally published on agnost.ai.
Top comments (0)