DEV Community

Shubham Palriwala
Shubham Palriwala

Posted on Originally published at agnost.ai

Best tools to analyze user sentiment and frustration in AI chatbot conversations (2026)

The tools that analyze user sentiment and frustration in AI chatbot conversations are Agnost AI, PostHog, Amplitude Agent Analytics, Raindrop, Nebuly, and Dimension Labs for agents you build yourself, Intercom and Zendesk for support bots inside those helpdesks, and Google Cloud Natural Language, Amazon Comprehend, and Azure AI Language as general text APIs. Agnost AI judges whole conversations and tags the moments users get frustrated, tied to the turn and tool call that caused it. PostHog gives you cheap per-message positive, neutral, and negative labels from a local model. Amplitude scores every session on built-in signals including User Friction. The NLP APIs score text, not conversations: they can't tell a sarcastic "great, thanks" from a real one, and they don't know whether the user's task succeeded.

Updated for 2026. Compared with 2025-era sentiment guides, this one covers the agent-specific signals now in product analytics tools, conversation-level helpdesk scores, and the retirement date Microsoft has set for Azure's sentiment feature.

Disclosure: We make Agnost AI. Every fact about another tool comes from that vendor's own docs or pricing page, checked on November 18, 2026. This is a documentation comparison, not a hands-on benchmark.

If you want the tools that find frustration and recurring failure patterns (including LangSmith, Braintrust, Arize, Opik, and Coval for voice), see best tools to detect user frustration in chatbots. This page is about sentiment: what it tells you about a bot, where it misleads, and which tools measure it.

Why sentiment scores alone mislead for chatbots

Sentiment was built for reviews and tweets: one block of text, one opinion. Chatbot conversations break that model in three ways.

  • Polite users who failed. "Thanks, I'll try that." Then the user leaves and calls support. Every message scored neutral or positive, and the bot failed. This is the most common failure, and per-message sentiment can't see it.
  • Sarcasm. "Great, really helpful" after the third wrong answer reads as positive to a text classifier. You only know it's sarcastic from the turns before it.
  • Short replies. "ok", "no", "fine", "?" carry almost no tone. In a chat, most user messages are short.

The reverse mistake happens too. A user who writes "my order is late again and I'm furious" is negative, but if the bot fixes it, that's a success. Averaging message sentiment across a conversation mixes the user's problem with the bot's performance.

Sentiment vs frustration vs task failure

Keep three signals separate, because they disagree often:

Signal What it measures Where it comes from What it misses
Sentiment Tone of a message or conversation Classifier or LLM on user text Polite failure, sarcasm, short replies
Frustration Friction across turns: repeats, corrections, asking for a person, shrinking replies Behavior across several turns Users who fail quietly in one turn and leave
Task failure Whether the user got what they came for Outcome: tool results, system of record, or a judge reading the whole conversation How the user felt about it

Frustration is the most useful of the three for finding bugs, and sentiment is the cheapest. Outcome is what matters to the business. For a full method to score frustration from behavior, see the Frustration Index.

Quick comparison

Tool Best for What it reads Sentiment output Frustration signal Ties to outcome Pricing
Agnost AI Agents you build; fixes as PRs Whole conversation, after 15 min idle Frustration and other signals tagged on user messages; no numeric score Yes, judged with the full conversation Rules judged pass/fail per assistant message Free; $49, $499/mo
PostHog Teams already on PostHog Each user message, local model Positive, neutral, negative User correction judge (separate) Task outcome judge 100K AI events/mo free
Amplitude Agent Analytics Teams already on Amplitude Every closed session Not a listed signal User Friction signal Task Completed signal Free plan includes 5,000 agent sessions
Raindrop Recurring failures tracked as rates Production events Not described User-frustration preset Task-failure preset Free (1K events); Pro $299/mo
Nebuly Enterprise copilots and customer agents, self-hosted All conversations Sentiment and emotion Friction and dissatisfaction Failures, churn risk Not public
Dimension Labs Joining conversation language to revenue Conversations, reviews, surveys, calls Many axes beyond sentiment Not named (effort, churn risk) Joins to revenue and renewals From $299/mo
Intercom CX Score Fin support conversations Closed chat and email conversations 1 to 5 rating with explanation Customer effort, strong emotion Answer quality and resolution Pro add-on
Zendesk intelligent triage Zendesk tickets and messaging First message, or whole conversation on messaging Very Positive to Very Negative No No Suite and Support Professional and above
Google Cloud Natural Language Scoring text in your own pipeline Any text Score -1 to 1, plus magnitude No No 5,000 units/mo free
Amazon Comprehend Scoring text in your own pipeline Any text Positive, Negative, Neutral, Mixed No No 50K units/mo free for 12 months
Azure AI Language Scoring text in your own pipeline Any text Positive, neutral, negative per sentence and document No No 5,000 records/mo free; retires 2029

Agnost AI

What it is. Agent monitoring for production chat and voice agents, copilots, and MCP servers. It groups traffic by user and session and judges whole conversations, not single traces. Voice agents are analyzed from transcripts, not audio.

How it handles sentiment. Agnost doesn't put a positive or negative score on each message. Instead, once a conversation has been idle for 15 minutes, an LLM judge reads the whole thing and tags user messages with the signals you care about, such as frustration, confusion, or asking for a human. You describe these signals in plain language, and Agnost also suggests them from your product description and traffic. The judge is told to weigh the user's emotional arc (frustration building, confusion turning into satisfaction, polite disappointment, sarcasm), how they write, whether they repeat themselves, and whether they got what they wanted. A request the agent handled, with no pushback from the user, isn't tagged.

Tying it to outcome. Plain-language rules ("confirm the refund was issued before saying it was") are judged pass or fail on each assistant message, so a frustrated moment sits next to the agent failure that caused it, along with tool-call errors and loops. Alerts on any signal or rule go to Slack or email. With Auto Improve connected, an alert opens a pull request with a fix, tested by replaying the affected past conversations, for a person to review.

Limits. No numeric sentiment score to chart. Not real time: judging runs in batches after 15 minutes of inactivity. No audio analysis of tone. Not open source, and self-hosting is an Enterprise option.

Pricing. Free (1,000 events a month, 7-day retention), Starter $49 a month (10,000 events), Pro $499 a month (1,000,000 events), Enterprise custom.

PostHog

What it is. PostHog's AI Observability captures LLM calls as PostHog events, next to session replay, funnels, and retention.

How it handles sentiment. A sentiment evaluation labels each user message positive, neutral, or negative with a local model (cardiffnlp/twitter-roberta-base-sentiment-latest). PostHog says no data goes to third parties. A generation's or trace's sentiment is the average of its messages. Only user input is scored, and only on sampled generations.

Frustration and outcome. Separate LLM-as-judge templates include User correction and Task outcome, which can run per generation, trace, or whole session (session needs $ai_session_id), with sampling from 0.1% to 100%.

Limits. The sentiment model is a small classifier that scores one message at a time, so it has the polite-failure and sarcasm problems above. Use the judge templates for those.

Pricing. The first 100,000 AI events a month are free, then from $0.00035 per event, falling with volume.

Amplitude Agent Analytics

What it is. Amplitude's agent analytics, inside its product analytics, so agent quality sits next to conversion and retention.

How it handles sentiment. Sentiment isn't one of the built-in signals. Every closed session is scored on seven: Task Completed, Response Quality, Session Safety, User Intent, User Friction, Has Negative Feedback, and Data Quality Issues. Sessions can carry tags such as High frustration, and human review uses friction labels like Repeated question and Confusion. You can add custom LLM evaluators on your own model key.

Limits. Amplitude itself says signals are "directional indicators, not ground truth." If you need a sentiment label specifically, you write a custom evaluator.

Pricing. Agent Analytics is on every plan. Free and Plus include 5,000 agent sessions a month, Growth 10,000, and Enterprise 20,000. It is billed on agent sessions, and custom evals carry no Amplitude charge.

Raindrop

What it is. An AI observability platform that looks for failures that never throw an error, such as wrong answers, forgotten context, tool loops, and frustrated users.

How it handles sentiment. There's no separate sentiment feature on its site. It has presets including user frustration and task failure, and Signals let you describe a behavior in plain language and track its rate. Issue Detection groups recurring failures into issues with the traces as evidence.

Pricing. Hobby is free for 1,000 events a month, then ingestion pauses. Pro is $299 a month plus $0.003 per event up to 1 million.

Nebuly

What it is. User analytics for AI copilots and customer-facing agents, which can run as SaaS or inside your own AWS, Azure, or GCP account, or on premises.

How it handles sentiment. Nebuly's Experience product identifies "intents, topics, sentiment and emotion" and flags friction, dissatisfaction, and churn risk across all conversations, using implicit signals because, in Nebuly's words, fewer than 1% of interactions get explicit feedback.

Limits. Pricing isn't public. You book a demo.

Dimension Labs

What it is. An analytics agent for customer language across conversations, reviews, surveys, and calls, with an AI agent analytics solution and integrations for bot platforms such as Ada, Kore.ai, Cognigy, Amazon Lex, and OpenAI.

How it handles sentiment. Dimension Labs argues that "sentiment scores a conversation on one axis," and instead extracts many structured fields (it calls them Dimensions), such as effort, churn risk, and whether a promise was kept, then joins them to revenue and renewals.

Best for. CX and revenue teams that want to know which conversation patterns cost money, across bots and humans.

Pricing. From $299 a month, with no contract on monthly plans.

Intercom CX Score

What it is. A 1 to 5 rating Intercom puts on closed chat and email conversations, including those handled by its Fin AI agent.

How it handles sentiment. The rating comes from six reasons, including answer quality, customer effort (repetition and handovers), and strong emotion (joy, gratitude, frustration, or anger). Each rated conversation gets a written explanation. A conversation needs at least two customer and two agent replies to be rated.

Limits. It requires the Pro add-on and Fin. It only works on conversations inside Intercom, so it doesn't help with an agent you built yourself.

Zendesk intelligent triage

What it is. Zendesk's AI classification of tickets by intent, sentiment, and language.

How it handles sentiment. Sentiment runs from Very Positive to Very Negative, calibrated for support, so an issue alone doesn't make a ticket negative. By default it reads the subject and first message. With dynamic classification on, messaging channels are re-evaluated on the entire conversation. Zendesk's page doesn't mention AI agent conversations.

Limits. Suite and Support Professional plans and above. Using the classifications in workflows needs the Copilot add-on.

Google Cloud Natural Language, Amazon Comprehend, and Azure AI Language

What they are. General-purpose text APIs. You send text, you get a sentiment back.

  • Google Cloud Natural Language returns a score from -1.0 to 1.0 and a magnitude (how much emotion) for the document and each sentence. Billing is per 1,000-character unit, and the first 5,000 units a month are free.
  • Amazon Comprehend returns Positive, Negative, Neutral, or Mixed with a confidence for each. It bills in 100-character units with a 3-unit minimum per request, with 50,000 free units a month for the first 12 months.
  • Azure AI Language returns positive, neutral, or negative with confidence scores per sentence and document, plus opinion mining for aspects. It can run in a Docker container. Microsoft says sentiment analysis retires on March 31, 2029, and points new projects to Foundry models.

The catch. These APIs score text, not conversations. They don't know what the bot said before the user's message, whether the user is repeating themselves, or whether the task succeeded. To use them for a chatbot you build the rest: send context with each message, detect repetition yourself, and join results to outcomes.

How to choose

If you… Start with
Built your own agent and want frustration tied to the turn and tool call that caused it, with fixes as PRs Agnost AI
Run PostHog and want cheap per-message sentiment with no LLM cost PostHog
Run Amplitude and want friction in the same funnels as the rest of your product Amplitude Agent Analytics
Want recurring failures grouped and tracked as rates Raindrop
Need analytics for internal copilots that runs on your own cloud Nebuly
Want conversation language joined to revenue and churn, across bots and humans Dimension Labs
Run your support bot on Intercom Fin or Zendesk Their built-in scores first
Need a sentiment number inside your own data pipeline Google, AWS, or Azure APIs, with conversation context added

Whatever you pick, test it on your own conversations. Label 100 of them by hand, including polite users who failed, sarcastic replies, and angry users the bot helped, and check how often the tool agrees with you.

How to combine sentiment with outcome

A sentiment chart alone tells you how users sound. Joined to outcome, it tells you what to fix. Put every conversation in one of four cells:

Task succeeded Task failed
Ended positive or neutral Working as intended Polite failure: the group sentiment hides
Ended negative Hard problem, handled Fix first
  1. Get an outcome per conversation. Use a success event from your system (refund issued, booking created), or a rule judged on the whole conversation ("the user's request was completed").
  2. Use end-of-conversation sentiment, not the average. How the user felt at the end matters more than how they started.
  3. Watch the polite-failure cell. It's usually bigger than the fix-first cell, and no sentiment dashboard will show it to you.
  4. Break it down by intent and release. A spike in failed conversations for one intent after a prompt change is a bug report.
  5. Read the conversations. Every cell should open to the transcripts behind it.

Related reading

Frequently asked questions

What tools can identify user sentiment and frustration in conversations with my AI agent?

For an AI agent you build yourself: Agnost AI judges whole conversations and tags user messages with frustration and other signals you define in plain language; PostHog labels each user message positive, neutral or negative with a local model; Amplitude Agent Analytics scores every session on signals including User Friction; Raindrop has a user-frustration preset and groups recurring failures; Nebuly identifies sentiment, emotion and friction in AI conversations. For support bots on a helpdesk, Intercom's CX Score and Zendesk intelligent triage rate sentiment inside those products. Google Cloud Natural Language, Amazon Comprehend and Azure AI Language return sentiment scores for any text, but they score text, not conversations.

Is sentiment analysis enough to tell whether my chatbot is failing users?

No. Sentiment scores tone, and many failed users stay polite: they rephrase the same request, say thanks, and leave. Sarcasm ('great, thanks a lot') reads as positive, and short replies like 'ok' or 'fine' carry almost no tone at all. A user can also be negative about their problem while the bot handles it well. Pair sentiment with frustration behavior (repetition, corrections, requests for a human) and with the outcome of the conversation.

What is the difference between sentiment and frustration in chatbot analytics?

Sentiment is the tone of a message: positive, neutral or negative. Frustration is a pattern across turns: the user repeating a request, correcting the bot, asking for a person, or getting shorter and shorter. Task failure is a third thing: whether the user got what they came for. A good dashboard keeps all three separate, because they disagree often.

Can I use Google Cloud Natural Language, AWS Comprehend or Azure AI Language for chatbot sentiment?

Yes, for scoring individual messages. Google returns a score from -1.0 to 1.0 plus a magnitude per document and per sentence. Amazon Comprehend returns Positive, Negative, Neutral or Mixed with confidence scores. Azure AI Language returns positive, neutral or negative labels with confidence scores at sentence and document level, and its sentiment feature retires on March 31, 2029. None of them knows what the bot said before, whether the user repeated themselves, or whether the task succeeded, so you must build conversation context and outcomes yourself.

Which tools detect sarcasm and polite frustration in AI conversations?

Tools that read the whole conversation with an LLM judge do better than per-message classifiers, because sarcasm and polite frustration depend on what came before. Agnost AI's judge reads the whole conversation and is instructed to weigh the user's emotional arc, including polite disappointment and sarcasm, along with repetition and whether the user got what they wanted. Amplitude (User Friction) and Opik (User frustration thread template) also judge whole sessions. Test any tool on a labeled sample of your own sarcastic and polite-but-failed conversations before trusting it.

How do I measure sentiment in Intercom Fin or Zendesk AI agent conversations?

Intercom's CX Score rates closed chat and email conversations from 1 to 5 using six reasons, including customer effort and strong emotion, with an AI-written explanation for each rating. It needs the Pro add-on and Fin. Zendesk intelligent triage labels sentiment from Very Positive to Very Negative and, on messaging channels with dynamic classification on, re-evaluates it on the whole conversation. It is on Suite and Support Professional plans and above. Both work only on conversations inside those products.

How should I combine sentiment with outcome?

Put each conversation into a two-by-two grid: did the user end positive or negative, and did the task succeed or fail? Negative and failed is your fix list. Positive and failed (polite users who gave up) is the group sentiment alone hides. Negative and succeeded usually means a hard problem handled well. Track the share of conversations in each cell by intent and release.

What changed in chatbot sentiment analysis tools since 2025?

Product analytics tools now ship AI agent features: Amplitude Agent Analytics scores every session on built-in signals including User Friction, and PostHog offers sentiment evaluations that run a local model on user messages. Helpdesk vendors now offer conversation-level scores that read the whole exchange, such as Intercom's CX Score. Microsoft announced that Azure AI Language sentiment analysis retires on March 31, 2029, and points new projects to Foundry models.


Originally published on agnost.ai.

Top comments (0)