DEV Community

Shubham Palriwala
Shubham Palriwala

Posted on Originally published at agnost.ai

How to cluster AI chatbot conversations by user intent, and which tools do it (2026)

To cluster AI chatbot conversations by user intent, embed what the user asked for (the first user message, or a one-line summary of their goal), group the vectors with HDBSCAN or k-means, and have an LLM name each cluster from its most typical examples. Then freeze the good clusters into a fixed intent list and classify new conversations against it, so your numbers stay comparable week to week. If you'd rather not run the pipeline yourself, Agnost AI, Braintrust, LangSmith, PostHog, Amplitude, Nebuly, Dimension Labs and Intercom all do some version of it, with different inputs and different outputs.

This page is the hands-on version: a working Python sketch, the mistakes that make intent clusters useless, and how to get from clusters to a failure rate per intent. For choosing an intent tool on accuracy, multilingual quality and cost, see the intent analysis tools buyer guide. For finding unmet needs, see how to find unmet user needs from AI assistant chats.

Disclosure: We make Agnost AI. Facts about other tools and libraries come from their own docs and pricing pages, checked on November 18, 2026. This is a documentation comparison, not a hands-on benchmark. Updated for 2026: since 2025, most LLM observability tools have added some form of automatic clustering (Braintrust Topics, LangSmith Insights, PostHog clusters, Arize Signal), so building it yourself is now a choice rather than the only option.

The DIY pipeline in five steps

  1. Pick the text that carries the intent. One string per conversation.
  2. Embed it. A local sentence-embedding model or an embeddings API.
  3. Cluster. HDBSCAN if you don't know how many intents exist; k-means if you need a fixed number.
  4. Name the clusters. Show an LLM the most typical examples of each cluster and ask for a short goal-style name.
  5. Freeze and classify. Review the names, merge and split by hand, and turn the result into a versioned intent list.

Step 1: choose what to embed

Input Use when Watch out for
First user message Users state their goal up front (most support and Q&A bots) Greetings like "hi" or "can you help?" carry no intent
LLM summary of the user's goal The goal emerges over several turns, or first messages are vague Costs one LLM call per conversation; the summary prompt shapes the clusters
Each user turn You want to see goal changes inside a conversation Many more points, and follow-ups like "thanks" form junk clusters
Whole transcript Almost never The assistant's boilerplate dominates and you cluster by length and tone

Strip anything that shouldn't drive similarity before you embed: order numbers, emails, names, pasted error logs. Two people asking to "refund order 48213" and "refund order 99102" should land together.

Step 2 and 3: embed and cluster

This sketch uses Sentence Transformers for embeddings and scikit-learn's HDBSCAN, which ships with scikit-learn 1.3 and later.

# pip install sentence-transformers scikit-learn pandas
import numpy as np
from sentence_transformers import SentenceTransformer
from sklearn.cluster import HDBSCAN

# One string per conversation: the first user message, or a summary of the user's goal.
texts = [c["first_user_message"] for c in conversations]

model = SentenceTransformer("sentence-transformers/all-mpnet-base-v2")
embeddings = model.encode(
    texts,
    batch_size=64,
    normalize_embeddings=True,   # unit length, so Euclidean distance ranks like cosine
    show_progress_bar=True,
)

clusterer = HDBSCAN(min_cluster_size=15, min_samples=5)
labels = clusterer.fit_predict(embeddings)   # -1 means noise: fits no cluster

print("clusters:", len(set(labels) - {-1}), "noise:", int((labels == -1).sum()))
Enter fullscreen mode Exit fullscreen mode

min_cluster_size is the smallest group you'd call an intent. Raise it if you get hundreds of tiny clusters; lower it if real intents vanish into noise. min_samples defaults to min_cluster_size; a smaller value makes clustering less conservative and puts fewer points in noise.

To use OpenAI embeddings instead, swap the encode call. OpenAI's docs say its embeddings are already normalized to length 1:

from openai import OpenAI

client = OpenAI()

def embed(batch):
    resp = client.embeddings.create(model="text-embedding-3-small", input=batch)
    return [d.embedding for d in resp.data]

embeddings = np.array([v for i in range(0, len(texts), 500) for v in embed(texts[i:i + 500])])
Enter fullscreen mode Exit fullscreen mode

If you need a fixed number of groups, for example one per support queue, use k-means instead. Every point gets a cluster, so there is no noise bucket:

from sklearn.cluster import KMeans

labels = KMeans(n_clusters=30, n_init="auto", random_state=0).fit_predict(embeddings)
Enter fullscreen mode Exit fullscreen mode

On tens of thousands of conversations, density-based clustering on 768-dimensional vectors gets slow and less sharp. Reducing dimensions first (with PCA, or UMAP from the umap-learn package) is the usual fix. Check that the clusters still make sense after reduction.

Step 4: name each cluster with an LLM

HDBSCAN's probabilities_ attribute gives each point's membership strength, so the highest-scoring members are the cluster's most typical examples. Send 10 to 20 of them to an LLM:

import pandas as pd

df = pd.DataFrame({
    "text": texts,
    "cluster": labels,
    "strength": clusterer.probabilities_,
})

def examples_for(cluster_id, n=15):
    rows = df[df.cluster == cluster_id].nlargest(n, "strength")
    return "\n".join(f"- {t}" for t in rows.text)

PROMPT = """These are messages users sent to our support assistant. They were grouped together.
Name the single goal they share in 2 to 5 words, starting with a verb (for example "Cancel subscription").
If they don't share one goal, answer MIXED.

{examples}"""

# For each cluster: name = your_llm(PROMPT.format(examples=examples_for(cid)))
Enter fullscreen mode Exit fullscreen mode

Ask for verb-first goal names ("Change delivery address"), not topics ("Delivery"). A topic tells you what users talked about; a goal tells you what they needed, which is what you'll measure success against. Treat MIXED answers as a signal to split that cluster.

Step 5: freeze the list and classify new traffic

Clusters are for discovery, not reporting. If you re-cluster every week, cluster 7 this week isn't cluster 7 last week, and every trend line breaks. Instead:

  1. Review the named clusters by hand. Merge duplicates ("Reset password" and "Can't log in" may be one intent for you, or two).
  2. Write each keeper down as an intent with a one-sentence definition and two or three example messages.
  3. Version the list.
  4. Classify new conversations against the list, with an LLM classifier or nearest-centroid on the embeddings, and send anything that fits nothing to unknown.
  5. Re-cluster only the unknown bucket on a schedule. That is where new intents show up.

Pitfalls that make intent clusters useless

Multi-intent conversations. A user asks to change a plan, then asks why they were double-charged. One label per conversation hides the second goal, which is often the one that went wrong. Either embed each user turn that introduces a new request, or have the summary step list every goal and let a conversation carry several intents. The buyer guide covers detecting goal changes in more depth.

Drift. Launches, pricing changes, outages and seasons all move the mix. A new feature produces a new intent that your frozen list doesn't have, so it lands in unknown (good) or gets forced into a near neighbour (bad). Watch the unknown rate: when it climbs, re-cluster it. And watch for clusters that grow suddenly, which often means an outage rather than a trend.

Cluster naming. LLM-written names are confident and often wrong in subtle ways: too broad ("Account questions"), too narrow (named after one vivid example), or named after the assistant's reply instead of the user's request. Always read a sample of each cluster before you trust its name, and keep the names goal-shaped.

Clustering on the assistant's words. If the embedded text includes the assistant's reply, conversations cluster by how the bot answered ("I'm sorry, I can't help with that") rather than what the user wanted. Embed user text, or a summary of it.

Greeting clusters. "Hi", "hello" and "can you help me" form a large, useless cluster. Skip messages under a few words, or move to the first substantive user message.

Mistaking volume for priority. The biggest cluster is often the one your bot already handles well. What you want is the intent where volume and failure rate are both high, which needs the next step.

From clusters to failure rates per intent

An intent count tells you what people want. It doesn't tell you whether they got it. To find what to fix, join each conversation's intent to an outcome:

  • a rule your assistant broke (for example "never quotes a refund amount it hasn't looked up"),
  • a frustration signal: the user rephrased, repeated themselves or complained,
  • an escalation or a request for a human,
  • the user leaving without the task done,
  • a tool call that errored.
# failed: one boolean per conversation from your evals, escalation flags or frustration labels
df["failed"] = failed

report = (
    df[df.cluster != -1]
    .groupby("cluster")
    .agg(conversations=("text", "size"), failure_rate=("failed", "mean"))
    .assign(failed_conversations=lambda d: (d.conversations * d.failure_rate).round().astype(int))
    .sort_values("failed_conversations", ascending=False)
)
Enter fullscreen mode Exit fullscreen mode

Sort by failed conversations, not failure rate alone. A 60% failure rate on 12 conversations matters less than 15% on 4,000.

Then cluster the failures. This is the "error analysis" step people search for. Take only the failed conversations in your worst intent, have an LLM write one sentence on what went wrong in each ("agent looked up the wrong order", "agent promised a callback it can't schedule"), and cluster those sentences. Intent clusters tell you where users fail; failure clusters tell you why. Fix the biggest failure cluster, then check its rate again after the change ships.

The hard part in practice is not the clustering. It is getting a trustworthy failure label on every conversation, which needs whole-conversation judging, since many failures (a wrong answer on turn 3, a user who gives up on turn 6) never raise an error. See how to catch wrong answers that don't show up as errors in traces.

Tools that cluster chatbot conversations by intent

If you don't want to run the pipeline yourself, these tools do some or all of it. They differ on what they read (your traces, a bot platform's transcripts, or a help desk's tickets) and on whether they tie clusters to failures.

Tool What it groups Intent or topic Ties groups to failures Reads data from Entry price
Agnost AI Whole production conversations, by user and session Auto-generated and custom intents Yes: failures, frustration, tool errors, rule violations OpenTelemetry, Python and TypeScript SDKs Free; Starter $49/mo
Braintrust Topics Logs, via per-log facet summaries Task facet (user intent), plus Sentiment and Issues Issues facet Braintrust logs Uses model credits
LangSmith Insights A sample of up to 1,000 traces per report Usage patterns and agent behaviors Failure modes LangSmith traces Plus and Enterprise plans
Arize Signal Traces Not documented Groups recurring failures into ranked issues Arize AX traces Free 10 issues/mo
PostHog Traces, generations or eval results, daily AI-titled clusters Per-cluster error rate PostHog LLM events 100,000 AI events/mo free
Amplitude Agent Analytics Agent sessions User Intent signal and Topics Combine with Task Completed and User Friction Amplitude agent sessions Session allowance by tier
Nebuly Conversations Task behind each conversation Frustration metric Integrations, Interaction API Not published
Dimension Labs Bot conversations and tickets Intent field or contact-reason Dimension, grouped into themes Failure types such as looping Bot platforms, upload, API Explore $299/mo
Intercom Topics Explorer Support conversations, Fin and human AI topics and subtopics Resolution rate and CX Score per topic Intercom Pro add-on
Zendesk intelligent triage New tickets Topic (formerly "intent") classification Not documented Zendesk Suite or Support Professional
Langfuse Nothing built in Write your own labels as scores Build it yourself Langfuse traces and sessions Open source

Agnost AI

What it does. Agnost AI monitors AI agents in production (chat agents, voice agents through transcripts, copilots and MCP servers). It groups traffic by user and session and classifies each whole conversation against your intents. You can generate suggested intents from your product description and observed traffic, then edit them or write your own definitions. Each intent shows its volume over time and links to the conversations behind it, and you can set a threshold alert on any intent.

Why it fits this job. Intents sit next to the outcome signals from the section above: failures, user frustration, tool-call errors, loops and silent wrong answers. Online evals judge every conversation against plain-language rules (pass or fail per assistant message) once it has been idle 15 minutes, with no sampling, so the failure rate per intent covers all traffic rather than a sample. When an alert fires, Auto Improve can open a pull request with a fix tested against past conversations for a person to review.

Limits. Intents are a fixed, governed list (generated or written), not an exploratory scatter plot of clusters. Verdicts are pass or fail, not scores. Judging runs in batches after 15 minutes idle, not in real time. It is not open source and has no offline datasets or experiments.

Pricing. Free covers 1,000 events a month with 7-day retention. Starter is $49 a month (10,000 events), Pro $499 a month (1,000,000 events), Enterprise custom.

Braintrust Topics

What it does. Braintrust writes a short summary of each log for each facet, embeds the summaries, clusters them, and has a naming model title each cluster. Built-in facets are Task (what the user is trying to do), Sentiment and Issues, and you can add custom facets. It needs at least 100 summaries, regenerates topics daily, and classifies new logs continuously.

Best for. Eval-driven teams already logging to Braintrust who want intent clusters to feed datasets.

Limits. Usage draws on Braintrust model credits, then on-demand token rates. It clusters logs rather than reading users across sessions.

LangSmith Insights

What it does. Insights analyzes a sample of traces (up to 1,000 per report) and sorts them into hierarchical categories of usage patterns, agent behaviors and failure modes, with an executive summary. It uses a model you configure from OpenAI or Anthropic for clustering and a second for per-trace summaries.

Best for. LangChain and LangGraph teams that want a periodic report rather than a live intent dashboard.

Limits. Plus and Enterprise plans only. Reports run on a sample, and the model calls run on your provider account.

Arize Signal

What it does. Signal scans a tracing project's traces on a schedule (every 6 hours by default) and groups recurring failure patterns into ranked issues, each with trace evidence and a suggested fix. It does the "cluster the failures" step, not intent clustering; we found no topic or intent clustering in the AX docs.

Pricing. Free includes 10 issues a month and Pro 25. Enterprise is unlimited and adds repo-backed fix PRs.

PostHog

What it does. PostHog's AI observability clusters traces, generations or evaluation results in daily jobs. AI writes a title and description for each cluster, and each shows average cost, latency, tokens and error rate. Up to 10 jobs per project; each needs at least 10 matching traces from the last 7 days. It doesn't backfill older traces, and the docs don't say which clustering method it uses.

Best for. Teams already using PostHog who want clusters beside product analytics.

Pricing. The first 100,000 AI observability events a month are free, then priced per event.

Amplitude Agent Analytics

What it does. Each agent session gets a User Intent signal, and Topics clusters sessions across all online traffic on a map. Amplitude suggests combining Topics with its Task Completed, User Friction and Helpfulness signals to see which topics succeed. Billing is by agent session, with allowances that scale by tier.

Best for. Product teams already on Amplitude.

Nebuly

What it does. Nebuly says "every conversation is mapped to the task behind it," and its dashboard shows labelled intents with business tags (churn risk, revenue risk, upsell, unmet demand) and a frustration metric. It plugs into agents you already run, through integrations and an Interaction API, without changing them. It runs as SaaS or inside your own AWS, Azure or GCP account or on premises.

Limits. Its pages don't describe how conversations are grouped. Prices aren't published.

Dimension Labs

What it does. For intent-based bots it reports on the bot's own intent.name; for LLM bots, which have no intents, it extracts a contact-reason field from the text instead. It groups intents or contact reasons into broader themes, groups fallback messages by theme with examples, and suggests fields for failure type (such as "looping"), escalation, completion and unmet need. Data comes from platforms such as Intercom, Ada, Microsoft Copilot Studio, Cognigy, Dialogflow CX, Botpress, Voiceflow and LivePerson, or from upload and API.

Best for. CX teams whose bot runs on a platform rather than in their own code.

Pricing. Explore is $299 a month for 1,000 conversations, rising to Scale at $6,999 for 100,000; Enterprise from $150,000 a year. 14-day trial.

Intercom Topics Explorer and Zendesk intelligent triage

Intercom. The Topics Explorer builds topics from the past 90 days of conversations, clustering similar questions into subtopics and grouping those into topics, then assigns newly closed conversations daily. It covers both Fin and human conversations and shows CX Score, resolution rate and handling time per topic. It needs the Pro add-on and Fin live on at least one channel.

Zendesk. Intelligent triage classifies new tickets by topic (called "intent" for customers who bought before June 11, 2026), sentiment, language and entities, and you can customize topic values. On messaging channels it re-evaluates based on the whole conversation. It is a classifier against a list, not a discovery tool; the page doesn't describe finding new intents.

Best for. Teams whose AI agent is Intercom Fin or runs inside Zendesk. If you built your own agent, these don't see its traces or tool calls.

Langfuse

Langfuse sessions group traces into conversations for replay, but it has no built-in topic or intent clustering. Teams on Langfuse run the pipeline above and write the intent label back as a score.

How to choose

Your situation Start with
You want to understand your data before buying anything The DIY pipeline above, on a month of first user messages
You build your own agent and want failure rates per intent on all traffic Agnost AI
You log to Braintrust and want clusters that feed eval datasets Braintrust Topics
You're on LangSmith Plus and want a periodic patterns-and-failures report LangSmith Insights
You only care about grouping failures, not intents Arize Signal, or the failure-clustering step above
Your product team lives in PostHog or Amplitude PostHog clusters or Amplitude Topics
Your bot runs on a platform such as Cognigy, Ada or Dialogflow Dimension Labs
Your AI agent is Intercom Fin, or your support runs on Zendesk Intercom Topics Explorer, or Zendesk intelligent triage
You're on Langfuse and want to keep it The DIY pipeline, writing labels back as scores

Whichever you pick, hand-label 200 conversations first and check the tool's intents against yours. If a tool's clusters don't match how your team thinks about user goals, its dashboards won't change any decisions.

Related reading

Frequently asked questions

How do I cluster AI chatbot conversations by user intent?

Take the first user message of each conversation (or an LLM summary of what the user wanted), embed it with a sentence-embedding model or an embeddings API, cluster the vectors with HDBSCAN or k-means, and have an LLM name each cluster from its most typical examples. Keep the noise bucket, review the names by hand, then freeze the good clusters into a fixed list of intents and classify new conversations against that list so weekly numbers stay comparable.

What tools cluster AI chatbot conversations by user intent?

Agnost AI classifies whole production conversations against auto-generated and custom intents and shows failures per intent. Braintrust Topics clusters logs by a Task facet (the user's intent). LangSmith Insights groups traces into usage patterns and failure modes on Plus and Enterprise. PostHog clusters traces and generations daily with AI-written titles. Amplitude Agent Analytics clusters sessions into Topics. Nebuly maps each conversation to the task behind it. Dimension Labs groups intents or contact reasons into themes for platform bots. Intercom's Topics Explorer clusters support conversations, including Fin's, into topics and subtopics.

Should I use HDBSCAN or k-means to cluster chatbot conversations?

Start with HDBSCAN. You don't have to pick the number of clusters, and it labels messages that fit no group as noise (-1) instead of forcing them into the nearest cluster, which is how new or odd requests stay visible. Use k-means when you need a fixed number of groups, for example to match a support team's queues, and accept that every message gets assigned somewhere.

What should I embed: the first message or the whole conversation?

Embed the first user message when users state their goal up front, which is common in support chat. Embed a one-sentence LLM summary of the user's goal when the first message is a greeting or the goal emerges over several turns. Embedding the whole transcript usually clusters by style and length rather than by intent, because the assistant's boilerplate dominates the text.

How do I find failure patterns in LLM conversations with clustering?

Cluster twice. First cluster all conversations by intent to get volume per intent. Then take only the conversations that failed (a rule broke, the user rephrased, asked for a human or left) and cluster a short description of what went wrong. The first view tells you which intents fail most; the second tells you why. Agnost AI, LangSmith Insights, Arize Signal and Braintrust Topics' Issues facet do versions of the second step for you.

How many conversations do I need before clustering is useful?

A few hundred is enough to see the main intents; a few thousand gives you stable long-tail clusters. Vendor tools set their own floors: Braintrust Topics needs at least 100 facet summaries, and PostHog needs at least 10 traces from the last 7 days per clustering job.

How do I stop intent clusters from changing every week?

Don't re-cluster from scratch for reporting. Run discovery clustering on a schedule, promote stable clusters into a versioned intent list, and classify new conversations against that list. Send anything that matches no intent to an unknown bucket and re-cluster only that bucket to find new intents.


Originally published on agnost.ai.

Top comments (0)