How to build an AI chatbot that's actually useful

Wiring a language model to a chat box takes an afternoon. Building a chatbot people trust and return to takes real engineering. The gap between "it responds" and "it's useful" is where most projects stall — filled with confident wrong answers, lost context, and a bot that cheerfully promises refunds it can't issue. The demo dazzles the stakeholders; the production version has to survive real users asking real, weird questions at 2 a.m. Here's how to cross that gap.

Start with a job, not a technology

Before any code, define the one job the bot exists to do: deflect support tickets, qualify leads, help users navigate a complex product, answer questions over your documentation. A bot with a job can be scoped, measured, and improved. A bot that's just "AI on our site" pleases no one and can't be evaluated because there's no definition of success. Write the job down in a sentence. Everything below serves it.

That sentence also sets your bar for "good enough." A support-deflection bot succeeds if it resolves a meaningful share of tickets and hands off the rest cleanly — it does not need to be right 100% of the time, it needs to know when it isn't. A lead-qualification bot succeeds if it books qualified calls, not if it holds a charming conversation. Defining the job defines the metric, and the metric is what tells you whether to keep tuning or ship. Without it, you'll polish forever and never know if you're done. This is the same discipline that separates a real feature from a demo across AI work generally — the framing in how to integrate AI into your product applies directly.

Ground it or it will make things up

A chatbot answering from the model's general knowledge is a liability the moment users ask about your product, pricing, or policies. It will invent plausible, confidently-wrong answers — and a confident wrong answer is worse than "I don't know," because users act on it. The fix is grounding: retrieve relevant facts from your own content at query time and feed them into the prompt, so answers come from your data, not the model's imagination.

This is retrieval-augmented generation (RAG) applied to conversation — the pattern we unpack for non-engineers in RAG explained for founders. The mechanics:

1. Chunk your knowledge — docs, help center, policies — into passages of a few hundred tokens, split on meaningful boundaries (sections, not arbitrary character counts).
2. Embed each chunk into a vector and store it. pgvector on PostgreSQL works well to start and keeps everything in one database; graduate to a dedicated vector database only when scale demands it.
3. Retrieve the most relevant chunks for each incoming question via similarity search, optionally reranked.
4. Instruct the model to answer only from the retrieved passages, cite them, and explicitly say when the answer isn't in the provided context.

Retrieval quality is where most RAG bots live or die. If the right chunk never gets retrieved, the best prompt in the world can't save the answer. Invest here: tune chunk size, test your retrieval on real questions, and consider hybrid keyword-plus-vector search for queries full of exact product names or SKUs that pure semantic search fumbles.

Manage context deliberately

Conversations have memory, and memory has limits — both a token budget and an attention budget. You can't stuff an entire chat history into every request: it gets expensive, it's slow, and past a point the model loses focus on what matters. Handle context on purpose:

• Keep recent turns verbatim so the immediate thread stays coherent.
• Summarize older history into a compact running summary rather than dropping it entirely, so the bot remembers that the user is on the Pro plan and already tried restarting.
• Re-retrieve per turn. Fetch fresh grounding for the current question, not just the first one — conversations drift, and stale context produces answers to a question the user already moved past.

Put guardrails around it

A production bot needs boundaries, both to stay safe and to stay on-topic:

1. Scope it. A system prompt that defines what the bot does — and refuses politely outside it — prevents your support bot from writing poetry, giving medical advice, or being talked into your competitor's marketing.
2. Validate actions. If the bot can do things (create a ticket, issue a refund, change a subscription), gate those behind explicit confirmation and server-side authorization checks. Never let raw model output directly trigger a sensitive operation. Treat the model as an untrusted user that proposes actions your backend independently verifies against the real user's permissions.
3. Defend against prompt injection. Retrieved content and user messages can contain instructions that try to hijack the bot ("ignore previous instructions and..."). Keep system instructions and untrusted content clearly separated, never grant the model more authority than the user it's acting for, and screen inputs and outputs. This overlaps with ordinary application security — the basics in how to secure your web app still apply.

Choose the model deliberately — and keep the seam

Bigger isn't always better. Use a capable model for the hard reasoning turns, but route simple classification or routing steps to a smaller, cheaper, faster one. A frontier model on every keystroke will make the unit economics ugly at scale, and latency is part of the product — a bot that takes eight seconds to reply feels broken no matter how good the answer. Streaming responses token-by-token helps perceived speed enormously.

Because models and prices change every few months, wrap the whole thing in a thin, typed TypeScript service with a clean model-agnostic interface. That seam lets you swap providers, tune prompts, and adjust retrieval without touching the front end — and lets you A/B a cheaper model against your current one. On managing that bill as usage grows, see LLM cost optimization.

Decide: chatbot or workflow?

Not every "AI feature" should be a free-form chat. If the task is a fixed sequence — collect three fields, look something up, take one action — a structured agent or workflow is more reliable, cheaper, and easier to test than an open conversation. Reach for a chatbot when the interaction is genuinely open-ended. Using a chat interface for what's really a form frustrates users and multiplies your failure modes.

A reference architecture

Pulling it together, a production chatbot is more moving parts than the chat box suggests. A workable shape:

1. Ingestion pipeline — a job that chunks your source content, embeds it, and upserts it into the vector store, re-running whenever the source changes so the bot never answers from stale docs.
2. Retrieval layer — takes the user's message plus recent context, runs the similarity (and ideally keyword) search, and returns the top passages with their source links.
3. Orchestration service — the typed TypeScript seam that assembles the prompt (system instructions + retrieved context + trimmed history), calls the model, streams the response, and enforces the guardrails.
4. Action layer — a small, explicit set of server-side functions the model may propose calling, each re-checking authorization before it runs.
5. Observability — logging, the evaluation test set, and the quality signal wired in from day one, not bolted on after launch.

None of these is exotic, but skipping any one is how a demo fails to become a product. The retrieval and observability layers are the ones teams most often shortcut, and the ones that most determine whether the bot is actually trusted.

Design the escape hatch

The single most trust-building feature of any chatbot is a graceful handoff. When the bot is unsure, when it's failed twice, or when the user is clearly frustrated, it should offer a human, a form, or a clear next step — not loop endlessly apologizing. A bot that knows its limits and hands off cleanly feels far more competent than one that fakes confidence to the bitter end. Detect frustration and repeated failure explicitly, and make the exit obvious.

Build the evaluation loop

You cannot improve a chatbot by vibes, and "it seemed fine when I tried it" is how bad bots ship. Set up a habit of measurement before you launch, not after:

• A test set of real questions with known-good answers, run automatically on every prompt, model, or retrieval change so you catch regressions before users do. This is the closest thing RAG has to unit tests.
• Logging of real conversations (respecting privacy and consent) so you see where it actually fails in the wild — the failures you didn't imagine are the ones that matter.
• A quality signal — thumbs up/down, a resolution flag, or whether the conversation ended in a handoff — so you can track quality as a number over time rather than a feeling.

Then close the loop: mine the failures, fix the retrieval gaps or prompt holes they reveal, and re-run the test set. That iteration cadence, not any single clever prompt, is what turns a mediocre bot into a good one.

A useful chatbot is scoped to a real job, grounded in your data, careful with context, guarded against misuse, honest about uncertainty, and continuously evaluated. Skip those and you have a demo that impresses in the meeting and disappoints in production. If you want to build one that actually earns its place in your product, let's talk.

---

Originally published on the Doktouri Agency blog. We build web, mobile, SaaS, and AI products — let's talk.