RAG Chatbots Without the Hallucinations: A Builder's Guide to Grounded AI Agents

Ask a raw LLM about your refund policy and it will happily invent one. Confident, fluent, and wrong. That's the whole problem RAG exists to solve.

Retrieval Augmented Generation is simple in concept: before the model answers, you fetch the relevant chunks of your data and stuff them into the prompt. The model stops guessing and starts summarizing from source material you control.

Most teams get the concept in five minutes and then spend three weeks fighting the parts nobody talks about. Let's cover both.

How RAG actually works

The pipeline has two phases: indexing (done once, or on a schedule) and retrieval (done per query).

Indexing

1. Load your documents (docs, tickets, PDFs, Notion, help center).
2. Split them into chunks small enough to be precise, large enough to keep context.
3. Embed each chunk into a vector using an embedding model.
4. Store vectors in a database like pgvector, Pinecone, or Qdrant.

Retrieval

1. Embed the user's question.
2. Find the top-k most similar chunks.
3. Inject those chunks into the prompt as context.
4. Let the LLM answer using only that context.

Here's the retrieval-and-answer step stripped to its essentials:

from openai import OpenAI
import numpy as np

client = OpenAI()

def embed(text):
    r = client.embeddings.create(model="text-embedding-3-small", input=text)
    return np.array(r.data[0].embedding)

def retrieve(question, chunks, embeddings, k=4):
    q = embed(question)
    scores = embeddings @ q / (
        np.linalg.norm(embeddings, axis=1) * np.linalg.norm(q)
    )
    top = scores.argsort()[-k:][::-1]
    return [chunks[i] for i in top]

def answer(question, chunks, embeddings):
    context = "\n\n".join(retrieve(question, chunks, embeddings))
    prompt = f"""Answer using ONLY the context below.
If the answer isn't there, say "I don't have that information."

Context:
{context}

Question: {question}"""
    r = client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[{"role": "user", "content": prompt}],
        temperature=0,
    )
    return r.choices[0].message.content

That's a working RAG bot in 30 lines. It's also a demo, not a product. The gap between the two is where the real work lives.

Where RAG quietly fails

Bad chunking ruins everything

If you split by fixed character count, you'll slice sentences in half and separate a policy from its exception. Chunk by structure instead: headings, paragraphs, list items. Add small overlaps (10-15%) so context bleeds across boundaries.

Retrieval misses the point

Vector search finds semantic similarity, not always relevance. "Do you offer refunds?" and "return policy for damaged goods" may not rank close together. Fixes that work:

• Hybrid search: combine vector similarity with keyword (BM25) search.
• Reranking: pull top 20 candidates, then use a cross-encoder reranker to pick the best 4.
• Query rewriting: have the LLM rephrase the question before retrieval.

The model still hallucinates

Even with context, models fill gaps. Two levers help: temperature=0 and an explicit instruction to refuse when the answer isn't present. Then verify it worked with citations.

No citations, no trust

Every answer should point back to a source. Return the chunk IDs alongside the response so users (and you) can audit it. A support bot that cites the exact help article is trusted. One that doesn't gets ignored after the first wrong answer.

Making it production-grade

A few upgrades separate hobby projects from systems people actually rely on.

Metadata filtering. Tag chunks with product, region, or access level, then filter before retrieval. A customer on the Starter plan shouldn't get answers about Enterprise features.

Freshness. Your docs change. Set up an incremental re-index so updated content flows in without rebuilding the whole store. Stale answers are just a slower kind of wrong.

Fallback paths. When retrieval confidence is low, route to a human or offer to open a ticket instead of forcing an answer. Knowing when not to answer is a feature.

Evaluation. Build a test set of real questions with known-good answers. Score retrieval hit rate and answer accuracy on every change. Without this you're tuning blind.

def eval_retrieval(test_cases, chunks, embeddings):
    hits = 0
    for q, expected_source in test_cases:
        results = retrieve(q, chunks, embeddings, k=4)
        if any(expected_source in r for r in results):
            hits += 1
    return hits / len(test_cases)

What good RAG delivers

Done right, a knowledge base agent deflects a large share of repetitive support tickets, answers sales questions instantly with accurate product details, and gives your team an internal assistant that actually knows your runbooks.

The business result isn't "we have a chatbot." It's faster response times, fewer escalations, and answers that hold up under scrutiny because they're grounded in your own data.

Start small, then scale

Don't try to index every document you own on day one. Pick one high-volume question type, wire up a tight retrieval loop, add citations, and measure accuracy. Once it's reliably above your threshold, expand the corpus.

RAG isn't magic. It's disciplined engineering around a language model. The teams that win treat it like a search problem with an LLM on top, not an LLM problem with search bolted on.

If you'd rather ship a grounded agent than debug chunking strategies for a month, that's exactly the kind of system we build at Michael AI.

---

Originally published at getmichaelai.com