Stop Your Chatbot From Hallucinating: A Practical Guide to RAG

Ask a raw LLM about your refund policy and it will confidently invent one. That's the core problem with using foundation models for company-specific answers: they know a lot about the world and nothing about you.

Retrieval Augmented Generation (RAG) fixes this. Instead of relying on the model's training data, you fetch relevant chunks from your own documents at query time and hand them to the model as context. The model stops guessing and starts summarizing facts you gave it.

Here's how it actually works, and how to build one that doesn't fall apart in production.

The core idea

A RAG pipeline has two phases: ingestion (done once, or on a schedule) and retrieval (done on every query).

Ingestion:

1. Split your documents into chunks.
2. Convert each chunk into a vector (an embedding).
3. Store those vectors in a vector database.

Retrieval:

1. Embed the user's question.
2. Find the most similar chunks in the vector DB.
3. Stuff those chunks into the prompt and ask the LLM to answer using only that context.

That's it. No fine-tuning, no retraining. When your policy changes, you re-ingest the document and the answers update instantly.

Building the ingestion pipeline

Chunking is where most RAG systems quietly fail. Chunk too big and you dilute relevance. Chunk too small and you lose context. A good starting point is 500-1000 tokens with a bit of overlap so sentences aren't cut mid-thought.

from openai import OpenAI
import chromadb

client = OpenAI()
db = chromadb.PersistentClient(path="./kb").get_or_create_collection("docs")

def chunk_text(text, size=800, overlap=100):
    words = text.split()
    chunks, i = [], 0
    while i < len(words):
        chunks.append(" ".join(words[i:i + size]))
        i += size - overlap
    return chunks

def embed(texts):
    resp = client.embeddings.create(
        model="text-embedding-3-small", input=texts
    )
    return [d.embedding for d in resp.data]

def ingest(doc_id, text, source):
    chunks = chunk_text(text)
    db.add(
        ids=[f"{doc_id}-{i}" for i in range(len(chunks))],
        documents=chunks,
        embeddings=embed(chunks),
        metadatas=[{"source": source} for _ in chunks],
    )

Note the metadatas. Storing the source alongside each chunk lets you cite where an answer came from later, which is non-negotiable for enterprise trust.

The retrieval and answer step

On each query you embed the question, pull the top matches, and constrain the model tightly.

def answer(question, k=4):
    q_vec = embed([question])[0]
    results = db.query(query_embeddings=[q_vec], n_results=k)

    context = "\n\n".join(results["documents"][0])
    sources = {m["source"] for m in results["metadatas"][0]}

    prompt = f"""Answer the question using ONLY the context below.
If the context does not contain the answer, say you don't know.

Context:
{context}

Question: {question}"""

    resp = client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[{"role": "user", "content": prompt}],
        temperature=0,
    )
    return resp.choices[0].message.content, sources

Two details do the heavy lifting here:

• temperature=0 keeps answers deterministic and factual.
• "If the context does not contain the answer, say you don't know" is the single most important line in the whole prompt. Without it, the model fills gaps with invention. With it, you get honest "I don't know" responses instead of confident lies.

Where RAG systems break in the real world

The demo above works. A production system needs more.

Bad retrieval beats good generation

If the right chunk never gets retrieved, no amount of prompt engineering saves you. Pure vector search misses exact-match terms like SKUs, error codes, or names. Combining vector search with keyword search (hybrid search) and then re-ranking the results dramatically improves accuracy.

Chunking your actual content

A PDF price table chunked by word count becomes garbage. Structured content needs structure-aware splitting: keep tables intact, split markdown by headings, and keep FAQ question-answer pairs together.

Stale data

A knowledge base is only as good as its freshness. Wire ingestion to your source of truth — a Notion workspace, a Zendesk help center, a Google Drive folder — and re-run it on a schedule or a webhook so answers never drift from reality.

Evaluation

You can't improve what you don't measure. Build a set of 30-50 real questions with known correct answers and run them after every change. Track how often the retrieved chunks actually contain the answer, and how often the final response is correct.

When to reach for RAG vs. fine-tuning

Use RAG when answers depend on facts that change: policies, pricing, product docs, internal wikis. Use fine-tuning when you need to change behavior — tone, format, a specialized reasoning style. Most business chatbots need the first, not the second. RAG is cheaper, faster to update, and far easier to audit.

The takeaway

A RAG chatbot is not magic — it's a retrieval problem wearing an LLM costume. Get the retrieval right, constrain the generation, cite your sources, and keep your knowledge base fresh. Do that and you get an AI agent that answers from your facts, admits when it doesn't know, and stops embarrassing you in front of customers.

That's the difference between a demo and a system you can actually put in front of a client. If you'd rather have that system built and maintained for you, that's the kind of work we do at Michael AI.

---

Originally published at getmichaelai.com