How to Build a RAG Assistant for Your Company Docs in Python

Your team already knows the answers. They are just buried.

The refund policy lives in a PDF from two years ago. The onboarding steps sit in a shared drive folder nobody can find. The answer to "do we offer weekend service?" is in an email thread from last spring. So people ask each other, over and over, and the same questions eat the same hours every week.

A RAG assistant fixes that. RAG stands for retrieval augmented generation. In plain words: before the AI answers, it first looks up the right passages in your own documents, then writes an answer based on those passages and tells you where it found them.

This guide walks through building a small, working version in Python. It is aimed at small business owners who want to understand what they are buying, and at the developer on the team who will actually build it.

Why RAG instead of just asking a chatbot

A general chatbot knows a lot about the world and nothing about your business. Ask it about your return window and it will either refuse or, worse, guess.

You could paste your whole handbook into every prompt, but that gets slow and expensive fast, and it stops working once your documents outgrow the model's context window.

RAG takes a smarter path:

1. Split your documents into small chunks.
2. Turn each chunk into an embedding, a list of numbers that captures its meaning.
3. When a question comes in, find the chunks whose meaning is closest to the question.
4. Hand only those chunks to the language model and ask it to answer from them.

The result is an assistant that answers from your sources, stays current when you update a file, and can show its work with citations.

What you need

Keep the first version small. You need:

• A folder of documents. Start with plain text or Markdown exports of your FAQs, policies, and how to guides.
• Python 3.10 or newer.
• An embedding model. We will use a free model that runs on your own machine through the sentence-transformers library.
• A language model API for the final answer. The example uses the OpenAI Python client, but any provider with a chat API works the same way.

Install the pieces:

pip install sentence-transformers numpy openai

Library APIs change, so pin versions in a requirements.txt once things work.

Step 1: Load and chunk your documents

Chunking matters more than most people expect. Chunks that are too big bury the answer in noise. Chunks that are too small lose context. A few hundred words with a little overlap is a sensible starting point.

from pathlib import Path

def load_docs(folder: str):
    docs = []
    for path in Path(folder).rglob("*"):
        if path.suffix.lower() in {".txt", ".md"}:
            docs.append({"source": path.name, "text": path.read_text(encoding="utf-8")})
    return docs

def chunk_text(text: str, size: int = 200, overlap: int = 40):
    words = text.split()
    chunks = []
    start = 0
    while start < len(words):
        chunks.append(" ".join(words[start:start + size]))
        start += size - overlap
    return chunks

docs = load_docs("company_docs")
chunks = []
for doc in docs:
    for i, piece in enumerate(chunk_text(doc["text"])):
        chunks.append({"id": f"{doc['source']}#{i}", "source": doc["source"], "text": piece})

print(f"{len(docs)} documents, {len(chunks)} chunks")

Each chunk keeps its source file name. That is what lets the assistant cite where an answer came from later.

Step 2: Create embeddings

Now we turn every chunk into a vector. all-MiniLM-L6-v2 is a small, popular model that runs fine on a laptop CPU.

import numpy as np
from sentence_transformers import SentenceTransformer

embedder = SentenceTransformer("sentence-transformers/all-MiniLM-L6-v2")

texts = [c["text"] for c in chunks]
vectors = embedder.encode(texts, normalize_embeddings=True, show_progress_bar=True)
vectors = np.asarray(vectors, dtype="float32")

np.save("vectors.npy", vectors)

Setting normalize_embeddings=True means a simple dot product gives you cosine similarity, which keeps the search code tiny.

For a few thousand chunks, a NumPy array in memory is all you need. When you grow into tens of thousands of chunks or want filtering by department or date, move to a vector database such as pgvector, Qdrant, or Chroma. The logic stays the same.

Step 3: Retrieve the best matches

When a question arrives, embed it with the same model and find the closest chunks.

def retrieve(question: str, k: int = 4, min_score: float = 0.3):
    q = embedder.encode([question], normalize_embeddings=True)[0]
    scores = vectors @ q
    best = np.argsort(scores)[::-1][:k]
    return [
        {**chunks[i], "score": float(scores[i])}
        for i in best
        if scores[i] >= min_score
    ]

for hit in retrieve("How long do customers have to request a refund?"):
    print(round(hit["score"], 3), hit["id"])

That min_score threshold is your first guardrail. If nothing in your documents is a reasonable match, you want the assistant to say so instead of stretching a weak match into an answer. Tune the number by testing real questions against your own files.

Step 4: Generate a grounded answer

Here is where the language model comes in. The prompt does three jobs: it supplies the retrieved passages, tells the model to answer only from them, and asks it to cite sources.

from openai import OpenAI

client = OpenAI()  # reads OPENAI_API_KEY from the environment

SYSTEM = (
    "You answer questions for our staff using ONLY the provided context. "
    "If the context does not contain the answer, say you could not find it "
    "and suggest who to ask. Cite sources in square brackets, like [refunds.md]."
)

def answer(question: str):
    hits = retrieve(question)
    if not hits:
        return "I could not find that in our documents. Try asking the office manager."

    context = "\n\n".join(f"[{h['source']}]\n{h['text']}" for h in hits)
    response = client.chat.completions.create(
        model="gpt-4o-mini",
        temperature=0,
        messages=[
            {"role": "system", "content": SYSTEM},
            {"role": "user", "content": f"Context:\n{context}\n\nQuestion: {question}"},
        ],
    )
    return response.choices[0].message.content

print(answer("Do we offer weekend service appointments?"))

Swap the model name for whatever your provider offers. A temperature of zero keeps answers consistent from one run to the next, which is what you want for policy questions.

That is a complete RAG assistant in well under a hundred lines. Wrap answer() in a small web app, a Slack bot, or an internal page and your team can start using it.

Step 5: Test it like you mean it

The demo always looks great. The real test is the question nobody planned for.

Before anyone relies on the assistant, build a simple evaluation set:

• Write down 20 to 30 real questions your team actually asks.
• Next to each one, note the correct answer and the document it should come from.
• Include a few questions your documents cannot answer, so you can confirm the assistant admits it.

tests = [
    {"q": "What is our refund window?", "expect_source": "refunds.md"},
    {"q": "Who approves overtime?", "expect_source": "hr_policy.md"},
    {"q": "What is the CEO's favorite color?", "expect_source": None},
]

for t in tests:
    hits = retrieve(t["q"])
    sources = {h["source"] for h in hits}
    if t["expect_source"] is None:
        ok = not hits
    else:
        ok = t["expect_source"] in sources
    print("PASS" if ok else "FAIL", t["q"], sorted(sources))

This checks retrieval, which is where most RAG problems start. If the right chunk never reaches the model, no prompt can save the answer. Rerun this list every time you change chunk sizes, the embedding model, or the prompt.

Common mistakes to avoid

Messy source documents. Scanned PDFs, tables pasted as images, and five versions of the same policy will confuse any assistant. Clean up and remove outdated files first. If much of your knowledge lives in scanned paperwork, that is a document processing problem to solve before retrieval.

No access control. If payroll files and customer FAQs sit in the same index, someone will eventually get an answer they should not see. Tag chunks with who may read them and filter at retrieval time.

Treating documents as trusted. Text inside a document can try to instruct the model. Keep the assistant read only, and never let retrieved text trigger actions on its own.

Skipping the "I don't know" path. An assistant that admits it cannot find something earns trust. One that confidently invents a policy loses it in a single afternoon.

Never updating the index. Rebuild embeddings whenever documents change. A nightly job is usually enough for a small team.

Where this goes next

Once the basic version is answering well, there are a few natural upgrades:

• Hybrid search. Combine embeddings with classic keyword search so exact terms like part numbers and policy codes still match.
• Reranking. Retrieve 20 candidates, then use a cross encoder model to pick the best 4. This often improves answer quality noticeably.
• More file types. Add loaders for PDFs, Word files, and your help desk or wiki.
• Agents. Let the assistant not only answer, but also look up an order status or open a ticket, with a person approving any action.

The customer facing version of this pattern is a support assistant on your website. If that is where you are headed, our guide to AI chatbots for customer service covers what to plan for before you launch one.

Build it yourself or bring in help?

The code above is honest about what RAG is: a search step and a writing step. A capable developer can have a prototype running in a day.

The harder part is everything around it. Cleaning years of documents, connecting to the systems your knowledge actually lives in, setting permissions, and building an evaluation set that reflects how your team really talks. That is where most of the effort, and most of the value, sits.

If you would rather have that handled for you, we build custom AI solutions like RAG knowledge assistants that plug into the tools you already use. We also work with smaller teams through our AI for small business service, with a written scope and clear acceptance criteria up front.

Wrapping up

You do not need a giant platform to give your team instant answers from your own documents. Chunk the files, embed them, retrieve the best matches, and ask the model to answer only from what it found. Then test it with real questions before anyone depends on it.

Start with one folder and one team. Once people stop asking each other the same five questions every week, you will know it is working.

---

Want a RAG assistant built around your company's documents and systems? Business Edge Analytics designs and builds practical AI for small and mid sized businesses, fully remote from Louisville, KY. Reach out through the site to talk through your use case.