Build a RAG Chatbot That Admits When It Doesn't Know (Pure Python, No Dependencies)

The problem: chatbots that guess

Every team wants a chatbot that answers questions from their own documents — support FAQs, internal policies, product manuals. The naive approach is to paste documents into a prompt or fine-tune a model, and the failure mode is always the same: when the documents don't contain the answer, the model invents one. A confident, wrong refund policy is worse than no answer at all.

Retrieval-augmented generation (RAG) fixes the retrieval half of this: you search your documents first, and only answer from what you found. But retrieval alone isn't enough. You also need a gate — a relevance threshold below which the bot refuses to answer — and citations, so a human can verify every claim.

In this tutorial we'll build a minimal version of that pattern in pure Python. No LangChain, no vector database, no dependencies at all: just the standard library. It runs completely offline using TF-IDF, and if you set an OPENAI_API_KEY it transparently switches to real embeddings and a chat model. The full code is about 300 lines across three files.

Step 1: Chunk the documents

Embeddings work on passages, not whole files, so ingestion starts by splitting each document into chunks. A simple paragraph-aware splitter is good enough for a demo: accumulate paragraphs until you approach the size limit, and window-split any single paragraph that's too long.

def chunk_text(text, chunk_size=600, overlap=100):
    paras = [p.strip() for p in re.split(r"\n\s*\n", text) if p.strip()]
    chunks, buf = [], ""
    for para in paras:
        if len(para) > chunk_size:
            if buf:
                chunks.append(buf); buf = ""
            step = max(1, chunk_size - overlap)
            for i in range(0, len(para), step):
                chunks.append(para[i:i + chunk_size])
        elif not buf:
            buf = para
        elif len(buf) + len(para) + 2 <= chunk_size:
            buf = buf + "\n\n" + para
        else:
            chunks.append(buf); buf = para
    if buf:
        chunks.append(buf)
    return chunks

Each chunk keeps its source filename and position, which is what later lets us print citations like [Source: product-faq.txt, chunk 2].

Step 2: Build the index (TF-IDF offline, embeddings online)

Offline, we represent every chunk as a TF-IDF vector: term counts weighted by inverse document frequency, L2-normalised so cosine similarity is a plain dot product. It's fifty lines with collections.Counter and math.log — and for small, keyword-friendly document sets it works surprisingly well.

When an API key is available, we embed the chunks with OpenAI's text-embedding-3-small instead. Notice we don't even need the SDK; a JSON POST with urllib does the job:

def openai_embed(texts, api_key, model="text-embedding-3-small"):
    payload = json.dumps({"model": model, "input": texts}).encode()
    req = urllib.request.Request(
        "https://api.openai.com/v1/embeddings", data=payload,
        headers={"Authorization": f"Bearer {api_key}",
                 "Content-Type": "application/json"}, method="POST")
    with urllib.request.urlopen(req, timeout=60) as resp:
        data = json.loads(resp.read().decode())
    return [item["embedding"] for item in data["data"]]

Either way, the result is one boring index.json: the chunks, their vectors, the mode, and a relevance threshold (0.10 for TF-IDF — its cosine scores run lower than embedding similarities, so each mode gets its own threshold).

Step 3: The honest gate

This is the most important fifteen lines in the project. At query time we embed the question the same way we embedded the chunks, rank by cosine similarity — and then check the best score before generating anything:

scored = sorted(
    ((cosine(query_vec, vec), chunk)
     for chunk, vec in zip(chunks, vectors)),
    key=lambda pair: pair[0], reverse=True)[:3]

if not scored or scored[0][0] < threshold:
    print(FALLBACK)   # "I don't know -- that information isn't
    return            #  in the documents provided."

If nothing retrieved clears the bar, the program prints the fallback and stops. The language model is never called, so it cannot hallucinate an answer the documents don't support. When the gate passes, the chat model receives only the retrieved chunks and is instructed to answer strictly from them, citing every claim. In offline mode there's no model at all: the top cited excerpts are printed as the answer, which is less fluent but arguably even more transparent.

Step 4: Run it

python ingest.py
python ask.py "How long is the warranty for the Nimbus kettle?"

The demo ships with two small sample documents — a fictional product FAQ and a fictional company policy — so it works out of the box. The smoke test, run fully offline:

Retrieved sources:
  [0.383] [Source: product-faq.txt, chunk 1]
  [0.353] [Source: product-faq.txt, chunk 2]

Answer (top excerpts from your documents):
How long is the warranty? Every Nimbus Smart Kettle comes with a
2-year warranty that covers manufacturing defects... [Source: product-faq.txt, chunk 2]

And the question the documents can't answer:

python ask.py "What is the current stock price of Tesla?"

Best match score: 0.000 (threshold: 0.1)
I don't know -- that information isn't in the documents provided.

Exactly the behaviour we want: an answer with receipts when the documents support one, and a refusal when they don't.

Honest limitations

This is a teaching demo, not a product. TF-IDF matches keywords, not meaning, so a paraphrased question ("how long does the guarantee last?") can miss a chunk that says "warranty". A JSON file is a fine index for hundreds of chunks and a poor one for millions. There's no PDF parsing, no web ingestion, no conversation memory, no server — it's a CLI that answers one question at a time. Thresholds also need tuning per document set; 0.10 worked for the samples, not for every corpus.

Where to go next

The demo is MIT-licensed and lives at https://github.com/mahuan2928/rag-chatbot-demo — fork it, drop your own .txt and .md files into docs/, and break it.

If you need the production shape of this pattern — PDF and web-page ingestion, a FastAPI server with an embeddable chat widget, Chroma/pgvector/Pinecone instead of a JSON file, session memory, per-IP rate limiting, and config-driven setup — that's the full RAG Chatbot Starter Kit I sell here: https://mahuan220.gumroad.com/l/ohfaru ($49, one commercial project per license). It's the same architecture, hardened for client work.

Either way: gate your retrieval, cite your sources, and let your chatbot say "I don't know."