Build a Tiny Retrieval System Before Reaching for a RAG Framework

Retrieval-augmented generation (RAG) is easier to understand when you first remove the generation. Build a tiny program that accepts a question, ranks a few notes, and returns the most relevant passage. Once retrieval is visible, adding a language model becomes a separate and understandable step.

This project uses Python and scikit-learn. It does not need an API key or a vector database.

What you will build

The corpus contains five short notes about a fictional command-line tool. Given this question:

How do I change where reports are saved?

the program should rank the note about output_dir above unrelated installation and authentication notes.

The pipeline is:

notes → TF-IDF vectors → question vector → cosine similarity → ranked passages

Prerequisites

• Python 3.10 or newer
• a virtual environment
• scikit-learn

Install the dependency:

python -m venv .venv
source .venv/bin/activate   # Windows PowerShell: .venv\Scripts\Activate.ps1
python -m pip install scikit-learn

Add the corpus

Create retrieve.py:

from dataclasses import dataclass

from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.metrics.pairwise import cosine_similarity


@dataclass(frozen=True)
class Note:
    title: str
    text: str


NOTES = [
    Note("Install", "Install Acorn CLI with pipx install acorn-cli."),
    Note("Login", "Run acorn auth login and paste a workspace token."),
    Note(
        "Report folder",
        "Set output_dir in acorn.toml to choose where generated reports are saved.",
    ),
    Note("Formats", "The report command supports json, markdown, and csv formats."),
    Note("Reset", "Run acorn config reset to restore the default configuration."),
]

Small, inspectable data is a feature here. You can tell whether a result is correct without trusting another system.

Index and rank the notes

Add this function:

def search(question: str, limit: int = 3) -> list[tuple[Note, float]]:
    documents = [note.text for note in NOTES]
    vectorizer = TfidfVectorizer(stop_words="english")

    document_vectors = vectorizer.fit_transform(documents)
    question_vector = vectorizer.transform([question])
    scores = cosine_similarity(question_vector, document_vectors)[0]

    ranked_indexes = scores.argsort()[::-1][:limit]
    return [(NOTES[index], float(scores[index])) for index in ranked_indexes]


if __name__ == "__main__":
    query = "How do I change where reports are saved?"
    for note, score in search(query):
        print(f"{score:.3f}  {note.title}: {note.text}")

Run it:

python retrieve.py

The exact scores can vary with library versions and text changes, but Report folder should be first. If it is not, print the vocabulary and scores before changing models or adding infrastructure.

What the code is doing

TfidfVectorizer gives more weight to terms that are useful in one note and less weight to terms common across many notes. cosine_similarity compares the direction of the question vector with each note vector. A larger score means more overlap under this representation.

This is lexical retrieval: related words with different spellings may not match. A question about a “destination directory,” for example, might rank poorly if the corpus only says “output folder.” That failure is useful because it exposes the job an embedding model would perform.

Turn retrieval into a RAG request

A RAG system adds a generation step after ranking. The application could assemble a request like this:

Answer only from the passages below. If they do not contain the answer,
say that the documentation does not answer the question.

[Report folder]
Set output_dir in acorn.toml to choose where generated reports are saved.

Question: How do I change where reports are saved?

The model does not perform authorization or decide which private notes the user may see. Filter the corpus before retrieval, using application identity and permissions. Also keep passage titles or IDs so the interface can show where an answer came from.

The original RAG paper describes combining a model’s parametric memory with retrieved non-parametric memory: Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks. This toy project implements only the retrieval half, which is exactly why it is a good first step.

Three experiments to try

1. Add a synonym failure

Ask, “Where is the destination directory configured?” Observe the ranking, then add “destination directory” to the relevant note. This shows the limits of lexical matching.

2. Split a long document

Replace one note with a page containing several topics. Compare retrieving the whole page with retrieving paragraph-sized chunks. Very large chunks can bury the matching sentence; very small chunks can lose context.

3. Add a threshold

Try an unrelated question such as “How do I invite a teammate?” If every score is near zero, return “no relevant passage” instead of sending arbitrary context to a model. Choose the threshold from a labeled set of questions, not from one example.

Common errors

• ModuleNotFoundError: activate the virtual environment and reinstall the dependency.
• Every score is 0.000: none of the question terms survived tokenization or appeared in the corpus.
• A generic note ranks first: inspect repeated words and consider improving the document text or using n-grams.
• Retrieval is correct but a later answer is wrong: preserve the ranked passages so you can evaluate retrieval and generation separately.

Frameworks become helpful when you need persistent indexes, multiple data sources, embedding services, or evaluation pipelines. Starting with a five-note retriever gives those components a purpose instead of turning them into unexplained boxes.