RAG without a vector database: fastembed + numpy, cosine by hand

title: RAG without a vector database: fastembed + numpy, cosine by hand
published: false
description: "JobCopilot part 2: I skipped the vector database. A 33 MB embedding model, numpy, one dot product — and one retrieval bug a tutorial gave me for free."
tags: python, ai, fastembed, numpy

I'm building JobCopilot in public: a small Python agent that reads a job description, pulls the most relevant chunks of my resume, and lets a skeptical-recruiter prompt do the matching. No framework — raw code first, abstraction only when the pain justifies it. Last week I wrote about the structured-output layer: pydantic, typed errors, retry. This week: the step before that call — deciding which parts of my resume deserve to be in the prompt at all.

I came to Python from React. My first instinct was infrastructure-shopping: every "build your own RAG" tutorial reaches for the same cast — a vector database, a chunking framework, an embeddings API. For my scale, a resume that chunks into single digits and one job posting at a time, that's not architecture. That's a second project to babysit, with an API key attached.

So I skipped all of it: fastembed for embeddings, numpy for search, a 33 MB model instead of a server. It survived every job posting I threw at it — after I fixed one retrieval bug a tutorial handed me for free.

1. Chunks in, embeddings out — the one-time script

The index side of "RAG" turned out to be embarrassingly small. This is the store-building script — I run it once whenever my resume changes:

"""Build my resume store: chunk once, embed once, keep for later."""
import json
import re
from pathlib import Path

import fastembed
import numpy as np

MODEL = "BAAI/bge-small-en-v1.5"   # 384 dims, ~33 MB on disk, CPU-first
TOP_K = 8

_model = None
def _embedder() -> fastembed.TextEmbedding:
    global _model
    if _model is None:                      # lazy: don't pay on import
        _model = fastembed.TextEmbedding(model_name=MODEL)
    return _model

def chunk_resume(md: str) -> list[str]:
    """Split by ## headings; each section becomes one chunk."""
    parts = re.split(r"\n(?=## )", md)
    return [p.strip() for p in parts if p.strip()]

def build_store(md_path: str, out_path: str) -> dict:
    md = Path(md_path).read_text()
    chunks = chunk_resume(md)
    vecs = list(_embedder().embed(chunks))
    emb = np.stack(vecs)
    emb = emb / np.linalg.norm(emb, axis=1, keepdims=True)   # normalize once
    Path(out_path).write_text(json.dumps({"chunks": chunks, "emb": emb.tolist()}))
    return {"chunks": len(chunks), "dims": emb.shape[1]}

if __name__ == "__main__":
    print(build_store("resume.md", "resume_store.json"))

Why each piece earns its line:

• MODEL = "BAAI/bge-small-en-v1.5" — 384-dimension vectors, ~33 MB on disk, runs on CPU, no API key, no runtime network (the model downloads once, then it's local). For a single-language corpus at this scale, "small embedding model" isn't a compromise — it's the correct answer.
• Lazy _embedder() — the model loads on first use, not on import, so scripts and tests that never embed anything don't pay for it.
• re.split(r"\n(?=## )", md) — one regex line does the chunking: each ## section keeps its heading attached to its body, bodyless headings get dropped by the emptiness check. My resume lands at a handful of chunks — no chunking framework required.
• Normalize once, at build time (emb / norm) — this pays off in the runtime section: cosine similarity on unit vectors collapses into a plain dot product. I'll come back to exactly why.

One run of python store_resume.py writes a JSON file: chunk strings plus a 384-float vector each. If you can regenerate your index with one command, it's not infrastructure — it's a build artifact. That reframe took most of the mystique out of RAG for me.

2. The whole runtime is 10 lines: scores = emb @ q

The query side — this is retrieve.py, the entire retrieval engine:

"""Runtime: pull the TOP_K chunks that matter for this job posting."""
import json
from pathlib import Path

import numpy as np

from store_resume import TOP_K, _embedder

def retrieve(query: str, store_path: str = "resume_store.json") -> list[tuple[str, float]]:
    store = json.loads(Path(store_path).read_text())
    chunks: list[str] = store["chunks"]
    emb = np.array(store["emb"])

    q = np.stack(list(_embedder().embed([query])))[0]
    q = q / np.linalg.norm(q)

    scores = emb @ q                      # cosine by hand: one dot product
    order = np.argsort(scores)[::-1][:TOP_K]

    return [(chunks[i], float(scores[i])) for i in order]

emb is (n_chunks, 384), q is (384,). One matmul broadcasts the query over every chunk row; the argsort sorts descending and keeps the top 8.

Why no vector DB, honestly answered: at single-digit chunks the whole "index" is a matrix, and a full scan is one matmul — microseconds. Ann indexes buy speed at hundreds of thousands of vectors and pay with recall errors plus another service to run. At my scale that trade is upside-down: complexity buys speed I don't need.

Coming from JavaScript, I'd have typed the cosine formula out manually, looping. The numpy docs (the senior-er voice in my head) said: if every vector is unit length, the dot product IS the cosine similarity. That's why build_store and retrieve both normalize immediately.

What the scores actually look like for the query "Kubernetes, Terraform, SRE on-call" against a small demo resume (6 chunks — this is my sandbox example, not the real store):

0.784  '## Kubernetes & SRE track record...'
0.573  '## Side projects...'
0.550  '# Demo Resume'
0.504  '## Experience...'
0.496  '## Skills...'
0.487  '## Interests...'

The shape is the story: the relevant chunk breaks away at 0.78, then comes the noise band — chunks that don't match the query, they match the genre of resumes. Retrieval is only useful while that separation exists. Which brings me to the bug.

3. The bug that got me: TOP_K = 12 — ranking, downgraded to bulk-loading

Now the part I'm genuinely not proud of: my first working retrieval had TOP_K = 12, because a tutorial used it and I kept it. It felt robust — "grab a lot, the LLM will figure out relevance." The effect was the opposite.

The failure is invisible — no crash, no warning. argsort()[::-1][:12] just hands you a dozen chunks: the one that actually matched the posting, plus eleven that match the genre of job postings and resumes — Python keywords, generic resume lines. That tail was exactly what got pasted into the prompt, and the LLM, agreeable as ever, worked it into the missing-skills list with full confidence. Garbage in, confident garbage out.

The score line told the story in two numbers. With an over-fed TOP_K on my real store, the top of the list and the tail it dragged along looked like this:

0.635  '## Work Experience — the relevant one'
...
0.450  '## Languages — pure filler for this query'

A 0.185 spread where only the first row deserved attention; everything below ~0.55 was noise-band bulk. No separation. Retrieval had stopped ranking and started bulk-loading with extra steps. My fix was one line:

TOP_K = 8    # was 12 from a tutorial; enough for a resume, small enough to keep the gap

After:

0.635  '## Work Experience — the relevant one'
0.592  '## Technical Skills — adjacent, still on-topic'

And the 8-slot list ends in the languages/education band — nothing under 0.45 enters the prompt. The top row visibly separates again.

The rule I use now (embarrassingly late in my Python learning): TOP_K must stay small enough that the top-1 score visibly separates from the rest. The moment it stops separating, retrieval is decoration and the prompt is a firehose. On my resume, 8 is enough. And the tutorial never mentioned any of this — perfectly correct code can still be wrong, because a constant you copied was tuned for someone else's scale.

4. From ranked chunks to a prompt, with a character budget

Retrieval gives me ranked chunks with scores; the prompt needs one string. I could .join() the top 8 and move on — but the naive join has two failure modes: a monster section eating the prompt's attention budget, and a torn tail — a chunk cut in half by a [:2400] slice, half a bullet floating in the prompt.

build_context fixes both:

def build_context(query: str, store_path: str = "resume_store.json",
                  max_chars: int = 2400) -> str:
    """Join top chunks into one string for the prompt, and STOP when full."""
    out, size = [], 0
    for text, _score in retrieve(query, store_path):
        if size + len(text) > max_chars:
            break                  # a chunk that doesn't fit is dropped WHOLE
        out.append(text)
        size += len(text)
    return "\n\n".join(out)

Two details matter:

• Stop at max_chars even if that uses fewer than TOP_K chunks. A prompt-quality knob, not a storage quota: 2400 tightly selected characters beat 6000 of everything.
• Cut at chunk boundaries. A chunk that would overflow is skipped whole — no torn half-bullet floating in the prompt.

I tested a Represent this sentence for searching relevant passages: query prefix (documented for bge models) against my store: top score +0.014, but the gap to the second row shrank — a wash for ranking, which is what actually matters at my scale. So there's no query-prefix engineering in this pipeline; normalization is the only trick worth keeping.

Wrapping up

Retrieval without a vector DB: chunk by headings → embed once with a 33 MB model → normalize at build time → emb @ q at runtime → keep top-8 → join into a char-budgeted context. About 60 lines total — and I understand every one of them, which is more than I could say for a vector DB I don't need yet.

Three things this React-dev-learning-Python took from it:

• For small corpora, a "vector store" is a JSON file and a matmul. The tutorials don't tell you when your scale justifies something heavier — your own score gap does.
• Copy-pasted defaults have their owner's scale baked in. TOP_K = 12 was fine in the tutorial's corpus and quietly destructive in mine.
• The most honest measure isn't the top score — it's the gap between the top score and the next one. If the gap is gone, nothing downstream matters.

Next Tuesday, part 3: my first agent graph — nodes that return partial state patches instead of raising, routing on data instead of exceptions, and the LangGraph mental shift from "function calls function" to "state flows through a graph".

Follow if you want to watch the rest of the copilot get built — including the parts that break. Build log + daily code: t.me/novamind_hub.

How do you handle retrieval at small scale — still reaching for a vector DB, or plain numpy? And what default did you copy from a tutorial that quietly cost you?