Say "RAG" out loud and a specific picture forms: an embedding model, a vector database like Pinecone or pgvector, and an embedding API call on every single query. It feels like the price of entry — real infrastructure, a real bill, a real operational surface — just to let a chatbot answer from your own documents.
For a lot of projects, that picture is overkill. RAG — retrieval-augmented generation — is, stripped to its core, three steps: find the text relevant to a question, paste it into the prompt, let the model answer from it. Nothing in that definition says the "find" step has to be a vector search. If your knowledge base covers a focused domain with a consistent vocabulary, plain keyword matching often retrieves the same chunks a vector search would — with no embeddings, no vector store, no extra network hop, and no database at all. This piece walks through how to build exactly that, and, just as importantly, when the simple version stops being enough.
When keyword retrieval is genuinely good enough
Semantic search — embeddings in a vector store — earns its reputation on fuzzy language. It understands that "feeling down" and "depression" are related, that "let go of an employee" means "fire." When users phrase things in words that don't appear in your documents, embeddings bridge the gap.
But many knowledge bases don't have that problem. In a focused domain — a specific area of law, a medical specialty, the docs for one product — the users tend to ask using the domain's own terms, because those are the terms the subject is about. When the query and the right passage share the actual vocabulary, keyword overlap finds it. And keyword retrieval brings things vectors can't: it's deterministic (the same query always returns the same chunks), it needs zero extra infrastructure, and there's no per-query embedding call adding latency and cost. The trade is real and it cuts both ways — you give up synonym understanding to gain simplicity, speed, and predictability. For a narrow corpus, that's frequently a trade worth making.
Building the retrieval
The whole retriever is small. Three moves: chunk the documents, turn each chunk into a set of keywords ahead of time, and score incoming queries against those sets.
Chunk by meaning, not by blind length. Instead of slicing documents into fixed token windows, split on structure — markdown ## headings, for example — so each chunk is a coherent section with a title. That heading is itself a strong keyword signal.
Tokenize and precompute keywords at build time. Lowercase the text, split into words, drop stopwords and very short tokens, and deduplicate. Do this once when you build the index, not on every request:
const STOPWORDS = new Set(["the", "a", "an", "and", "or", "but", "in", "on", /* … */]);
function tokenize(text: string): string[] {
const words = text.toLowerCase().match(/[a-z]+/gi) ?? [];
return [...new Set(words.filter(w => w.length > 2 && !STOPWORDS.has(w)))];
}
Score with Jaccard similarity. For a query, compare its token set against each chunk's precomputed keyword set: the size of the intersection over the size of the union. More shared words, higher score. Keep the top-K chunks with a non-zero score.
function retrieve(query: string, chunks: Chunk[], topK = 3): Chunk[] {
const q = new Set(tokenize(query));
if (q.size === 0) return [];
return chunks
.map(chunk => {
const overlap = [...q].filter(t => chunk.keywords.includes(t)).length;
const union = new Set([...q, ...chunk.keywords]).size;
return { chunk, score: union ? overlap / union : 0 };
})
.filter(c => c.score > 0)
.sort((a, b) => b.score - a.score)
.slice(0, topK)
.map(c => c.chunk);
}
That's the entire retrieval engine. If you want better ranking on longer queries, swap Jaccard for BM25, the classic keyword-ranking function search engines have used for decades — it weighs rarer terms more heavily and handles document length better, while staying pure keyword math with no embeddings.
Tuning what you retrieve
Two knobs decide retrieval quality, and they pull against each other. Chunk size: split too coarsely and a matched chunk is mostly irrelevant text that dilutes the prompt; split too finely and you lose the surrounding context that made the passage make sense. Sectioning by heading usually lands in a sensible middle, but if your sections are long, consider splitting further. Top-K: how many chunks you inject. Too few and you miss the relevant one; too many and you bury the answer in noise and burn tokens. For a focused bot, two or three well-matched chunks is often the sweet spot.
It helps to see one query end to end. Suppose the question is "why do I get angry at people close to me," the tokens (after dropping stopwords) are {angry, people, close}, and a chunk on a relevant concept has keywords {anger, shadow, projection, close, relationships}. The overlap is {close} — thin, and a reminder that exact-token matching is literal: "angry" and "anger" don't match unless you handle word forms (more on that below). When the match is good, you assemble the prompt by pasting the chunk in with its source, so the model can ground its answer and cite:
Relevant articles (cite these):
[Source 1]: Anger and the Shadow → https://example.org/wiki/shadow/
> The parts of ourselves we reject tend to surface as irritation at others...
User question: why do I get angry at people close to me?
The trick that removes the database: ship the knowledge in the bundle
Here's the move that makes this zero-infrastructure. At build time, generate a source file — say knowledge.ts — containing your chunks and their precomputed keywords as a plain array, and import it into the app. When you deploy, the entire knowledge base ships inside the code bundle and lives in memory. Search is an in-memory array scan: no database connection, no network call, results in single-digit milliseconds.
docs/**/*.md ──► build-knowledge script ──► knowledge.ts (chunks + keywords)
│
imported into the app,
loaded into memory on deploy
It sounds reckless and it works beautifully at the right scale. A few hundred chunks and a few hundred kilobytes is nothing for a modern runtime. Be honest about the ceiling, though: this only works while the knowledge base is small enough to fit comfortably in your deploy bundle and in memory. Serverless platforms cap bundle size (often around a megabyte or so on free tiers), and you don't want to hold hundreds of megabytes of text in a worker. Small, stable corpus: embed it and enjoy the zero-ops search. Large or fast-growing corpus: that's one of the signals you've outgrown this approach.
Wiring up the rest, without a backend
The retriever is the interesting part; the surrounding bot is mostly small, careful pieces. Building it on a serverless edge platform (Cloudflare Workers, here) keeps the no-backend theme going, but it imposes a few constraints worth knowing.
State lives outside the function. Edge functions are stateless — every request is a clean slate — so conversation history has to be stored externally. A key-value store (Cloudflare KV) is a natural fit: one key per user, a cap on how many messages you keep, and a TTL so old sessions expire on their own.
async function addMessage(kv: KVNamespace, userId: number, msg: Message) {
const history = JSON.parse((await kv.get(`session:${userId}`)) ?? "[]");
history.push(msg);
await kv.put(`session:${userId}`, JSON.stringify(history.slice(-20)), {
expirationTtl: 86400 * 7, // 7 days
});
}
Call the model with fetch, not a heavy SDK. Edge runtimes aren't full Node.js, and a big vendor SDK may not run there or may drag in a pile of dependencies. Since most providers expose an OpenAI-compatible endpoint, a tiny fetch wrapper is more robust than any SDK — and lets you swap providers by changing a URL:
const res = await fetch("https://api.groq.com/openai/v1/chat/completions", {
method: "POST",
headers: { Authorization: `Bearer ${apiKey}`, "Content-Type": "application/json" },
body: JSON.stringify({ model, messages, temperature: 0.7, max_tokens: 2048 }),
});
Inject the retrieved chunks with their sources so the model can cite. Format each chunk with its title and a URL back to the source, then hand them to the model alongside the question — and instruct it to cite only the URLs it was given, never invented ones.
Two non-obvious lessons round it out. First, on a webhook-driven bot (Telegram, say), always return HTTP 200 — even when something failed internally. A non-2xx tells the platform the message wasn't delivered, so it retries with backoff and buries your function in duplicate updates. Second, store the original user message in history, not the version you stuffed full of retrieved context. If you save the augmented prompt, every following turn drags whole articles along with it, and within a few exchanges you've blown the model's context window. Retrieve fresh each turn; remember only what the user actually said.
Updating the knowledge is two commands
A quietly large benefit of baking knowledge into the bundle: updates are trivial. Add or edit a document, then rebuild and redeploy.
build # regenerates knowledge.ts from your docs
deploy # ships the updated bundle
No migrations, no re-indexing job, no embedding batch to re-run and pay for. If the docs live in their own repository, a CI job can rebuild and redeploy on every change, so editing a markdown file is all it takes to update what the bot knows. Compare that to the vector path, where changing your chunking or embedding model means re-embedding the entire corpus.
Know when it's good enough — measure it
"Keyword retrieval is fine for a narrow domain" is a claim you should verify on your corpus, not take on faith. The check is cheap: write down a couple dozen realistic questions and, for each, the document you'd expect to be retrieved. Run them through the retriever and count how often the right source lands in the top-K — that's your hit rate. Do it again whenever you change chunking, tokenization, or top-K, and you can see whether a tweak actually helped instead of guessing.
That same harness is your signal for the whole decision in this article. If keyword retrieval hits, say, 90%+ of your test questions, you're done — no vectors needed. If it's missing a lot, look at why it misses before reaching for embeddings: very often the failures are the same handful of vocabulary mismatches, which have a cheaper fix than a vector store.
Cheaper fixes before you reach for vectors
When keyword matching misses, it's usually because the user's word and the document's word are forms of the same idea, not different ideas. You can close most of that gap without embeddings:
- Stemming or lemmatization. Reduce words to a root before matching, so "angry," "anger," and "angrier" collapse to one token, as do "running"/"ran"/"runs." A standard stemmer is a small library and turns a lot of near-misses into hits.
- A small synonym map. For a focused domain you usually know the handful of equivalences that matter — expand the query (or the chunk keywords) with them. Map "fired" → "termination," your product's nickname → its formal name, the common-language term → the clinical one. A few dozen entries can outperform a generic embedding model on your specific jargon.
- BM25 over plain Jaccard. As mentioned, it ranks better on longer queries by weighting rare, distinctive terms.
These are the middle rungs of the ladder. They keep the determinism and the zero infrastructure while recovering much of what naive keyword matching loses — and they're worth exhausting before you take on a vector store.
When you actually do want vectors
This is not an argument against vector search — it's an argument against reaching for it by default. There are clear signals that you've crossed into territory where embeddings earn their keep:
| Signal | Keyword (Jaccard/BM25) | Vectors / hybrid |
|---|---|---|
| Users speak the domain's own terms | ✅ great | overkill |
| Synonym-heavy or vague queries ("feeling stuck") | misses | ✅ understands meaning |
| Corpus fits in a deploy bundle / memory | ✅ ship it in code | — |
| Large or fast-growing corpus | outgrows it | ✅ needs a real store |
| Cross-language retrieval | weak | ✅ embeddings shine |
| Need determinism / zero infra / zero cost | ✅ | adds infra + per-query cost |
When you do make the jump, you don't have to abandon what you built: hybrid search — keyword (BM25) and vector, with the scores combined — consistently beats either alone, and is the standard production answer. And you can often stay on one platform: an edge provider may offer a managed vector index (Cloudflare's Vectorize) and an embeddings model (Workers AI) right next to the function you're already running, so "add vectors" doesn't mean "add a new vendor."
If you're self-hosting in Go, I personally use gosqlite.com (the pure-Go, CGo-free github.com/go-again/sqlite package) across a few projects for the same reason — it puts vector search, BM25 full-text, and hybrid ranking inside the same SQLite file your app already uses, so "add vectors" stays a one-process change instead of a new vendor or a new daemon.
One responsibility note, since "answer from a knowledge base" bots increasingly cover sensitive domains: if yours touches health, mental health, or anything where a user might be in crisis, build an explicit safety path that surfaces real helpline resources rather than relying on the model to improvise. That's a design requirement, not a nice-to-have — and it's independent of how you do retrieval.
The reframe
The reason "you need a vector database for RAG" became conventional wisdom is that the demos that made RAG famous were built on it. But the demo's architecture isn't a law. RAG is just "retrieve, then generate," and retrieval is a spectrum: at one end, an in-memory keyword match over a few hundred chunks baked into your code; at the other, a managed vector index over millions of documents. Start at the simple end. For a focused corpus the cheap, deterministic, zero-infrastructure version is frequently indistinguishable from the expensive one in answer quality — and you can always graduate to vectors the day your domain's language gets fuzzy or your corpus gets big. Pay for that machinery when the problem demands it, not because a diagram told you RAG looks a certain way.
Top comments (5)
This matches what we found running retrieval over a fairly narrow internal docs corpus — BM25 quietly beat our first pgvector setup on the queries that mattered, and the win wasn't just cost. The thing nobody warns you about with a vector store isn't the per-query embedding call, it's the ingestion side: every doc edit means re-chunking and re-embedding, and if that job silently lags, retrieval degrades in a way that's invisible until someone gets a stale answer. Keyword indices don't have that failure mode — reindex is cheap and observable.
The place I've seen the simple version actually break is acronym/synonym drift: the moment two teams call the same thing by different names, lexical overlap collapses and you genuinely need embeddings to bridge it. Curious where you draw that line in practice — do you reach for a hybrid BM25-plus-rerank step before a full vector store, or jump straight to embeddings once keyword recall drops?
The determinism point is underrated: keyword retrieval returning the same chunks for the same query makes failures reproducible, which is most of the battle when a retrieval bug ships. Chunking on markdown headings so the heading itself becomes a keyword signal is a nice touch, though Jaccard on token sets will punish a right passage that phrases the concept in different words, which is exactly the synonym gap you flagged. Before reaching for embeddings I would measure recall on a small labeled query set first, since a domain with tight vocabulary often does not clear the bar where a vector store earns its cost.
The determinism you list as a nice-to-have is actually the strongest argument, and it is not about consistent results, it is about legible failures. When BM25 misses, you can read exactly why: the query term was not in the doc, or the acronym did not match. When a vector search misses, you cannot read why it ranked the right doc fourth, the failure is opaque. For a small team iterating, a retrieval layer whose misses you can debug by eye beats one whose misses you have to guess at, and that barely shows up in the usual vectors-are-smarter framing.
And the vocabulary drift that pushes people to vectors has a cheaper fix that keeps the determinism: generate the questions each chunk answers at ingestion and index those keywords too. Now the user's phrasing matches the synthetic question instead of the source text, which closes most of the synonym and acronym gap without an embedding call. It pushes the graduate-to-vectors line much further out than corpus size suggests, because the real trigger was never size, it was the gap between how users ask and how the docs are written.
The framing I'd add on top of this: it isn't BM25 vs embeddings, it's BM25 first and then measure what it can't reach. Take your eval set, run keyword retrieval, and bucket the misses - the queries where the correct chunk shares literally no vocabulary with the question. That residual is the only thing embeddings are buying you, and now you can size it. For a lot of narrow corpora it's single digits, and paying for a vector store plus an embedding hop on every query to fix 3% of queries is a bad trade you can now see clearly.
When the residual is big enough to matter, keep BM25 as the deterministic base and add embeddings as a top-up on just the vocabulary-gap queries, so most traffic stays legible and cheap. The determinism point above is exactly why: you want the common path to fail in a way you can read.
Building on the BM25-first framing: the step after "measure what it can't reach" does not have to reintroduce a vector database. Reciprocal rank fusion lets you keep BM25 as the base and fuse in a second ranker by rank position, not score, so you get the recall you were missing without giving up the determinism the thread keeps pointing at. RRF is order-based, so two runs on the same query still return the same fused order, which is the property that makes failures reproducible.
The practical win is that the second signal can be anything, another lexical index, a metadata filter, a small cross-encoder, and you only reach for embeddings if measurement actually shows a gap they close. I wrote up a no-vector-DB version of exactly this (RRF over multiple signals) here: gist.github.com/renezander030/41af...