You Probably Don't Need a Vector Database for RAG

Say "RAG" out loud and a specific picture forms: an embedding model, a vector database like Pinecone or pgvector, and an embedding API call on every single query. It feels like the price of entry — real infrastructure, a real bill, a real operational surface — just to let a chatbot answer from your own documents.

For a lot of projects, that picture is overkill. RAG — retrieval-augmented generation — is, stripped to its core, three steps: find the text relevant to a question, paste it into the prompt, let the model answer from it. Nothing in that definition says the "find" step has to be a vector search. If your knowledge base covers a focused domain with a consistent vocabulary, plain keyword matching often retrieves the same chunks a vector search would — with no embeddings, no vector store, no extra network hop, and no database at all. This piece walks through how to build exactly that, and, just as importantly, when the simple version stops being enough.

When keyword retrieval is genuinely good enough

Semantic search — embeddings in a vector store — earns its reputation on fuzzy language. It understands that "feeling down" and "depression" are related, that "let go of an employee" means "fire." When users phrase things in words that don't appear in your documents, embeddings bridge the gap.

But many knowledge bases don't have that problem. In a focused domain — a specific area of law, a medical specialty, the docs for one product — the users tend to ask using the domain's own terms, because those are the terms the subject is about. When the query and the right passage share the actual vocabulary, keyword overlap finds it. And keyword retrieval brings things vectors can't: it's deterministic (the same query always returns the same chunks), it needs zero extra infrastructure, and there's no per-query embedding call adding latency and cost. The trade is real and it cuts both ways — you give up synonym understanding to gain simplicity, speed, and predictability. For a narrow corpus, that's frequently a trade worth making.

Building the retrieval

The whole retriever is small. Three moves: chunk the documents, turn each chunk into a set of keywords ahead of time, and score incoming queries against those sets.

Chunk by meaning, not by blind length. Instead of slicing documents into fixed token windows, split on structure — markdown ## headings, for example — so each chunk is a coherent section with a title. That heading is itself a strong keyword signal.

Tokenize and precompute keywords at build time. Lowercase the text, split into words, drop stopwords and very short tokens, and deduplicate. Do this once when you build the index, not on every request:

const STOPWORDS = new Set(["the", "a", "an", "and", "or", "but", "in", "on", /* … */]);

function tokenize(text: string): string[] {
  const words = text.toLowerCase().match(/[a-z]+/gi) ?? [];
  return [...new Set(words.filter(w => w.length > 2 && !STOPWORDS.has(w)))];
}

Score with Jaccard similarity. For a query, compare its token set against each chunk's precomputed keyword set: the size of the intersection over the size of the union. More shared words, higher score. Keep the top-K chunks with a non-zero score.

function retrieve(query: string, chunks: Chunk[], topK = 3): Chunk[] {
  const q = new Set(tokenize(query));
  if (q.size === 0) return [];

  return chunks
    .map(chunk => {
      const overlap = [...q].filter(t => chunk.keywords.includes(t)).length;
      const union = new Set([...q, ...chunk.keywords]).size;
      return { chunk, score: union ? overlap / union : 0 };
    })
    .filter(c => c.score > 0)
    .sort((a, b) => b.score - a.score)
    .slice(0, topK)
    .map(c => c.chunk);
}

That's the entire retrieval engine. If you want better ranking on longer queries, swap Jaccard for BM25, the classic keyword-ranking function search engines have used for decades — it weighs rarer terms more heavily and handles document length better, while staying pure keyword math with no embeddings.

Tuning what you retrieve

Two knobs decide retrieval quality, and they pull against each other. Chunk size: split too coarsely and a matched chunk is mostly irrelevant text that dilutes the prompt; split too finely and you lose the surrounding context that made the passage make sense. Sectioning by heading usually lands in a sensible middle, but if your sections are long, consider splitting further. Top-K: how many chunks you inject. Too few and you miss the relevant one; too many and you bury the answer in noise and burn tokens. For a focused bot, two or three well-matched chunks is often the sweet spot.

It helps to see one query end to end. Suppose the question is "why do I get angry at people close to me," the tokens (after dropping stopwords) are {angry, people, close}, and a chunk on a relevant concept has keywords {anger, shadow, projection, close, relationships}. The overlap is {close} — thin, and a reminder that exact-token matching is literal: "angry" and "anger" don't match unless you handle word forms (more on that below). When the match is good, you assemble the prompt by pasting the chunk in with its source, so the model can ground its answer and cite:

Relevant articles (cite these):

[Source 1]: Anger and the Shadow → https://example.org/wiki/shadow/
> The parts of ourselves we reject tend to surface as irritation at others...

User question: why do I get angry at people close to me?

The trick that removes the database: ship the knowledge in the bundle

Here's the move that makes this zero-infrastructure. At build time, generate a source file — say knowledge.ts — containing your chunks and their precomputed keywords as a plain array, and import it into the app. When you deploy, the entire knowledge base ships inside the code bundle and lives in memory. Search is an in-memory array scan: no database connection, no network call, results in single-digit milliseconds.

docs/**/*.md  ──►  build-knowledge script  ──►  knowledge.ts  (chunks + keywords)
                                                      │
                                              imported into the app,
                                              loaded into memory on deploy

It sounds reckless and it works beautifully at the right scale. A few hundred chunks and a few hundred kilobytes is nothing for a modern runtime. Be honest about the ceiling, though: this only works while the knowledge base is small enough to fit comfortably in your deploy bundle and in memory. Serverless platforms cap bundle size (often around a megabyte or so on free tiers), and you don't want to hold hundreds of megabytes of text in a worker. Small, stable corpus: embed it and enjoy the zero-ops search. Large or fast-growing corpus: that's one of the signals you've outgrown this approach.

Wiring up the rest, without a backend

The retriever is the interesting part; the surrounding bot is mostly small, careful pieces. Building it on a serverless edge platform (Cloudflare Workers, here) keeps the no-backend theme going, but it imposes a few constraints worth knowing.

State lives outside the function. Edge functions are stateless — every request is a clean slate — so conversation history has to be stored externally. A key-value store (Cloudflare KV) is a natural fit: one key per user, a cap on how many messages you keep, and a TTL so old sessions expire on their own.

async function addMessage(kv: KVNamespace, userId: number, msg: Message) {
  const history = JSON.parse((await kv.get(`session:${userId}`)) ?? "[]");
  history.push(msg);
  await kv.put(`session:${userId}`, JSON.stringify(history.slice(-20)), {
    expirationTtl: 86400 * 7, // 7 days
  });
}

Call the model with fetch, not a heavy SDK. Edge runtimes aren't full Node.js, and a big vendor SDK may not run there or may drag in a pile of dependencies. Since most providers expose an OpenAI-compatible endpoint, a tiny fetch wrapper is more robust than any SDK — and lets you swap providers by changing a URL:

const res = await fetch("https://api.groq.com/openai/v1/chat/completions", {
  method: "POST",
  headers: { Authorization: `Bearer ${apiKey}`, "Content-Type": "application/json" },
  body: JSON.stringify({ model, messages, temperature: 0.7, max_tokens: 2048 }),
});

Inject the retrieved chunks with their sources so the model can cite. Format each chunk with its title and a URL back to the source, then hand them to the model alongside the question — and instruct it to cite only the URLs it was given, never invented ones.

Two non-obvious lessons round it out. First, on a webhook-driven bot (Telegram, say), always return HTTP 200 — even when something failed internally. A non-2xx tells the platform the message wasn't delivered, so it retries with backoff and buries your function in duplicate updates. Second, store the original user message in history, not the version you stuffed full of retrieved context. If you save the augmented prompt, every following turn drags whole articles along with it, and within a few exchanges you've blown the model's context window. Retrieve fresh each turn; remember only what the user actually said.

Updating the knowledge is two commands

A quietly large benefit of baking knowledge into the bundle: updates are trivial. Add or edit a document, then rebuild and redeploy.

build   # regenerates knowledge.ts from your docs
deploy  # ships the updated bundle

No migrations, no re-indexing job, no embedding batch to re-run and pay for. If the docs live in their own repository, a CI job can rebuild and redeploy on every change, so editing a markdown file is all it takes to update what the bot knows. Compare that to the vector path, where changing your chunking or embedding model means re-embedding the entire corpus.

Know when it's good enough — measure it

"Keyword retrieval is fine for a narrow domain" is a claim you should verify on your corpus, not take on faith. The check is cheap: write down a couple dozen realistic questions and, for each, the document you'd expect to be retrieved. Run them through the retriever and count how often the right source lands in the top-K — that's your hit rate. Do it again whenever you change chunking, tokenization, or top-K, and you can see whether a tweak actually helped instead of guessing.

That same harness is your signal for the whole decision in this article. If keyword retrieval hits, say, 90%+ of your test questions, you're done — no vectors needed. If it's missing a lot, look at why it misses before reaching for embeddings: very often the failures are the same handful of vocabulary mismatches, which have a cheaper fix than a vector store.

Cheaper fixes before you reach for vectors

When keyword matching misses, it's usually because the user's word and the document's word are forms of the same idea, not different ideas. You can close most of that gap without embeddings:

• Stemming or lemmatization. Reduce words to a root before matching, so "angry," "anger," and "angrier" collapse to one token, as do "running"/"ran"/"runs." A standard stemmer is a small library and turns a lot of near-misses into hits.
• A small synonym map. For a focused domain you usually know the handful of equivalences that matter — expand the query (or the chunk keywords) with them. Map "fired" → "termination," your product's nickname → its formal name, the common-language term → the clinical one. A few dozen entries can outperform a generic embedding model on your specific jargon.
• BM25 over plain Jaccard. As mentioned, it ranks better on longer queries by weighting rare, distinctive terms.

These are the middle rungs of the ladder. They keep the determinism and the zero infrastructure while recovering much of what naive keyword matching loses — and they're worth exhausting before you take on a vector store.

When you actually do want vectors

This is not an argument against vector search — it's an argument against reaching for it by default. There are clear signals that you've crossed into territory where embeddings earn their keep:

Signal	Keyword (Jaccard/BM25)	Vectors / hybrid
Users speak the domain's own terms	✅ great	overkill
Synonym-heavy or vague queries ("feeling stuck")	misses	✅ understands meaning
Corpus fits in a deploy bundle / memory	✅ ship it in code	—
Large or fast-growing corpus	outgrows it	✅ needs a real store
Cross-language retrieval	weak	✅ embeddings shine
Need determinism / zero infra / zero cost	✅	adds infra + per-query cost

When you do make the jump, you don't have to abandon what you built: hybrid search — keyword (BM25) and vector, with the scores combined — consistently beats either alone, and is the standard production answer. And you can often stay on one platform: an edge provider may offer a managed vector index (Cloudflare's Vectorize) and an embeddings model (Workers AI) right next to the function you're already running, so "add vectors" doesn't mean "add a new vendor."

If you're self-hosting in Go, I personally use gosqlite.com (the pure-Go, CGo-free github.com/go-again/sqlite package) across a few projects for the same reason — it puts vector search, BM25 full-text, and hybrid ranking inside the same SQLite file your app already uses, so "add vectors" stays a one-process change instead of a new vendor or a new daemon.

One responsibility note, since "answer from a knowledge base" bots increasingly cover sensitive domains: if yours touches health, mental health, or anything where a user might be in crisis, build an explicit safety path that surfaces real helpline resources rather than relying on the model to improvise. That's a design requirement, not a nice-to-have — and it's independent of how you do retrieval.

The reframe

The reason "you need a vector database for RAG" became conventional wisdom is that the demos that made RAG famous were built on it. But the demo's architecture isn't a law. RAG is just "retrieve, then generate," and retrieval is a spectrum: at one end, an in-memory keyword match over a few hundred chunks baked into your code; at the other, a managed vector index over millions of documents. Start at the simple end. For a focused corpus the cheap, deterministic, zero-infrastructure version is frequently indistinguishable from the expensive one in answer quality — and you can always graduate to vectors the day your domain's language gets fuzzy or your corpus gets big. Pay for that machinery when the problem demands it, not because a diagram told you RAG looks a certain way.