Every RAG tutorial runs on a laptop with a few hundred documents, and it feels effortless. Then you point it at a real corpus — a company's docs, years of tickets, a knowledge base — and suddenly memory is the whole conversation.
RAG doesn't scale by CPU. It scales by RAM.
Why the index wants memory
Retrieval works by turning every chunk of text into an embedding — a vector, a few hundred to a couple thousand numbers long. Search means comparing your query vector against all of them, fast. "Fast" is the operative word: for low latency the index needs to live in RAM. On disk it works, but every query pays a penalty, and low-latency retrieval was the point of self-hosting in the first place.
So the memory bill scales with two things: how many chunks you have, and how wide each vector is.
Real numbers, roughly
Measure your own — dimension and index type move this a lot — but as a starting feel:
A few hundred thousand embeddings — comfortable in 2–4 GB. A personal knowledge base, a single product's docs.
Low millions — with the app, the model client, and the OS around it, plan for 16–32 GB. This is a serious company knowledge base or a multi-source RAG.
Tens of millions, or high-dimension vectors — now you're at 48–80 GB, and past that across several boxes. Large document estates, multi-tenant retrieval, or you're keeping several indexes hot at once.
A multi-agent system that also holds a big index stacks both costs on the same box — that's how a 32 GB plan turns into a 64 GB one quietly.
The engine choice, briefly
If you already run Postgres, pgvector is the least-effort option — it's an extension, not a new service to babysit. When you have millions of vectors and want fast filtered search, a dedicated engine like Qdrant or Weaviate earns the separate process. Don't over-engineer it on day one; run what you already operate and split it out when search actually slows.
Why bother self-hosting
Two reasons people actually do this, and neither is "to save a few dollars":
Privacy. Embeddings aren't abstract — they encode the text they came from. Your docs, your customers' content, your internal notes, turned into vectors and shipped to a third party's servers. Self-hosting keeps that on a machine you control. If the data is sensitive enough that you're also paying in crypto with no KYC, a managed vector cloud undoes the whole point.
Flat cost. Managed vector services bill by vectors stored and queries run. A VPS is one monthly number and you can hammer it as hard as you like. At scale, predictable beats metered.
What this means for sizing
Start by measuring your corpus, not by guessing. Get your embedding count and dimension, load a sample, watch the resident memory, extrapolate. Then pick a plan with headroom for the index plus everything around it — the app, the model client, room to grow.
For anything past a couple million vectors held privately, the Pro line runs 32 to 80 GB with a dedicated IP and nightly backups, which matters when the index is the product and losing it hurts.
Top comments (4)
The multi-agent stacking point is something that bit us. We had a graph-based entity resolution layer running alongside the vector index on the same box, and the combined resident set hit 48 GB on a corpus we'd estimated at 20 GB because we hadn't accounted for the adjacency list the graph engine keeps hot. The vector side was close to the estimates in this table, but the graph overlay added roughly 60% on top. We ended up splitting them onto separate hosts, which was annoying but honestly probably the right call for query isolation too.
That graph overlay blind spot is a great addition — thanks for putting real numbers on it. The adjacency-list-stays-hot problem never shows up in vector-store sizing math because everyone benchmarks the index in isolation and forgets the graph engine is holding its own working set right next to it.
The 60% figure tracks with what we see when a graph layer co-locates with embeddings — the two access patterns are almost adversarial. The vector side wants big sequential reads into a mostly-static index; the graph side wants random pointer-chasing across a mutable structure that grows with edge density, not node count. So the graph's memory scales with how connected your entities are, which is exactly what you can't read off corpus size. You estimated by data volume; the adjacency list bills you by relationship count.
And splitting was probably right beyond just the RAM — a sequential-scan workload and a random-access one sharing the same NUMA node interfere on tail latency in their own way, separate from the memory pressure. Query isolation buys you predictability, not just headroom.
Genuinely curious: when you split them, did you size the graph host by trial, or did you land on a heuristic that held? The adjacency memory is the one number I've never seen anyone estimate cleanly up front — it always seems to come down to "run it and watch RSS."
Mostly trial, but we'd measured peak edge count after a full ingest and used 20 bytes per pointer as a rough adjacency estimate, doubled for the hot set. That gave us a starting bid before provisioning rather than guessing blind. What actually tripped us wasn't the initial sizing but a later batch that added a lot of cross-entity edges; sparse-to-connected spikes are invisible in corpus volume metrics, so the "stable" corpus suddenly wasn't.
The 20-bytes-per-pointer-times-peak-edges heuristic is genuinely useful — that's the first concrete adjacency estimate I've seen someone actually commit to before provisioning instead of hand-waving. Doubling for the hot set is a nice touch too; the resident portion is what bites you, not the total.
And the sparse-to-connected spike is the real trap, because it breaks the one assumption everyone builds sizing on — that a "stable corpus" means stable memory. It doesn't. A batch that adds no new entities but heavily cross-links existing ones grows your resident set without moving a single metric you're probably watching. Document count flat, byte count flat, edge count quietly up 3x. You'd only catch it if you were graphing edge density over time, and almost nobody is.
The thing that makes this operationally nasty is that it's non-monotonic in a way disk isn't — you can't just provision for peak-so-far and relax, because the next connectivity batch redefines peak. Which is exactly the argument for treating the graph host's memory as a variable you top up rather than a fixed size you buy once. Match the resource to the workload's real curve, not its average.
Thanks for spelling out the actual numbers — this is the kind of war story that saves someone else a 2am OOM.