Why a System That Just Stores Numbers Becomes One of the Most Expensive Things You Run
Vector database day one felt like a non-event. A few thousand vectors, a single node, queries coming back in well under a hundred milliseconds. I remember thinking this was the easy part of the pipeline — extraction had bugs, chunking had trade-offs, metadata had gaps, but the vector database just... worked. Store a vector, search a vector, done.
Then real production load hit it. Not a demo. Not a test set. The actual document volume, the actual query traffic, running continuously instead of in short bursts.
Latency crept up first. Queries that used to return instantly started taking noticeably longer, especially during the hours ingestion jobs were running alongside live traffic. Then memory usage started climbing in a way that didn't match how much data I thought I'd added. Then a bulk update — adding a batch of new documents — locked things up long enough that new content wasn't searchable for a while after it was technically "in" the database.
None of that happened on day one, with a few thousand vectors on a single node. All of it showed up once the numbers got real. That's when it hit me: I'd been thinking of the vector database as storage. It isn't. It's a live system that has to stay in memory, stay indexed, and stay searchable in milliseconds, all day, every day — and every one of those requirements taxes a different resource as scale goes up.
The Idea This Whole Article Is About
A vector database isn't a place you keep vectors. It's a system you keep running — in memory, 24/7 — and storage, indexing, querying, and updating each tax it in a completely different way.
Embeddings get created once per chunk. The vector database has to hold the result of that forever, and answer a similarity search against it instantly, no matter how large it's grown or how often it's changing underneath you. That's a fundamentally different kind of cost than anything earlier in the pipeline — it doesn't happen once. It runs continuously, whether or not anyone's asking a question right now.
Cost 1 — Storage Is Bigger Than It Looks
The first surprise: the index costs more space than the raw vectors do.
A raw vector is just numbers — dimension count times bytes per number. But most vector databases don't search raw vectors directly at scale. They build an index structure on top (commonly something like HNSW or IVF) to make search fast, and that index structure carries its own overhead — graph connections, cluster centroids, auxiliary bookkeeping — sitting on top of the vectors themselves.
Raw vectors
│
▼
Index structure built on top (HNSW / IVF / etc.)
│
▼
Index overhead adds meaningfully more memory than the vectors alone
│
▼
Actual footprint is noticeably bigger than "vector count × dimension size" suggests
Illustrative, not measured: a million vectors at a common embedding size might sit around a few gigabytes as raw numbers — but once you add index overhead, that same million vectors can realistically occupy a good deal more than that in actual memory. The exact multiplier depends entirely on the index type and settings you choose. The point isn't the specific number — it's that "vector count times dimension" undercounts the real footprint every time.
Cost 2 — RAM Is the Real Recurring Bill
Storage on disk is cheap. Storage in memory is not — and for the index to answer queries in milliseconds, most of it needs to live in RAM, not on disk.
Index needs to stay in memory for fast search
│
▼
More vectors → more RAM required to hold the index
│
▼
RAM is the most expensive resource per gigabyte in most infrastructure
│
▼
This cost exists 24/7, whether or not a single query is running right now
This is the cost that makes a vector database feel different from a normal database. A normal database can happily keep most of its data on disk and page things in as needed. A vector index that has to page in and out of memory constantly to answer a similarity search stops being fast — so the whole point of paying for RAM is to avoid that. You're not paying to store the data. You're paying to keep it instantly reachable.
Cost 3 — Building the Index Isn't Free Either
Adding a vector to a flat list is trivial. Adding a vector to an HNSW graph means recalculating where it fits relative to its neighbors — that's real CPU work, and it happens for every single vector you add, not just at the end.
New batch of documents ingested
│
▼
Each new vector needs to be placed correctly in the index graph
│
▼
CPU-intensive, scales with both index size and batch size
│
▼
Large bulk ingests can visibly slow down or briefly block live queries
This is the "bulk update locked things up" moment from the opening story. Index construction isn't a side effect of storage — it's an active, ongoing computation that has to happen every time new data arrives, and it competes for the same CPU and memory that's simultaneously trying to serve live search traffic.
Cost 4 — Querying Gets More Expensive as the Index Grows
Even with a good index, search compute isn't free, and it isn't flat as data grows.
Small index
│
▼
Search touches a small neighborhood of the graph → fast, cheap
Large index (same query, much more data)
│
▼
Search has to traverse more of the graph to find the same quality of match
│
▼
More compute per query, potentially slower response, especially under concurrent load
Approximate indexes exist specifically to keep this cost from growing linearly with data size — but "approximate" is doing real work in that sentence. There's a dial (often called something like ef or nprobe depending on the system) that trades search thoroughness for speed. Turn it down for speed, and you're now trading a little bit of retrieval accuracy for it — which quietly becomes a different cost, a few episodes back: a slightly worse match retrieved, a slightly less complete answer, a retry.
Cost 5 — Latency at Scale
This is really Costs 2 through 4, felt by an actual user.
More vectors + more concurrent queries
│
▼
More RAM pressure, more CPU contention, larger graph to traverse
│
▼
Response time per query creeps upward
│
▼
Users notice the assistant feels "slower" than it used to
Nothing has to be broken for this to happen. Nothing throws an error. The system just gradually gets heavier as it grows, and unless someone is watching query latency over time, this cost hides in plain sight — until a user finally says the bot feels sluggish, and there's no single bug to point at, just accumulated scale.
Cost 6 — Replicas and High Availability
A single node holding your entire index is one hardware failure away from your whole RAG system going down. So production systems run replicas — multiple copies of the same index, able to serve queries in parallel and take over if one node fails.
1 node → 1× RAM cost, 1× CPU cost, no redundancy
3 nodes → 3× RAM cost, 3× CPU cost, real redundancy and more query throughput
This is a straightforward multiplier, and it's easy to underestimate because it doesn't feel like a "new" cost — it feels like the same cost, just safer. But every one of Costs 1 through 5 above gets multiplied by however many replicas you decide you need for uptime and query throughput.
Putting a Number on It
Same as Episode 3 — a small worked example makes this easier to feel than a paragraph of description alone. These figures are illustrative, not measured from a real system — the point is the shape of the trade-off, not the exact digits.
Example
1 million vectors, single replica
│
▼
Raw vectors: a few GB
Index overhead on top: noticeably more
│
▼
Rough ballpark: somewhere in the low tens of GB of RAM to keep it fast
Now scale that by replicas and by a common cost-saving move:
1 replica → ~X GB RAM (baseline, no redundancy)
3 replicas → ~3× X GB RAM (same data, held three times, for uptime + throughput)
Apply quantization (e.g. compress vectors to lower precision)
│
▼
RAM footprint drops meaningfully — often by roughly half or more,
depending on the technique
│
▼
In exchange: a small, tunable dip in search accuracy
Nothing about the vector count changed in that last step. The only thing that moved was a compression setting — and it moved the RAM bill in one direction and the accuracy dial in the other. That's the entire vector database cost story in one example: every lever you pull to bring the resource bill down quietly pulls a different lever — accuracy, latency, or engineering effort — in the opposite direction.
Cost 7 — Updating and Deleting (the one nobody plans for)
Adding new vectors is one problem. Removing or updating old ones is a different, uglier problem, and it's the vector-database version of Episode 3's rechunking cost.
Many index structures don't handle deletion cleanly. A "deleted" vector is often just marked as a tombstone rather than actually removed — the index still has to carry it around, still touches it during traversal, until a periodic compaction or rebuild actually reclaims that space.
Document gets updated or removed
│
▼
Old vector marked as deleted, not actually removed
│
▼
Index keeps carrying the dead weight
│
▼
Search quality and speed slowly degrade as tombstones accumulate
│
▼
Eventually requires a full index rebuild to actually clean up
│
▼
Rebuild is CPU-heavy and, depending on the system, can mean a period of reduced search quality or availability while it runs
This is the cost that quietly punishes systems where documents change often — policies get updated, products get discontinued, prices change. Every one of those real-world updates leaves a small trace in the index that has to be paid for eventually, usually by an engineer noticing search quality has degraded and tracing it back to a compaction that never ran.
How Production Actually Makes This Better
None of the above means vector databases are a lost cause at scale — it means the cost has to be actively managed instead of assumed away. A few things production systems lean on:
- Quantization — storing vectors at lower precision (for example, converting from 32-bit floats to 8-bit integers or similar compressed forms) to cut RAM usage substantially, at a small, tunable cost to search accuracy.
- Tiered storage — keeping frequently-accessed or recent vectors in memory, and pushing colder, rarely-queried vectors to disk, so RAM is spent where it actually earns its cost.
- Sharding — splitting the index across multiple nodes by some logical boundary, so no single node has to hold the entire dataset in memory.
- Metadata pre-filtering — the Episode 4 payoff shows up here directly. Filtering by metadata before the similarity search runs means the search only has to traverse a relevant subset of the index, not the whole thing.
- Batched, off-peak reindexing — running index rebuilds and compaction during low-traffic windows instead of live, so Cost 7's cleanup doesn't compete with real user queries.
- Caching — for genuinely repeated queries, skipping the vector search entirely and serving a cached result, which is the cheapest possible answer to "how do I reduce vector DB load."
Every one of these is a real trade-off, not a free win — quantization costs some accuracy, tiered storage costs some latency on cold data, sharding costs engineering complexity. Production tuning isn't about eliminating these costs. It's about deciding, deliberately, which resource you'd rather spend.
Cost Isn't Just a Dollar Amount, Again
Same pattern as the last two episodes. The vector database spends:
- RAM — the single biggest recurring cost, paid 24/7 regardless of query volume
- CPU — index construction, rebuilds, and compaction
- Disk I/O — for tiered or on-disk portions of the index
- Network — moving query results and replicating data across nodes
- Latency — felt directly by users as the system grows
- Availability — the cost of not having enough replicas when a node fails
- Search accuracy — every speed optimization trades away some amount of this
- Engineering time — tuning index parameters, planning reindex windows, monitoring degradation that never throws an error
- User trust — spent quietly, the moment "the bot feels slow" becomes something people mention out loud
Bringing It Back to One Sentence
Everything earlier in this series — extraction, chunking, metadata — happens once, or close to it, per document. The vector database is the first stage in this series that never stops running. It's not a cost you pay once at ingestion. It's a cost you keep paying, every second the system is on, that grows with every document you add and every replica you need for it to stay fast and available.
A vector database doesn't charge you for the vectors. It charges you for keeping them instantly reachable — and "instantly reachable, always" is one of the most expensive promises in the entire pipeline.
Coming up in Episode 6
The vector database taught me that just keeping data searchable is expensive on its own. But there's a question I've been quietly avoiding: even with a fast, well-tuned index, am I actually sending the LLM the right chunks — or just the closest ones?
Why fewer, better-chosen chunks usually beat a bigger, more expensive model.
Less noise, more action. Let's dig.
Top comments (0)