A vector search system can feel surprisingly simple at the beginning. A document is split into chunks, each chunk becomes an embedding, and a query is converted into the same vector space so we can search for the nearest matches.
That model works well until the scale becomes large enough that the search algorithm is no longer the only thing that matters.
For this article, I want to take a hypothetical production system with around 500 million vectors, each with 768 dimensions. The system serves a large number of search requests while documents are continuously uploaded, updated, and deleted. It also has to support multiple tenants, metadata filters, access-control rules, and fresh data without letting query latency grow out of control.
At that point, the interesting questions start changing.
How much of the index can realistically stay in memory? How do new vectors become searchable while the existing index is serving traffic? What happens when an ACL filter removes most of the candidates returned by ANN? How do we delete or update vectors without constantly rebuilding the index? And once the dataset no longer fits comfortably on one machine, how do we distribute the search without destroying tail latency?
Those are the questions I want to explore here.
The similarity calculation itself remains familiar. The real engineering work begins with everything we have to build around it.
Start With Constraints, Not a Vector Database
The first architectural decision should not be whether to use HNSW, DiskANN, Pinecone, Qdrant, Milvus, or anything else. Before choosing any of them, I would write down the workload the system actually has to support.
For our 500-million-vector system, a few numbers matter immediately: the embedding dimension, expected read QPS, ingestion rate, p99 latency target, acceptable Recall@K, and how quickly a newly uploaded document needs to appear in search.
Then come the constraints that are easier to underestimate. How selective are the metadata and ACL filters? Are tenants roughly the same size, or can one customer own millions of vectors? How often are documents updated or deleted? Do searches need read-after-write consistency, or is a few seconds of freshness lag acceptable?
These answers change the architecture considerably.
A read-heavy corpus that changes once a day has very different requirements from a RAG platform receiving thousands of document updates every second. A system where almost every query searches the entire corpus behaves differently from one where every query is restricted to a small tenant and a strict ACL.
The storage budget matters too. Keeping a large index entirely in memory may produce excellent latency, but that design only makes sense if its cost is acceptable.
So before thinking about a particular vector database, I would reduce the problem to a set of measurable requirements:
- corpus size and vector dimensions
- read and write throughput
- latency and recall targets
- freshness requirements
- filtering and tenancy model
- update and deletion frequency
- durability and consistency expectations
- infrastructure budget
Once those constraints are clear, the next question becomes much easier: what part of our current search approach breaks first?
The First Thing That Breaks: Exact Search
At smaller scale, the most straightforward search strategy is also the most accurate one. Take the query vector, compare it against every stored vector, sort by distance, and return the nearest results.
There is nothing wrong with this approach. In fact, it gives us a useful ground truth because we know we are not skipping any candidates.
The problem is the amount of work.
With 500 million vectors and 768 dimensions stored as float32, the raw vector data alone is roughly:
500,000,000 × 768 × 4 bytes
≈ 1.536 TB
That is before adding metadata, index structures, replicas, caches, or operational headroom.
More importantly, an exact query would need to compare against all 500 million vectors. Doing that for one request is expensive. Doing it continuously under high QPS quickly becomes unrealistic.
This is where approximate nearest-neighbor search becomes necessary.
The useful way to think about ANN is not as a smarter similarity function. The distance calculation is still the same. The optimization comes from avoiding most of the corpus.
HNSW, for example, navigates through a graph of nearby vectors instead of scanning everything. IVF narrows the search to a few promising regions before comparing individual vectors.
Both are answering the same architectural question:
How much of the dataset can we safely avoid touching for each query?
That introduces the first real trade-off in the system.
Search fewer candidates and latency improves, but the probability of missing a true nearest neighbor increases. Search more candidates and recall improves, but query cost rises again.
So the system is no longer optimizing only for speed. It has to choose a point on the recall-versus-latency curve that matches the product requirement.
For a production system, that choice should be measured, not guessed. If our target is Recall@10, we can compare ANN results against exact search on a representative evaluation set and decide how much approximation we are actually willing to accept.
Filtering Is Where ANN Assumptions Start to Break
At this point, the search itself is faster, but production queries rarely ask for "the nearest vectors in the entire database."
They usually look more like this:
Find the nearest vectors
where tenant_id = 42
and document_type = "policy"
and language = "en"
and the user has permission to read them
That changes the problem.
Suppose ANN retrieves the top 100 candidates from the full corpus, but after applying tenant and ACL filters, only two of them are actually valid. The most relevant document for that user might exist somewhere deeper in the index, but because it never entered those first 100 candidates, the filter cannot recover it.
Increasing the candidate count can help, but it is not a complete solution. If the filter is extremely selective, we may end up searching thousands of irrelevant vectors just to find a handful of valid ones.
There are a few ways to handle this.
For a very small filtered population, it can actually be cheaper to apply the filter first and run an exact search over the remaining vectors. For larger populations, the ANN structure itself may need to understand the filter so that search spends more time exploring valid candidates.
This means the query planner needs some awareness of cardinality.
Very selective filter
→ filter first
→ search a small candidate set
Broad filter
→ ANN first or filter-aware ANN
→ return candidates efficiently
ACLs make this even more important. A result that is semantically perfect but not authorized is still unusable, so access control cannot be treated as an afterthought.
This was one of the parts of vector search that surprised me the most. ANN performance is not determined only by the index. The shape of the filters can completely change how that index behaves.
The Write Path Nobody Explains
Once documents are being uploaded continuously, vector search stops being only a read problem. The write path starts to matter just as much.
A new document may go through something like this:
Upload
→ Chunk
→ Generate embeddings
→ Attach metadata + version
→ Write to a durable log
→ Add to a mutable searchable segment
→ Build ANN structures in the background
→ Seal into an optimized segment
The durable log, usually a WAL or something serving the same purpose, is the safety net.
Before the system starts modifying indexes and in-memory state, it records the accepted operation somewhere durable. If the process crashes halfway through, that operation can be replayed after restart instead of being lost.
That leads to an important distinction:
Durable ≠ Searchable ≠ Fully Indexed
A write can already be safe from a crash while still waiting to appear in search. It can also become searchable before it has been added to the expensive ANN structure.
That second case is especially useful.
Suppose the existing 500 million vectors are already stored in optimized segments. A few thousand new vectors arrive. Rebuilding the large ANN index for every upload would make no sense.
Instead, those new vectors can first live in a smaller mutable segment. Because that segment is still small, the system may search it directly or with a lightweight index.
In the background, the database can batch enough new data, build the optimized ANN representation, and eventually seal it into another immutable segment.
Over time, multiple smaller segments may accumulate. Compaction then combines or rebuilds them so that query fanout, deleted data, and storage overhead do not keep growing forever.
The interesting part is that ingestion does not need to stop while this happens. New writes continue landing in the mutable path while older data remains available through optimized segments.
That is the basic pattern that lets a vector database stay writable without constantly rebuilding the entire search structure.
The Read Path Has to Understand Freshness
The write path creates two different search surfaces: recent mutable data and older optimized segments. The read path has to search both.
A production query might look like this:
Query
→ Generate query embedding
→ Apply tenant / shard routing
→ Search mutable + optimized segments
→ Merge candidates
→ Re-score promising results
→ Apply final ACL checks
→ Fetch chunks
→ Send context to the LLM
The important part is the concurrent search across recent and optimized data.
If the system searches only the optimized ANN segments, a document uploaded a few seconds ago may be completely invisible until background indexing finishes. Searching the mutable segment as well gives us better freshness without forcing every write to wait for ANN construction.
The results from those paths are then merged into one candidate set.
This is also where approximate search and quantization become easier to reason about. The first stage does not need to return the final ten results directly. Its job is to find a reasonably good pool of candidates quickly.
For example:
500M vectors
→ ANN search
→ 100 promising candidates
→ full-precision scoring
→ top 10
The second stage can afford to be more expensive because it is working on 100 vectors rather than 500 million.
There is one important limitation, though. If ANN fails to include the true nearest vector in that candidate set, the later re-ranking stage cannot recover it. That is why Recall@K remains an important metric even when we add a more accurate re-ranking step.
Freshness therefore becomes part of the read architecture itself. The query is not just asking, "Which vectors are nearest?" It is also asking, "Which parts of the index represent the latest searchable state?"
When RAM Stops Being the Database
By now, ANN has reduced how many vectors we examine per query. But there is another problem hiding underneath the search algorithm: where do we keep all of this data?
At 500 million vectors with 768 float32 dimensions, the raw vectors alone take roughly 1.536 TB. Once we add graph links, metadata, replicas, and operational headroom, keeping everything in memory becomes expensive very quickly.
This is where the architecture starts separating the representation used for search from the representation used for final scoring.
Instead of keeping every vector in full precision on the hottest storage tier, we can keep a compressed representation for candidate generation.
Compressed representation
→ find promising candidates quickly
→ fetch higher-fidelity vectors
→ re-score accurately
Scalar quantization and Product Quantization both reduce the amount of data that has to remain close to the search engine. The important architectural idea is not the compression technique itself. It is that the first stage only needs enough information to narrow 500 million vectors down to a much smaller candidate set.
That also changes how we think about storage.
Frequently accessed search structures may stay in RAM. Larger vector or index data can move to NVMe or SSD. Durable data can live even further down in object storage, with caches keeping the active working set closer to compute.
RAM
→ fastest, most expensive
NVMe / SSD
→ larger capacity, slower access
Object storage
→ cheapest durable layer, highest access cost
Disk-aware approaches such as DiskANN come from this same constraint. Once the dataset becomes too expensive to keep entirely in memory, the search algorithm has to be designed around the storage medium as well.
At this scale, retrieval performance is no longer only about the ANN algorithm. It is also about deciding which representation deserves the fastest and most expensive storage.
Updates, Deletes, and Model Changes Are All Migrations
A production vector index is rarely static. Documents change, permissions change, chunks disappear, and eventually the embedding model itself changes.
That creates a different class of problem from search.
For updates, I would avoid thinking in terms of modifying one vector in place. A document edit can change the chunk boundaries, the embedding, and sometimes the metadata. In practice, it is often safer to treat the new state as another version and make sure stale writes cannot overwrite a newer one.
Deletes have a similar pattern. The fastest way to remove a result from search is usually a logical delete first.
Mark as deleted
→ stop returning it
→ clean it up physically later
That physical cleanup may happen through compaction, vacuuming, or rebuilding affected index structures in the background. Rebuilding a 500-million-vector index every time one document is deleted would be far too expensive.
The biggest migration happens when the embedding model changes.
Suppose V1 uses one embedding model and V2 uses another. Even if both produce vectors with the same number of dimensions, their coordinate spaces may not be compatible. Mixing them in one index can make similarity scores meaningless.
A safer migration looks like this:
V1 index continues serving traffic
New writes
→ V1 embedding
→ V2 embedding
Existing documents
→ backfill into V2
Evaluate V2
→ switch reads
→ keep V1 for rollback
→ retire V1 later
This is essentially a blue-green migration for the retrieval layer.
The pattern is the same across all three cases: keep the current system correct, build the new state in parallel, and remove the old state only when the replacement is ready.
When One Machine Stops Being Enough
Even after reducing memory usage and moving colder data to cheaper storage, there is a point where one machine is no longer enough.
At that stage, the vector space has to be partitioned across multiple shards.
A query now looks more like this:
Query
→ Router
→ Search relevant shards
→ Each shard returns local candidates
→ Merge candidates
→ Produce global top-K
The local search on each shard is familiar. The harder part is deciding how much of the cluster each query should touch.
If every request fans out to hundreds of shards, the system may have plenty of total compute but still suffer from poor tail latency. The query only finishes when the slowest participating shard responds.
That makes routing important.
For a multitenant system, tenant-aware routing can avoid searching data that could never belong to the user. Large tenants may still need to be split across several shards, while smaller tenants can share capacity.
Replication solves a different problem. Sharding gives us capacity. Replication gives us additional copies for availability and read scaling.
Hot tenants are another practical concern. A cluster may look healthy on average while one shard is overloaded because a single customer is generating most of the traffic.
At this point, vector search has become a distributed systems problem. The ANN algorithm still runs inside each shard, but overall performance now depends just as much on routing, fanout, replication, load distribution, and the global merge.
How an Architect Actually Chooses the System
By this point, the question is no longer "Which vector database is the best?" The better question is "What does this workload actually require?"
I would start with a small set of numbers:
- How many vectors are we storing, and at what dimension?
- What are the read QPS and write throughput?
- What p99 latency can the product tolerate?
- What Recall@K do we need?
- How quickly must new data become searchable?
- How selective are tenant, metadata, and ACL filters?
- How frequently do documents change or get deleted?
- How much of the working set can we afford to keep in RAM?
- What durability and consistency guarantees matter?
Those answers narrow the design space quickly.
A smaller, read-heavy workload may work perfectly well with a RAM-first HNSW index. A much larger corpus may need quantization and SSD-aware search. If the dataset is huge but only a small portion is hot at any moment, separating compute from durable object storage may make more economic sense.
The same applies to database choice. A team already running PostgreSQL may decide that vector support inside the existing database is enough for its scale. Another system with hundreds of millions of vectors, heavy filtering, and strict latency targets may justify a dedicated retrieval engine.
The important part is to benchmark using the actual workload.
That means testing realistic filters, ingestion rates, update patterns, concurrency, and recall targets, not just running an ANN benchmark on a clean static dataset.
A vector database should be the result of the architecture decision, not the starting point.
Conclusion
At small scale, vector search looks like a nearest-neighbor problem.
At 500 million vectors, that description is no longer enough.
The search algorithm still matters, but it now sits inside a much larger system. We have to think about selective filters, mutable and immutable data, write durability, freshness, candidate generation, storage hierarchy, updates, deletes, model migrations, sharding, routing, replication, and cost.
What I find interesting is that none of this changes the basic goal. We are still trying to find the vectors closest to a query.
What changes is how much engineering is required to make that operation practical.
ANN helps us avoid touching the entire corpus. Quantization reduces how much data has to stay on expensive storage. Mutable segments keep recent writes searchable while optimized indexes are built in the background. Compaction cleans up old state. Sharding lets the system grow beyond one machine. Routing prevents every query from becoming an expensive cluster-wide operation.
Each decision appears because the previous design eventually hits a physical or operational limit.
That is probably the most useful way to think about vector search architecture at scale. Start with the workload, identify what breaks first, and introduce complexity only when the constraint actually demands it.
The similarity equation stays almost the same.
The infrastructure around it is what keeps evolving.
In the next part, we'll take these constraints and design the system from the ground up starting with the workload, then building the query path, storage, indexing, write path, sharding, and scaling strategy around the problems we uncovered here.
📖 Blog by Naresh B. A.
👨💻 Backend & AI Systems Engineer | Distributed Systems · Production ML
🌐 Portfolio: [Naresh B A]
📫 Let's connect on [LinkedIn] | GitHub: [Naresh B A]
Thanks for reading. This is my personal engineering perspective, and I'd genuinely be interested in hearing where you agree or disagree. ❤️
Top comments (0)