Vector databases: how semantic search scales

#ai #database #search #machinelearning

You've embedded a million documents into vectors. Now a query comes in and you need the closest handful — in milliseconds. Checking all million one by one is far too slow. This is the problem vector databases exist to solve.

Vector databases went from niche to everywhere on the back of the RAG boom, and they're often treated as magic. They're not. Here's what they actually do and when you genuinely need one.

The problem: nearest-neighbor search at scale

Once your content lives as embeddings — vectors capturing meaning — "find the most relevant documents" becomes "find the vectors closest to this query vector." Simple in principle: compute the distance to every stored vector and take the smallest.

The catch is scale. Comparing your query against a million (or a billion) vectors on every request, exactly, is called a brute-force or exact nearest-neighbor search, and it's too slow for anything interactive. You need the closest vectors now, not after scanning the whole database.

The trick: approximate nearest neighbor

Vector databases pull off a clever trade. Instead of finding the exact closest vectors, they find the almost certainly closest ones — approximate nearest neighbor (ANN) search — and they do it dramatically faster.

The insight is that you don't have to compare against everything if you've organized the vectors intelligently ahead of time. Algorithms like HNSW build a navigable graph of the vectors so a search can "hop" toward the right neighborhood in a few steps, touching a tiny fraction of the data. You give up a sliver of accuracy — occasionally the 5th-closest instead of the 4th — for orders-of-magnitude more speed. For search and retrieval, that trade is almost always worth it. This kind of "approximate but fast beats exact but unusable" thinking shows up all over the systems I build.

What a vector database gives you beyond speed

A real vector database is more than an ANN index. It handles:

Metadata filtering — "closest vectors, but only from documents tagged 2024," combining semantic search with structured constraints.
Updates — adding, deleting, and re-indexing vectors as your data changes, without rebuilding everything.
Persistence and scale — storing far more than fits in memory, and staying fast as you grow.

Do you actually need one?

Honest answer: not always.

A few thousand vectors? An in-memory library, or even a plain array with exact search, is fine. Don't add infrastructure you don't need.
Hundreds of thousands to millions, with live updates and filtering? Now a dedicated vector database earns its place.

The mistake is reaching for heavy infrastructure at prototype scale. Start simple; adopt a vector database when your data size and query volume actually demand it.

The takeaway

A vector database is a specialized engine for one job: finding the nearest vectors, fast, at scale. It's the retrieval muscle behind semantic search and RAG — but like any tool, it's worth adopting only when the problem is big enough to need it. More of how I make that call at www.divyakush.com.

DEV Community