Level 4 of nine in Project Arc Rector — an agentic RAG stack built from free, self-hostable parts, one swappable level at a time. Level 3 was the framework question. This one is the layer underneath it: what does "similar" mean, and who stores it.
The page, with the index structures running in your browser: https://dev48.infy.uk/arcrector/level4-vector-databases.html
Repo: https://github.com/dev48v/arc-rector
A vector database is three decisions, not a product
1. The index. Flat (exact, brute force), IVF (cluster, then search a few clusters), or HNSW (a navigable small-world graph). This is the only one that changes your results, because everything but flat is approximate — and approximate means a recall number you are choosing whether or not you measure it.
2. The metric. Cosine, dot product or L2. Not interchangeable: if your embeddings are normalised, cosine and dot product rank identically and L2 does too — up to a monotone transform. If they are not normalised, dot product rewards long vectors, which usually means it rewards long documents.
3. Where it lives. In-process (FAISS, Chroma), a server (Qdrant, Weaviate, Milvus), or your existing database (pgvector). Arc Rector's default is Qdrant, Apache-2.0, because it runs as one container and its filtering is not an afterthought.
The number nobody tunes deliberately
Every ANN index has a knob that trades recall for latency — nprobe in IVF, ef_search in HNSW. The default is chosen by the library, and it decides how many of your true nearest neighbours you actually see.
ef_search=16 recall@10 = 0.71 fast
ef_search=64 recall@10 = 0.94
ef_search=256 recall@10 = 0.995 slow
Measure it against a flat index on your own data. It is one exhaustive search over a few thousand vectors, it takes minutes, and it is the difference between "retrieval is fine" and "retrieval silently drops a quarter of the right chunks".
That measurement is also the one that tells you whether a RAG problem is a retrieval problem or a generation problem — and without it, teams reliably spend weeks tuning the prompt for a recall bug.
Filtering is where the abstraction leaks
"Find similar chunks, but only from this tenant" is the most common real query and the one that breaks naive setups. Filter after the search and a tenant with few documents returns nothing, because the top-k came back entirely from someone else's data. Filter before it and you are doing a brute-force scan of the filtered subset.
Good engines integrate the filter into the graph traversal. It is worth knowing which yours does, because the failure mode is silent and tenant-shaped.
What this level does not fix
A vector store returns what is near. Near is not relevant, and the embedding decides what near means — which is Level 5, and it is the level with more leverage than this one. If your retrieval is bad, changing databases is the least likely fix on the list.
Level 5 is embeddings and reranking, where the question becomes what "similar" was ever supposed to mean.
The whole stack, nine levels, all free to self-host: https://dev48.infy.uk/arcrector.php
Top comments (0)