The leaderboards look impressive. They also test almost nothing that matters in production. Here is the gap.
The Number That Does Not Mean What You Think
Every vector database publishes benchmark results. Queries per second. Recall at various thresholds. Indexing throughput. P50 latency.
They look rigorous. They have tables and charts and methodology sections. And for most production use cases, they tell you almost nothing useful.
The reason is simple: benchmarks reward performance under static conditions. Production systems survive continuous writes, metadata filter combinations, and concurrency spikes. The conditions that determine whether a database works in production are almost never the conditions it was benchmarked under.
The Single Client Problem
VectorDBBench, the most widely used open-source benchmarking tool for vector databases, tests with a single client. One request at a time, measuring how fast the database responds.
Production systems do not have one client. They have 50, 100, or 500 concurrent clients hitting the database simultaneously, often with different queries and different metadata filter combinations.
Reddit's engineering team made this explicit after their 2025 deployment managing 340 million vectors. Under single-client conditions, performance looked fine. As concurrent users grew, the database spent more time resolving metadata filters than calculating similarity distances. P99 latency jumped by 10x.
A 10x P99 spike under concurrent load. That is the difference between a system that works and a system that is unusable at peak hours. Single-client benchmarks tell you nothing about whether this will happen to you.
The Static Data Problem
Benchmarks test after data ingestion completes. The index is built, the data is settled, the test begins. Milvus's own engineering team acknowledged this directly: "Benchmarks test after data ingestion completes, but production data never stops flowing."
Production RAG systems in 2026 require real-time data to be useful. Customer tickets, product inventory, regulatory updates, internal research: the knowledge base changes continuously. The database needs to re-index as quickly as it ingests, while still serving queries at low latency.
Some databases handle concurrent reads and writes gracefully. Others show significant latency degradation when writes and reads are happening simultaneously. Benchmarks run under static conditions will not tell you which category your candidate database falls into.
The Filter Benchmark That Actually Matters
Filtered vector search is the most common production query pattern and the most consistently underrepresented in benchmarks.
A real enterprise query looks like this: find documents semantically similar to this question, where the document belongs to this department, was created after this date, and is tagged with this category. The vector similarity search and the metadata filtering happen together, in a single query.
Most benchmarks test vector search separately from metadata filtering. The combined performance on realistic filter combinations, under concurrent load, is the number that determines whether your system works for real users.
The 2026 VectorDBBench analysis noted that the gap between filtered and unfiltered query performance is one of the largest and least discussed differences between vector databases. A database that ranks first on unfiltered recall may rank fourth on filtered recall at equivalent concurrency. The leaderboard does not show this because the leaderboard does not test it properly.
The Five Tests That Actually Predict Production Performance
Before committing to any vector database for a production workload, run these five tests yourself. Do not rely on the vendor's published results.
Concurrent filtered search at your expected peak load. Simulate 50 to 100 concurrent clients with realistic metadata filter combinations. Measure P95 and P99, not P50. Check whether P99 degrades more than 3x from P50 under load. If it does, you have a concurrency problem.
Write and read performance simultaneously. Send a continuous stream of writes while running read queries at production volume. Measure latency on the reads. Databases that handle this gracefully maintain stable read latency while ingesting. Databases that do not show read latency spikes proportional to write volume.
Recall at your actual data scale. Benchmarks commonly test at 1 million vectors. If your production workload is 50 million, test at 50 million. Recall degrades at scale for some indexes and holds stable for others. The difference is significant and invisible in small-scale tests.
Memory consumption at 2x your expected production size. Provision a node sized for your expected data volume and then load twice as much data. Does the database handle this gracefully with degraded performance, or does it fall over? Understanding the failure mode before production is significantly better than discovering it after.
Cold start query latency. Restart the database and measure latency on the first 1,000 queries. Some databases take time to warm up caches. In systems that restart periodically or fail over to new instances, cold start latency is the latency your users experience after any disruption.
The Benchmark Number That Is Actually Useful
Of all the numbers in a vector database benchmark report, the one that correlates most reliably with production performance is cost per billion queries at a fixed recall threshold.
This number captures efficiency. A database that achieves 98.5% recall on cheap hardware is more efficiently designed than one that achieves 98.5% recall on expensive hardware. Efficiency at the architectural level predicts efficiency under the varied conditions of production far better than peak performance under ideal conditions.
The March 2026 independent benchmark that tested eight configurations at 98.5% recall produced cost-per-billion-queries numbers ranging from $84 to $7,088 for comparable recall levels. The 84x gap reflects fundamentally different architectural efficiency. An architecturally efficient database is also, in practice, a database that handles resource pressure more gracefully under concurrent load. The two properties come from the same underlying design choices.
What This Means for Evaluation
The practical implication is that vendor-published benchmarks should be treated as directional, not definitive. They tell you roughly where to look, not what you will actually experience.
The teams that evaluate vector databases correctly run their own tests on their own data, at their expected production query patterns, with realistic concurrency, including writes. They check P99, not P50. They test at 2x their expected scale, not at demo scale.
This takes more time than reading a benchmark table. It also produces databases that work reliably in production instead of databases that worked in testing and failed under load.
The benchmark leaderboard is a starting point for the shortlist, not the endpoint for the decision.
Endee ranks first in the March 2026 independent VectorDBBench comparison across throughput, recall, latency, and cost simultaneously at 98.5% recall. Run your own tests on your own data at endee.io. Free to start, no credit card required.
Top comments (0)