DEV Community

Cover image for You Probably Don't Need a Dedicated Vector Database (Yet)
M.T.Ramkrushna
M.T.Ramkrushna

Posted on

You Probably Don't Need a Dedicated Vector Database (Yet)

#ai #database #postgres #rag

Hot take for this Wednesday: for most RAG apps, adding a dedicated vector database on day one is premature infrastructure. If you already run Postgres, pgvector is probably enough. If you already run Elasticsearch or OpenSearch, its vector search is probably enough. Reach for a separate vector store when you have measured a scale, latency, or filtering problem your current database can't handle, not because a "RAG stack" diagram had a box labeled "vector DB."

Embeddings are just another column. The hard part of RAG is usually chunking, retrieval quality, and evals. Where the floats live is rarely the bottleneck.

What "just use Postgres" actually gets you in 2026

This isn't the pgvector of 2023. From the project's own README and changelog:

  • Two ANN index types. HNSW ("better query performance than IVFFlat (in terms of speed-recall tradeoff), but has slower build times and uses more memory") and IVFFlat (faster builds, less memory, lower speed-recall).
  • Smaller vectors when you need them. vector indexes up to 2,000 dimensions, halfvec up to 4,000, bit up to 64,000, and sparsevec up to 1,000 non-zero elements. The README's scaling advice: use halfvec for tables and binary quantization (with re-ranking) for indexes.
  • Filtering that doesn't silently starve you. The README is upfront: with approximate indexes, filtering is applied after the index scan, so if a condition matches 10% of rows, HNSW with the default hnsw.ef_search of 40 returns only about 4 rows on average. Since 0.8.0 (Oct 2024), iterative index scans keep scanning until enough rows match. You can also use partial indexes, or list partitioning per tenant.
  • Hybrid search. Combine with Postgres full-text search and merge with Reciprocal Rank Fusion or a cross-encoder. The pgvector-python repo has examples of both.
  • Everything else Postgres does. In the README's words: "ACID compliance, point-in-time recovery, JOINs, and all of the other great features of Postgres."
  • Scaling paths. Vertically, with read replicas, or sharded with Citus or PgDog.
  • Active maintenance. The latest release is 0.8.7 (Oct 1, 2026), and 0.9.0 is in progress with async I/O for HNSW.

It's also available on the managed Postgres you probably already pay for. Amazon RDS for PostgreSQL, for example, announced pgvector 0.8.0 support in November 2024.

(I covered the hands-on setup in an earlier post: pgvector: build RAG retrieval inside the Postgres you already run. This one is about the decision.)

The hidden cost of a second database is sync

The most underrated cost of a separate vector store isn't the bill. It's that you now have two sources of truth.

Your documents, permissions, tenant IDs, and "is this deleted?" flags live in your main database. Their embeddings live somewhere else. Every create, update, delete, and permission change now has to reach both systems. If it doesn't, you get:

  • Ghost results: a deleted or unpublished document still gets retrieved.
  • Permission leaks: an ACL changed in Postgres, but the copy in the vector store still says the old thing.
  • Stale answers: the price, policy, or doc changed, and the embedding still reflects the old version.

Inside one Postgres, the row and its embedding commit together in one transaction, and your WHERE tenant_id = $1 AND deleted_at IS NULL is the same filter your app already trusts.

This isn't hypothetical. Instacart wrote about exactly this. They had moved full-text search from Elasticsearch to Postgres, then added semantic search through a separate FAISS-based ANN service. Their list of problems with that setup:

  • FAISS "had limitations on filtering documents based on their attributes at retrieval time," which forced overfetching and then post-filtering. As a result, "some relevant documents" were never fetched.
  • "Maintaining two separate services for recall came with developmental and operational overhead. Storing data in two separate systems and keeping them in-sync led to inconsistencies."

They moved semantic retrieval into Postgres with pgvector and used real-time availability as a pre-filter. pgvector was "marginally slower than FAISS for larger retailers," but "it had better recall performance." In a production A/B test, searches with zero results dropped by 6%. That's a catalog of billions of items, with billions of writes per day.

To be clear, this was a search-engine migration at Instacart scale, not a RAG chatbot. But the lesson carries over: keeping vectors next to the data they describe removed an entire class of bugs.

"But the benchmarks!"

You'll find benchmarks for every side. Here's one, and it should be read as the vendor claim it is. Timescale's pgvectorscale extension (a StreamingDiskANN index that complements pgvector) reports that on 50 million 768-dimension Cohere embeddings, Postgres with pgvector + pgvectorscale got 28x lower p95 latency and 16x higher query throughput than Pinecone's storage-optimized (s1) index at 99% recall, at 75% less cost when self-hosted on EC2.

That's Timescale benchmarking its own product against one Pinecone index type. I wouldn't quote "28x" in a design review. What it does show is that "Postgres can't do vectors at tens of millions of rows" is no longer a safe assumption, so you should test it on your own data instead of assuming.

The strongest case against me (and where it's right)

The best counterargument I've read is Alex Jacobs' "The Case Against pgvector" (Oct 2025). It's worth reading in full. His points, fairly summarized:

  • Index builds are heavy. He reports that building an HNSW index on a few million vectors "can consume 10+ GB of RAM or more," on your production database, potentially for hours. (pgvector's own docs warn about maintenance_work_mem and the HNSW graph not fitting in memory.)
  • Continuous inserts have a cost. IVFFlat clusters drift and need rebuilds. HNSW inserts update the graph and add load to your transactional database.
  • Filtered search pushes query-planner problems onto you. Pre-filter vs post-filter is "the difference between queries that take 50ms and queries that take 5 seconds."
  • Hybrid search is DIY. You write the RRF yourself.
  • pgvectorscale isn't on RDS.
  • His conclusion: for many teams, a managed vector DB is simpler and maybe cheaper.

Where he's right: if your corpus is in the millions and churning constantly, your Postgres is already busy with OLTP, and nobody on the team enjoys reading EXPLAIN, a managed vector store can genuinely be less work.

Where I'd push back:

  • Filtering got better. The Hacker News reply from Discourse's Rafael dos Santos Silva points out that the pre/post-filtering problem "was fixed in version 0.8.0 via Iterative Scans." He also says Discourse runs pgvector "in thousands of databases."
  • "Hybrid is DIY" is about 30 lines of SQL, and the pgvector-python repo has an RRF example. (My estimate, not a measured number.)
  • A dedicated DB doesn't remove the sync problem. It creates it. Now the pipeline that keeps two stores consistent is yours to own.
  • Most RAG corpora aren't "millions of vectors churning constantly." An internal docs bot, a support knowledge base, or a product's help center usually sits in the thousands to low millions of chunks. That's my opinion from the projects I've seen, not a survey.

When a dedicated vector database actually wins

I'm not anti-vector-DB. I'm anti-default. I'd reach for one when at least one of these is true and measured:

  1. Scale beyond one comfortable node. Hundreds of millions to billions of vectors, where index memory and build times stop fitting your Postgres instance, and sharding Postgres yourself is a bigger project than adopting a purpose-built system. For context, turbopuffer's published limits page says it has seen 1T+ documents globally and 250M+ namespaces. That's the kind of scale these systems are designed for (vendor-reported).
  2. Vector QPS that would starve your OLTP. If vector reads would compete with checkout or login traffic, isolate them. A read replica is the first step. A separate system is the next.
  3. Massive multitenancy. Millions of tiny per-user indexes (one per customer, workspace, or agent memory) are a natural fit for namespace-per-tenant designs on object storage.
  4. High-churn, real-time ingestion where index maintenance on your primary database is measurably hurting p95.
  5. Your source of truth isn't relational anyway. If your content lives in object storage or a lakehouse and Postgres would only be a vector cache, the "one database" argument goes away.

Notice what's not on the list: "we're building RAG," "the tutorial used Pinecone," or "we might need it later."

My decision checklist

Before adding a vector database to the architecture diagram:

  1. Start where your data lives. Postgres → pgvector. Elasticsearch/OpenSearch → their kNN (Elastic's docs say putting the filter inside the knn clause makes it a pre-filter, while filters outside it post-filter and "can return fewer than k results").
  2. Build an eval set first. 50 to 200 real queries with expected documents. Measure recall@10 and p95 latency on your corpus.
  3. Use the cheap levers before new infrastructure: HNSW over IVFFlat, tune ef_search, enable iterative scans for filtered queries, halfvec or binary quantization, partial indexes or partitioning per tenant, a read replica.
  4. Write down the trigger. For example: "We'll evaluate a dedicated store when p95 retrieval goes over X ms at Y QPS, or the index no longer fits in RAM." Pick the numbers yourself. Mine would be different from yours.
  5. If you do split, design the sync on day one. Use an outbox or CDC pipeline, idempotent upserts, delete propagation, and a reconciliation job that diffs IDs between stores. Filter permissions at query time, not just at ingestion.

The bottom line

A vector database is a scaling decision, not a RAG requirement. For most teams, the right first vector store is the database that already holds the documents, the permissions, and the backups. Measure first. When the numbers say you've outgrown it, migrate on purpose.

What's in your RAG stack right now: pgvector, your search engine, or a dedicated vector DB? And what pushed you there?

References

Top comments (0)