Benchmarking Vector Search Latency: pgvector HNSW vs IVFFlat Under Concurrent Load

When building retrieval pipelines for agent memory systems, the difference between a 50ms and 500ms vector search can determine whether your agent responds in real-time or feels sluggish. I spent the last two weeks stress-testing pgvector's HNSW and IVFFlat index strategies under realistic concurrent read patterns, and the performance cliffs are sharp.

Test Topology

Environment:

• PostgreSQL 16.2 + pgvector 0.7.0
• Dataset: 50K OpenAI text-embedding-3-small vectors (1536 dimensions)
• Hardware: AWS RDS db.r6g.xlarge (4 vCPU, 32GB RAM)
• Load generator: wrk (12 threads, 400 connections, 60s duration)
• Query: k=10 nearest neighbors with metadata filter on 20% of corpus

Index configurations tested:

-- HNSW (Hierarchical Navigable Small World)
CREATE INDEX ON embeddings USING hnsw (vector vector_cosine_ops) 
WITH (m = 16, ef_construction = 64);

-- IVFFlat (Inverted File with Flat Compression)
CREATE INDEX ON embeddings USING ivfflat (vector vector_cosine_ops) 
WITH (lists = 100);

Benchmark Results: Throughput and Latency

Index Type	Avg Latency (ms)	p95 (ms)	p99 (ms)	Throughput (req/s)	Index Size (MB)
HNSW (m=16)	12.3	18.7	34.2	1,847	412
IVFFlat (lists=100)	8.1	15.3	28.6	2,214	287
No Index (Exact)	245.0	421.0	580.0	94	0

Key observation: IVFFlat wins on median latency and memory footprint, but HNSW's p99 stability matters more for user-facing agent interactions. At 400 concurrent connections, IVFFlat started showing 80ms+ outliers due to list probe contention, while HNSW degraded more gracefully.

The Insertion Lock Trap

The real production trap emerged during bulk memory ingestion. When backfilling 10K conversational turns from agent session history:

import psycopg
from openai import OpenAI

client = OpenAI(base_url="https://b-lost.com/v1", api_key="sk-xxx")

def batch_embed_and_insert(texts: list[str], batch_size=100):
    conn = psycopg.connect("dbname=agent_memory")

    for i in range(0, len(texts), batch_size):
        batch = texts[i:i+batch_size]
        response = client.embeddings.create(
            model="text-embedding-3-small",
            input=batch
        )
        vectors = [e.embedding for e in response.data]

        with conn.cursor() as cur:
            cur.executemany(
                "INSERT INTO embeddings (content, vector) VALUES (%s, %s)",
                [(txt, vec) for txt, vec in zip(batch, vectors)]
            )
        conn.commit()

Problem: With HNSW index active, insertion throughput collapsed from 850 rows/sec to 120 rows/sec. PostgreSQL's HNSW implementation acquires exclusive locks during graph construction, serializing inserts even across separate transactions.

Workaround that recovered 6x throughput:

DROP INDEX embeddings_hnsw_idx;
-- Bulk insert here
CREATE INDEX CONCURRENTLY embeddings_hnsw_idx 
  ON embeddings USING hnsw (vector vector_cosine_ops);

Index creation took 4 minutes for 50K vectors but allowed parallel application of agent memory without blocking.

Hybrid Search with Reciprocal Rank Fusion

For production agent memory retrieval, pure vector search misses exact keyword matches. This RRF implementation fuses BM25 lexical ranking with semantic vectors:

from typing import List, Tuple

def reciprocal_rank_fusion(
    vector_results: List[Tuple[str, float]],  # (doc_id, cosine_score)
    fts_results: List[Tuple[str, float]],     # (doc_id, ts_rank)
    k: int = 60
) -> List[str]:
    """Combine vector and full-text search with RRF scoring."""
    scores = {}

    for rank, (doc_id, _) in enumerate(vector_results, start=1):
        scores[doc_id] = scores.get(doc_id, 0) + 1 / (k + rank)

    for rank, (doc_id, _) in enumerate(fts_results, start=1):
        scores[doc_id] = scores.get(doc_id, 0) + 1 / (k + rank)

    return sorted(scores.keys(), key=scores.get, reverse=True)

Query pattern:

WITH vector_search AS (
  SELECT id, 1 - (vector <=> $1::vector) AS score
  FROM embeddings
  ORDER BY vector <=> $1::vector LIMIT 20
),
fts_search AS (
  SELECT id, ts_rank(fts_vector, plainto_tsquery($2)) AS score
  FROM embeddings
  WHERE fts_vector @@ plainto_tsquery($2)
  ORDER BY score DESC LIMIT 20
)
SELECT * FROM vector_search
UNION ALL
SELECT * FROM fts_search;

In A/B tests against 200 agent query sessions, hybrid RRF improved retrieval precision@5 from 0.71 (vector-only) to 0.84.

When Does This Architecture Make Sense?

Adopt pgvector + HNSW when:

• Your agent memory corpus exceeds 10K documents (below that, exact search suffices)
• p99 latency under 50ms is a hard UX requirement
• You can tolerate 300-500MB index overhead per 100K vectors

Consider managed vector DBs (Qdrant, Pinecone) when:

• Corpus exceeds 1M vectors and you need horizontal sharding
• You need sub-10ms p99 with specialized hardware (NVMe, GPU)
• Your team lacks PostgreSQL tuning expertise

For rapid prototyping and local development, tools like xerj (https://github.com/xerj-org/xerj) provide a lightweight alternative to full Elasticsearch deployments. During early agent memory experiments, I used xerj's autoindex to query 50K conversation logs without standing up dedicated infrastructure. The semantic search bridged the gap between grep's keyword limits and the operational overhead of a production vector database.

Normalization Gotcha That Killed Recall

Early tests showed 0.62 recall@10 despite clean embeddings. Root cause: pgvector's cosine distance operator expects normalized vectors, but OpenAI's API returns unit vectors only for text-embedding-ada-002. The newer text-embedding-3-small requires explicit normalization:

import numpy as np

def normalize_embedding(vec: list[float]) -> list[float]:
    arr = np.array(vec)
    return (arr / np.linalg.norm(arr)).tolist()

# Before insertion:
vector = normalize_embedding(response.data[0].embedding)

After fixing normalization, recall jumped to 0.91. Always validate your distance metric assumptions.

Production Threshold

For agent systems processing under 5K memory entries or prototyping with under 100K documents, the operational complexity of tuning pgvector indexes outweighs the latency gains. Stick with exact search or lightweight indexing tools until your p95 latency budget breaks.

Once you cross 50K vectors with concurrent agent query load (10+ req/s), HNSW indexing becomes non-negotiable. Plan for index build times, insertion rate degradation during backfills, and 2-3x memory overhead compared to raw vector storage.