Keeping Vector Retrieval Stable While Your Knowledge Base Is Still Loading

At 3:17 AM, your agent platform stops responding. Ingestion workers are retrying document chunks. Retrieval latency has spiked from 80ms to 4 seconds. The HNSW index is fine. The problem is that ingestion and retrieval are fighting over the same PostgreSQL pages, and neither path has backpressure logic. This is the production failure that tutorials skip.

I am evaluating Tencent/WeKnora as an external integrator, using its stated document-to-RAG, agent, and self-maintaining Wiki scope. The design below is a proposed PostgreSQL retrieval boundary, not a claim about WeKnora's current implementation or measured performance.

Keep ingestion outside the answering path

Separate document parsing, chunking, and embedding into asynchronous workers. Those workers write versioned chunks into PostgreSQL. Retrieval reads only published document versions; generation receives retrieved text and provenance, never responsibility for repairing ingestion.

For agent memory, use the same boundary: proposed memories enter ingestion rather than becoming searchable immediately. Otherwise, an agent can retrieve its own unvalidated output and reinforce it.

My proposed generation benchmark topology places an optional relay at https://b-lost.com/v1, B-Lost's gateway endpoint. It remains outside embedding generation and database timing. Availability, authentication, and model compatibility must be verified before running that baseline; no relay measurements are claimed here.

That separation matters: changing an embedding model requires a new embedding space and reindexing, whereas changing a generation endpoint should not.

The failure that tutorials miss

Consider a production incident reconstruction, not a reported WeKnora outage: concurrent ingestion workers begin inserting into an already large HNSW index. Insert latency rises, workers retry, and retrieval tail latency follows.

HNSW updates can contend on internal index-page locks, but "the index is locked" is not a sufficient diagnosis. Correlate PostgreSQL wait events (pg_stat_activity.wait_event_type, wait_event) with ingestion concurrency, WAL volume, buffer pressure (shared_buffers exhaustion), and query plans. Storage saturation (IOPS ceiling, write queue depth) and unrelated transaction locks can produce similar symptoms. Check pg_stat_statements for insert durations and pg_locks for blocking queries.

The immediate response is bounded ingestion concurrency (semaphore or worker pool size), bounded batches (500-1000 rows per transaction, not 50,000), and exponential retry backoff with jitter, not more workers. For an initial bulk import, loading before building HNSW often reduces write amplification. An existing serving table needs a separately scheduled CREATE INDEX CONCURRENTLY to avoid blocking reads.

A second failure often appears as "bad vectors." With inner-product ranking, inconsistent embedding normalization lets vector magnitude distort relevance. With cosine distance, ordinary positive rescaling does not change ranking. Check model revisions (OpenAI text-embedding-3-small vs ada-002), truncation (padding vs truncating to 768 dimensions), zero vectors (NaN propagation from malformed input), and filters before blaming normalization. Validate that ingestion and retrieval use identical preprocessing.

One runnable retrieval boundary

This example requires Python 3, psycopg[binary], PostgreSQL, and pgvector 0.8 or newer. Save it as retrieve.py. Run python retrieve.py init once, then python retrieve.py query tenant-uuid embedding.json search words. The JSON file must contain a 768-number array from the same embedding model used during ingestion.

import json
import math
import os
import sys

import psycopg
from psycopg.rows import dict_row

DSN = os.environ["DATABASE_URL"]
SCHEMA = """
CREATE EXTENSION IF NOT EXISTS vector;
CREATE TABLE IF NOT EXISTS memory_chunk (
    id bigint GENERATED ALWAYS AS IDENTITY PRIMARY KEY,
    tenant_id uuid NOT NULL,
    content text NOT NULL,
    embedding vector(768) NOT NULL,
    search_document tsvector GENERATED ALWAYS AS
        (to_tsvector('english', content)) STORED
);
CREATE INDEX IF NOT EXISTS memory_tenant
    ON memory_chunk (tenant_id);
CREATE INDEX IF NOT EXISTS memory_embedding_hnsw
    ON memory_chunk USING hnsw (embedding vector_cosine_ops)
    WITH (m = 16, ef_construction = 128);
CREATE INDEX IF NOT EXISTS memory_search
    ON memory_chunk USING gin (search_document);
"""

def normalize(values):
    values = [float(x) for x in values]
    if len(values) != 768 or not all(map(math.isfinite, values)):
        raise ValueError("Expected 768 finite values")
    norm = math.sqrt(sum(x * x for x in values))
    if norm == 0:
        raise ValueError("Zero embedding")
    return [x / norm for x in values]

def rrf(rankings, k=60, limit=10):
    if k < 1:
        raise ValueError("k must be positive")
    scores, documents = {}, {}
    for ranking in rankings:
        seen = set()
        for rank, row in enumerate(ranking, start=1):
            key = row["id"]
            if key in seen:
                continue
            seen.add(key)
            documents[key] = row
            scores[key] = scores.get(key, 0.0) + 1.0 / (k + rank)
    ordered = sorted(scores, key=lambda key: (-scores[key], key))
    return [
        {**documents[key], "rrf_score": scores[key]}
        for key in ordered[:limit]
    ]

with psycopg.connect(DSN, row_factory=dict_row) as conn:
    if sys.argv[1] == "init":
        conn.execute(SCHEMA)
    else:
        tenant = sys.argv[2]
        with open(sys.argv[3]) as source:
            vector = json.dumps(normalize(json.load(source)))
        text = " ".join(sys.argv[4:])
        conn.execute("SET LOCAL hnsw.ef_search = 100")
        conn.execute("SET LOCAL hnsw.iterative_scan = strict_order")
        conn.execute("SET LOCAL hnsw.max_scan_tuples = 20000")
        dense = conn.execute("""
            SELECT id, content
            FROM memory_chunk
            WHERE tenant_id = %s::uuid
            ORDER BY embedding <=> %s::vector
            LIMIT 50
        """, (tenant, vector)).fetchall()
        lexical = conn.execute("""
            SELECT id, content
            FROM memory_chunk,
                 websearch_to_tsquery('english', %s) AS q
            WHERE tenant_id = %s::uuid AND search_document @@ q
            ORDER BY ts_rank_cd(search_document, q) DESC, id
            LIMIT 50
        """, (text, tenant)).fetchall()
        print(json.dumps(rrf([dense, lexical]), indent=2))

Apply the same validation and normalization during ingestion. Treat this schema as the retrieval kernel; document versions and publication state belong in the surrounding ingestion contract. Run initialization separately from serving traffic. Add connection pooling (pgbouncer in transaction mode, or application-level pools with pool_size=20, max_overflow=10) to prevent connection exhaustion under load.

Spend latency where recall needs it

Filtered approximate search is the central trade-off. HNSW may encounter many neighboring vectors belonging to other tenants before finding enough eligible rows. Iterative scanning (hnsw.iterative_scan = strict_order) helps, but its scan budget (hnsw.max_scan_tuples) is finite. For small tenant datasets (under 10,000 rows), an exact distance sort over tenant-filtered rows may be faster and more complete than approximate search with high filtering overhead.

RRF combines lexical and semantic ranks without pretending their raw scores share a scale. It rescues identifiers and exact phrases that embeddings sometimes miss, but cannot recover candidates absent from both lists. Test retrieval quality by measuring recall@10 against a labeled eval set, not by eyeballing results.

Finally, tenant predicates are not authorization. Derive tenant identity from authenticated context (JWT claims, session middleware) and enforce database isolation where appropriate (row-level security policies if tenants must share tables). Compare approximate results against exact tenant-filtered search while ingestion runs. That comparison distinguishes a fast query from a retrieval system an agent can trust.

What's the hardest choice you've made?

The question that keeps me up is this: when do you stop treating ingestion as best-effort background work and start throttling user-facing features to preserve database stability? I've seen teams add circuit breakers that pause document uploads when retrieval p99 crosses 500ms, and I've seen teams accept 2-second retrieval during bulk imports because agents can wait. Both choices are defensible. Both have failed under unexpected load.

What's your architecture? Have you hit the wall where approximate search stopped being approximate enough, or where exact search became too slow? What did you change, and what did you wish you'd known six months earlier?