Retrieval-Augmented Generation (RAG) is a straightforward concept: embed your documents, store the embeddings in a vector database, and at query time retrieve the most relevant chunks to include in the LLM's context. In practice, the overwhelming majority of RAG failures in production trace to the retrieval layer — not the generation layer. The LLM is doing exactly what it should; it just received bad context.
The Four Production Failure Modes
- Chunk boundary errors: Splitting documents at fixed character counts breaks semantic units — a sentence or table spans a chunk boundary and the critical fact is split across two chunks that never appear together in retrieval.
- Embedding model mismatch: Using a general-purpose embedding model for financial documents with dense numerical content, technical jargon, and tabular structure produces poor semantic similarity scores for domain-critical queries.
- Non-idempotent indexing: Re-indexing a document without checking whether it already exists produces duplicate chunks, corrupting retrieval rankings with redundant results.
- No retrieval evaluation: Without a retrieval evaluation harness (a set of queries with known ground-truth relevant chunks), there is no way to measure whether chunking strategy changes or embedding model upgrades improve or degrade retrieval quality.
pgvector Implementation
# Create the table with embedding column
CREATE TABLE documents (
id SERIAL PRIMARY KEY,
doc_hash TEXT UNIQUE, -- idempotency check
content TEXT,
embedding vector(1536), -- OpenAI ada-002 dimensions
metadata JSONB
);
CREATE INDEX ON documents USING ivfflat (embedding vector_cosine_ops)
WITH (lists = 100);
The doc_hash column (SHA-256 of content + metadata) prevents duplicate indexing. The IVFFlat index enables sub-millisecond approximate nearest-neighbor search at millions of vectors.
Top comments (0)