RAG from Zero: Embeddings, Vector DBs, and Retrieval

RAG from Zero: Embeddings, Vector DBs, and Retrieval

RAG is the most job-relevant skill in this series so far. In the 390 AI engineer job postings behind Part 1, RAG appeared in 26 percent, tied with fine-tuning and ahead of LangChain itself. Yet most RAG tutorials stop at a five-line demo, and most production RAG systems fail for one boring reason: retrieval. About 73 percent of enterprise RAG failures originate at the retrieval step, not the generation step. This part builds the whole pipeline from zero: embeddings, vector databases, and the retrieval techniques that actually move the needle, with real benchmark numbers at every step.

Part 4 of the AI Engineer Series. Part 1 mapped the roadmap from 390 real job postings, Part 2 covered the Python that matters for AI work, and Part 3 opened the black box: tokens, context, and temperature. This part builds the RAG pipeline on top of all three. Part 5 adds evals, so you can measure whether any of this is working.

What RAG actually is

RAG, retrieval-augmented generation, is a workaround for two hard limits. Models hallucinate when they lack facts, and their knowledge freezes at training time. RAG fixes both without retraining: you store your documents as searchable vectors, and for each question you retrieve the most relevant passages and paste them into the prompt. The model answers from your documents, not its memory.

The pipeline has five stages, and you will build every one of them here:

• Chunk: split documents into retrievable passages (token-aware, not character-aware)

• Embed: convert each chunk to a vector with an embedding model

• Index: store the vectors in a database built for similarity search

• Retrieve: embed the question, find the nearest chunks, optionally rerank

• Generate: stuff the winners into the prompt and call the model

Embeddings: text as coordinates

An embedding model converts text into a list of floating point numbers, typically a few hundred to a few thousand of them. Texts with similar meaning land close together in that space. Closeness is measured with cosine similarity: the cosine of the angle between two vectors, 1.0 for identical direction, 0 for unrelated, negative for opposite. Practical rule: normalize your vectors to unit length first, then plain dot product equals cosine similarity. Most embedding APIs return normalized vectors already.

Which model? The honest answer is that the leaderboard keeps moving, so here is a snapshot from one consistent leaderboard, MTEB English v2 retrieval, read October 2026:

• Qwen3-Embedding-8B: 69.44. Open weights, Apache 2.0, 4096 dimensions, 32K token input. The best score on this board, and it runs on a single consumer GPU at INT4 quantization (about 4.5 to 5.2 GB of VRAM, so an RTX 3060 12GB works).

• gemini-embedding-001: 64.35. Google's proprietary model, 3072 dimensions, $0.15 per million tokens.

• text-embedding-3-large: 57.98. OpenAI's flagship, 3072 dimensions, 8192 token input, $0.13 per million tokens.

The open model wins on this benchmark, and it costs nothing per token once it runs on your hardware. Two more self-hosted picks worth knowing: BGE-M3 (568M parameters, 1024 dimensions, 8K token context, MIT license, runs fine on CPU, and outputs dense vectors plus sparse weights plus multi-vectors in one pass) and nomic-embed-text-v1.5 (137M parameters, Ollama's most-pulled embedding model). For a pure API start, OpenAI's text-embedding-3-small costs $0.02 per million tokens and Voyage's voyage-4 costs $0.06, with 200M free tokens per account.

Dimensions are storage, forever

Every dimension costs 4 bytes per vector at float32. One million vectors at 3072 dimensions need about 12.3 GB just for the raw vectors; at 768 dimensions, 3.1 GB. That is a 4x difference that never goes away, and larger dimensions only buy marginally better retrieval. This is where Matryoshka truncation earns its keep: OpenAI's 3-series models were trained so the first N dimensions are themselves a valid embedding, and OpenAI's published benchmark shows 3-large truncated from 3072 down to 256 dimensions still beats the full 1536-dimensional ada-002. That is a 12x storage cut. Two rules: truncation only works on models trained for it (OpenAI 3-series, Gemini, Qwen3-Embedding, BGE family), and you must re-normalize after slicing, because truncation shrinks the vector's magnitude.

Cost math for planning. Embedding is input-only: no output charge. One million chunks of 512 tokens is 512M tokens, which costs about $10.24 once with text-embedding-3-small, or about $5.12 through the Batch API. A single generation call of 5K input plus 1K output tokens on a mid-tier model costs roughly $0.04. So indexing a million chunks once costs about the same as 250 to 500 generation calls. Embeddings are the cheapest part of RAG, which is why retrieval quality, not embedding price, should drive your choices.

from openai import OpenAI
import numpy as np

client = OpenAI()
# Batch + truncate: 512 dims is plenty for most retrieval
r = client.embeddings.create(
    model="text-embedding-3-small",
    input=["RAG retrieves passages before generating.",
            "Vector DBs index embeddings for fast search."],
    dimensions=512,
)
vecs = np.array([d.embedding for d in r.data])
vecs = vecs / np.linalg.norm(vecs, axis=1, keepdims=True)  # unit length
q = np.array(client.embeddings.create(
    model="text-embedding-3-small", input="How does RAG find documents?",
    dimensions=512).data[0].embedding)
q = q / np.linalg.norm(q)
print(vecs @ q)  # cosine similarities, dot product after normalization

Vector databases: search at speed

Brute-force search over a million 1536-dimensional vectors means a million dot products per query: seconds of compute, every query. Vector databases replace that with an approximate index, and the dominant one is HNSW, a layered graph where each vector links to its nearest neighbors. Search walks the graph greedily from the top layer down. It is approximate, which means it trades a little recall for a lot of speed, and three knobs control that trade:

• M (default 16): edges per node. Sets the recall ceiling but costs memory permanently, and changing it needs a full rebuild. Measured on 1M vectors at 768 dimensions: M=8 gives about 0.90 recall@10, M=16 about 0.96, M=64 about 0.99 at 5x build time and 4x memory.

• efConstruction (default around 200): candidate list size at insert time. Costs build time only, no extra latency or memory. Too low (around 40) permanently degrades graph quality.

• efSearch: candidate list at query time. Tunable per query with no rebuild. Same setup: ef=10 gives 0.72 recall at 0.35 ms, ef=64 gives 0.96 at 1.1 ms, ef=256 gives 0.995 at 3.4 ms. Start with M=16 and raise efSearch first.

Real numbers, 1M vectors at 1536 dimensions. An independent test of managed services (May 2026, 1,000 queries, k=10) measured Qdrant at 56 ms median and 130 ms p95 with 0.989 recall@10, Pinecone at 180 ms median and 650 ms p99 with 0.984 recall, and Weaviate at 98 ms median and 380 ms p99 with 0.981 recall, with Qdrant about 36 percent cheaper. A November 2026 comparison on comparable compute measured Qdrant at p50 3 ms and p95 8 ms with 99.2 percent recall, pgvector's HNSW at p50 5 ms and p95 12 ms with 98.5 percent, and Pinecone Serverless at p50 12 ms and p95 48 ms with 95 percent. One caveat to keep you honest: most headline numbers come from vendor-owned harnesses, and they disagree with each other, so treat any single number as condition-dependent, not settled fact.

Which one to pick is simpler than the benchmarks make it look:

• Already run Postgres? Use pgvector. One less system to operate, and 98.5 percent recall at 5 ms is plenty for most products.

• Want the fastest self-hosted option? Qdrant. Good Docker story, honest docs, strong recall per millisecond.

• Want zero ops? Pinecone Serverless. You pay more per query for never thinking about it.

• Huge scale or heavy filtering? Milvus. Built for billions of vectors.

• Local prototype? Chroma. Five lines to start, and it stays out of your way.
  

from qdrant_client import QdrantClient
from qdrant_client.models import Distance, VectorParams, PointStruct

db = QdrantClient(":memory:")  # swap for QdrantClient("http://localhost:6333")
db.create_collection("docs",
    vectors_config=VectorParams(size=512, distance=Distance.COSINE,
                                hnsw_config={"m": 16, "ef_construct": 200}))
db.upsert("docs", [PointStruct(id=i, vector=v.tolist(),
                               payload={"text": t})
                   for i, (v, t) in enumerate(zip(vecs, chunks))])
hits = db.query_points("docs", query=q.tolist(), limit=10,
                       search_params={"hnsw_ef": 64}).points

Retrieval is where RAG lives or dies

Here is the uncomfortable data. Retrieval, not generation, causes about 73 percent of enterprise RAG failures. In 47 to 67 percent of cases, the generator did not effectively use the top retrieved documents. And your eval set is probably lying to you: when the same retriever setups met real user questions instead of synthetic ones, Hit@5 fell 29 to 37 percentage points (Kucia and Gawlik, arXiv, September 2026). Four techniques fix most of it, in order of effort.

1. Chunk in tokens, not characters. A 500-character chunk is a different token count in English prose, German compounds, and Python code, which means uneven retrieval units and uneven bills. Our hands-on chunking guide measured which sizes retrieve best, and Part 3 showed why token counts are the only honest unit.

2. Hybrid search: BM25 plus vectors. Dense vectors catch meaning, keyword search catches exact terms like product codes and names. Our hybrid search guide builds this in Python. Anthropic's Contextual Retrieval work (September 2024) measured the stack precisely: baseline top-20 failure rate of 5.7 percent dropped to 3.7 percent with contextual embeddings, 2.9 percent adding contextual BM25, and 1.9 percent adding reranking. A 67 percent failure reduction, at a one-time indexing cost of about $1.02 per million document tokens.

3. Reranking: a second, smarter pass. A cross-encoder reads the query and each candidate document together, which is slower but far more accurate than vector similarity alone. The standard pattern is retrieve 100 with vectors, rerank to the top 10. Measured gains: Cohere Rerank 3 lifted recall@5 by 25.6 percent over a cosine baseline on BEIR queries; adding BGE reranker v2-m3 moved Hit Rate@5 from 75.4 to 92.1 percent; Cohere Rerank 3 moved it to 94.2 percent. Pricing: Cohere Rerank costs $2.00 per 1,000 searches, while a local BGE reranker processes 100 candidates in about 84 ms for the price of your own GPU. The crossover sits around a million queries per month: below that, the API is cheaper than operating a GPU for it.

4. Query rewriting. Short user queries live in a different part of embedding space than long documents. HyDE generates a hypothetical answer with an LLM and embeds that instead; on TREC DL19 it scored nDCG@10 of 61.3 against 50.6 for BM25. The cost is query-time latency, up 43 to 60 percent with small generation models, so reach for this after hybrid and reranking are in place.

# The retrieve-then-rerank pattern in one function
def answer(question, k_retrieve=100, k_final=8):
    qv = embed([question])[0]                      # 1. embed the question
    cands = db.query_points("docs", query=qv,
                            limit=k_retrieve).points  # 2. fast vector pass
    pairs = [(question, c.payload["text"]) for c in cands]
    scores = reranker.predict(pairs)               # 3. cross-encoder rerank
    top = [c for _, c in sorted(zip(scores, cands),
                                key=lambda x: x[0], reverse=True)][:k_final]
    context = "\n\n".join(c.payload["text"] for c in top)
    return llm(f"Answer from these passages, cite which one you used:\n{context}\n\nQ: {question}")

Failure modes nobody warns you about

• Context rot beats stuffing, every time. A focused 300-token input outperformed a 113,000-token full conversation on the same task, even though the answer was in both. Retrieval exists precisely so you never have to stuff.

• Effective context is far shorter than advertised. Eleven of thirteen models claiming 128K+ fell below half their short-context score at 32K tokens once keyword shortcuts were removed. Keep retrieved context lean and put critical instructions at the start or end.

• RAG reduces hallucinations but does not eliminate them. One local study measured hallucination rate falling from 60 to 35 percent with RAG; Stanford's 2024 study of legal AI tools found RAG-based products still hallucinated on 17 to 33 percent of queries. Cite passages, verify high-stakes answers, and never present RAG output as verified fact without a check.

• Measure on real questions. Synthetic eval sets inflate scores by 29 to 37 points. Collect real user queries from day one. Our RAG evals guide shows the harness, and Part 5 of this series makes evals a first-class skill.

The full pipeline, one script

Here is the whole thing with zero infrastructure, using sentence-transformers and numpy, so you can run it on a laptop. Each comment marks where the production swap goes: Qdrant for the index, an embedding API for the model, hybrid search for the retriever, a cross-encoder for the reranker.

from sentence_transformers import SentenceTransformer  # swap: OpenAI/Voyage API
import numpy as np

model = SentenceTransformer("BAAI/bge-m3")  # 568M params, CPU-friendly, MIT

def chunk_tokens(text, size=400, overlap=50):
    words = text.split()  # swap: tiktoken-based chunker from Part 3
    return [" ".join(words[i:i+size])
            for i in range(0, len(words), size - overlap)]

docs = [open(f).read() for f in ("manual.txt", "faq.txt")]
chunks = [c for d in docs for c in chunk_tokens(d)]
vecs = model.encode(chunks, normalize_embeddings=True)  # swap: asyncio batch embed (Part 2)

def retrieve(question, k=8):
    q = model.encode([question], normalize_embeddings=True)[0]
    sims = vecs @ q                      # swap: Qdrant query_points, hnsw_ef=64
    idx = np.argsort(sims)[::-1][:k*10]  # over-fetch...
    # swap: rerank idx[:k*10] with a cross-encoder, keep top k
    return [chunks[i] for i in idx[:k]]

context = "\n\n".join(retrieve("How do I reset the device?"))
# swap: client.responses.create(..., temperature=0) with the context in the prompt
print(context[:400])

The AI engineer's mental model

• Embeddings are the cheapest decision and the easiest to get right. Start with text-embedding-3-small or voyage-4 via API, or BGE-M3 on your own CPU. Truncate dimensions with Matryoshka and re-normalize.

• The index is a recall/latency trade you tune with one knob. M=16, efConstruction=200 at build; raise efSearch per query until recall hits target. If you already run Postgres, pgvector is enough to start.

• Retrieval quality is the product. Token-aware chunking, hybrid search, and reranking are not extras; they are the difference between a demo and a system. They stack: Anthropic measured a 67 percent failure reduction from all three together.

• Measure on real questions. Synthetic evals inflate scores by up to 37 points. Part 5 builds the eval habit properly.

What is Next in This Series

• Part 1: AI Engineer Roadmap 2026: What 390 Job Postings Actually Require

• Part 2: Python for AI Engineers: Asyncio, APIs, and What Tutorials Skip

• Part 3: How LLMs Actually Work: Tokens, Context, and Temperature

• Part 4: RAG from Zero: Embeddings, Vector DBs, and Retrieval (this article)

• Part 5: Evals: The Skill 56% of Jobs Want

• Part 6: Building Your First AI Agent with LangGraph

• Part 7: MCP: Connecting Agents to Real Tools Safely

• Part 8: LLMOps: Shipping AI to Production

Part 5 is the skill 56 percent of AI engineer postings ask for and almost nobody practices: evals. You will build a harness that grades your RAG pipeline automatically, so every change in Parts 6 through 8 gets measured instead of vibed.

---

Originally published on LogicStack. Run the code examples right in your browser with our free interactive Python and SQL playgrounds — no setup, no sign-up.