DEV Community: Krishnendu Chatterjee

How to build a RAG system from scratch in Python (chunk embed retrieve cite)( https://ai.studybydoing.in)

Krishnendu Chatterjee — Fri, 18 Sep 2026 03:21:53 +0000

Most "RAG tutorials" hand you a framework and a .from_documents() one-liner, and you never actually see what happens inside. So I built one by hand — chunking, embeddings, a tiny
vector store, hybrid retrieval, re-ranking, and cited generation — to understand each moving part. Here's the mental model and the two pieces that matter most.

## What RAG actually is

An LLM only knows what was in its training data. RAG (Retrieval-Augmented Generation) lets it answer questions about your private/current documents by retrieving relevant snippets at
query time and putting them in the prompt. The model then answers from that supplied context — facts, not guesses.

The pipeline has two timelines:

Offline (build the index once): Documents → Chunk (+metadata) → Embed → Vector store.
Online (per user question): Question → Retrieve (vector + keyword) → Re-rank → Build context → LLM → Answer + citations.

In one line: RAG = look things up first, then answer from what you found. The offline row is a librarian shelving books; the online row is you asking a question and getting the right
pages handed to you before you write your reply.

## The highest-leverage decision: chunking

Models retrieve chunks, not whole documents — so how you split matters more than almost anything else:

Too big → irrelevant text dilutes the answer and wastes tokens.
Too small → facts get split across chunks.

  import re

  def chunk_text(text, source, target_words=120, overlap=25):
      """Split on paragraphs, then pack into ~target_words chunks with overlap."""
      paras = [p.strip() for p in re.split(r"\n\s*\n", text) if p.strip()]
      chunks, buf = [], []
      for p in paras:
          buf.append(p)
          if sum(len(x.split()) for x in buf) >= target_words:
              chunks.append(" ".join(buf))
              buf = buf[-1:]  # carry last para as overlap
      if buf:
          chunks.append(" ".join(buf))
      return [{"text": c, "source": source, "id": f"{source}#{i}"} for i, c in enumerate(chunks)]

The rules that survived the labs:

Split on semantic boundaries (paragraphs, headings) — not blind fixed windows.
Keep 10–20% overlap so a fact near a boundary survives in at least one chunk.
Carry metadata (source, section, URL, date) — you need it for citations and filtering.
Aim ~100–250 words for precision; add a "parent document" fallback for context.

## Embeddings + a vector store in ~15 lines

An embedding turns text into a vector where similar meanings sit close together. Store each chunk's vector; at query time, embed the question and find the nearest chunks by cosine
similarity.

  import numpy as np
  from sentence_transformers import SentenceTransformer  # swap for any provider

  _model = SentenceTransformer("all-MiniLM-L6-v2")

  def embed(texts):
      # normalize -> cosine similarity collapses into a plain dot product
      return np.asarray(_model.encode(texts, normalize_embeddings=True))

  class VectorStore:
      def add(self, chunks):
          self.chunks = chunks
          self.vecs = embed([c["text"] for c in chunks])   # (N chunks x d)

      def search(self, query, k=4):
          q = embed([query])[0]
          sims = self.vecs @ q                             # similarity to every chunk, one step
          top = np.argsort(-sims)[:k]
          return [(self.chunks[i], float(sims[i])) for i in top]

Two things clicked for me here:

embed() is the one place text becomes numbers — wrap it behind a single function so you can swap providers without touching anything else. Golden rule: embed questions and documents with the same model.
This is genuinely what Pinecone / FAISS / pgvector do under the hood — they just add persistence, scale, and speed. Building the naive version first makes the real ones far less magical.

## Then: hybrid retrieval, re-ranking, and citations

The last labs add the parts that separate a demo from something usable:

Hybrid search — combine meaning-based (vector) with exact-word (BM25) so you don't miss literal matches like error codes or names.
Re-ranking — reorder candidates so the best rise to the top before they hit the prompt.
Grounded, cited generation — answer only from retrieved context, cite the source chunk, and refuse when the context doesn't contain the answer (this is what kills hallucinations in practice).

## The mental model to keep

RAG beats a bigger context window / fine-tuning when your knowledge is private, changing, or needs citations.
Chunking is the highest-leverage knob — get it wrong and nothing downstream saves you.
Retrieval is hybrid + re-rank, not just "nearest vector."
Generation must be grounded and honest — cite, or refuse.

If you want to run each stage yourself (the lesson has an in-browser Python terminal, no setup) the full build is here 👉 Build a RAG System From
Scratch. It's part of a free course that builds RAG, agents, eval, and production LLM systems by hand:
ai.studybydoing.in.

What is a KV cache in LLM inference? (and why it, not the weights, limits your throughput)

Krishnendu Chatterjee — Thu, 17 Sep 2026 18:13:53 +0000

I've been working through the inference chapter of a free course I've been studying — AI Engineering: Zero to Production — and
the KV-cache lesson finally made something click that I'd been hand-waving for months. Sharing my notes here in case it helps someone else.

The thing that surprised me: I always assumed the model weights were what filled up the GPU. Turns out the thing that actually caps how many users you can serve at
once is usually the KV cache. Here's what I took away.

## What the KV cache actually is

At each decode step, the model attends to every previous token. Recomputing the keys and values for all of them, every step, would be hugely wasteful — so they get
stored. That store is the KV cache.

One decode step, in plain terms:

New token — the latest token the model is processing.
Compute its K, V — think of K ("key") as what this token is about and V ("value") as the info it carries.
Append to the cache — this is why the cache grows every single step, and why long chats keep eating memory.
Attend over all cached K, V — to pick the next word, the model compares the new token against everything stored, without recomputing it. That's the whole speed win.

So: the KV cache is the model remembering the K/V of past tokens so it never redoes that work. Fast decode, but it's the main thing eating GPU memory.

## The part that actually surprised me — the arithmetic

  def kv_bytes(layers, heads, head_dim, seq_len, batch=1, bytes_per=2):
      # leading 2 = we store BOTH K and V
      return 2 * layers * heads * head_dim * seq_len * batch * bytes_per

  b = kv_bytes(layers=32, heads=32, head_dim=128, seq_len=2048)
  print(f"{b / 1024**2:.1f} MB")   # 1024.0 MB → 1 GiB for a SINGLE 2k sequence

~1 GB of KV cache for one 2,048-token request on an ordinary 32-layer/32-head model. That was the "oh" moment for me — the cache, not the weights, is what bounds
concurrency. And it grows on two axes at once:

Context length — longer prompts = bigger cache per request.
Concurrency — every simultaneous request needs its own.

## Two levers I hadn't fully understood

PagedAttention (the vLLM idea): instead of each request reserving one big contiguous block for its max possible length (wasteful), it manages the cache like OS
virtual memory — non-contiguous pages on demand. That's how a server fits way more concurrent requests in the same GPU.

Prompt caching: if requests share a prefix (long system prompt, RAG context, few-shot preamble), you cache its KV and skip re-prefilling it. On hosted APIs it's a
billing discount; self-hosting it's real compute saved.

  from anthropic import Anthropic

  client = Anthropic()
  BIG_SYSTEM = open("policy_manual.txt").read()   # long, reused preamble

  resp = client.messages.create(
      model="claude-opus-4-8",
      max_tokens=300,
      system=[{
          "type": "text",
          "text": BIG_SYSTEM,
          "cache_control": {"type": "ephemeral"},   # <- marks this block cacheable
      }],
      messages=[{"role": "user", "content": "Summarize section 4."}],
  )

  print(resp.usage)  # cache_creation_input_tokens vs cache_read_input_tokens

What made it concrete: run it twice. First call = big cache write, ~zero read. Second call = the read jumps, the write drops. The savings only land on the 2nd+
call that reuses the prefix.

## The mental model I'm keeping

KV cache = remembered K/V for past tokens → fast decode.
Grows with context length × concurrency → it, not the weights, limits how many users you serve.
PagedAttention cuts waste when self-hosting; prompt caching reuses shared prefixes to cut repeat cost.

If you want to actually run the size formula and the caching example (the lesson has an in-browser Python terminal, no setup), the original is here 👉 KV-Cache &
Attention Memory in LLM Inference. The whole course builds RAG, agents, eval, and production LLM stuff by hand:
ai.studybydoing.in.