DEV Community: surajrkhonde

Phase 2: Embeddings & Semantic Search

surajrkhonde — Tue, 23 Jun 2026 08:51:10 +0000

From Text to Vectors: The Complete Story

The Story Starts: Why Can't We Just Search for Words?

👦 Nephew: Uncle! Phase 1 was done. Now we have clean chunks. Can't we just search for keywords?

👨‍🦳 Uncle: Let me ask you something. Someone applies for a job and says:

Resume says: "I managed a team of 10 people for 3 years"

Job Description asks: "Do you have leadership experience?"

Are these related?

👦 Nephew: Obviously yes! But the words are different. Team management and leadership.

👨‍🦳 Uncle: Exactly. A computer doing simple keyword search would say NO.

Keyword Search:
Resume: "team management"
Question: "leadership"
Match? NO (different words)

This is the problem Phase 2 solves.

👦 Nephew: How?

👨‍🦳 Uncle: By understanding meaning, not just words.

PHASE 2 OVERVIEW: Three Key Concepts

Phase 1: Document Ingestion ✅
    ↓
Phase 2: Embeddings & Semantic Search ← WE ARE HERE
    │
    ├─ Step 1: Tokenization
    │         (Text → Token IDs)
    │
    ├─ Step 2: Embedding Layer
    │         (Token IDs → Vectors)
    │
    └─ Step 3: Semantic Search
              (Find Similar Vectors)
    ↓
Phase 3: Safety & Production

STEP 1: TOKENIZATION

What Is Tokenization?

👨‍🦳 Uncle: Computers don't understand words like "React" or "JavaScript". They only understand numbers.

A tokenizer converts text into numbers called token IDs.

Text:           "React is awesome"
                    ↓
Tokenizer converts to:
Token IDs:      [4521, 318, 9876]

👦 Nephew: But why not just one number per word?

👨‍🦳 Uncle: Good question. Look at this:

Word:          "JavaScriptDeveloper"

Tokenizer might see:
["Java", "Script", "Developer"]  ← 3 tokens
or
["JavaScript", "Developer"]      ← 2 tokens

Depends on the tokenizer's vocabulary.

Why Is This Important?

👨‍🦳 Uncle: Because the embedding model doesn't receive text. It receives token IDs.

Text Input:
"Employees must submit resignation 60 days before"
           ↓
Tokenizer (internal to embedding model):
[1001, 245, 782, 4521, 99, 1234, ...]
           ↓
Embedding Layer looks up each token:
1001 → [0.12, -0.45, 0.78, ...]
245  → [0.88, -0.12, 0.34, ...]
           ↓
Combines them into final vector

Real Example: Your Name

👨‍🦳 Uncle: Your name is Suraj Khonde. How does the tokenizer handle it?

If the tokenizer has seen "Suraj" often:
"Suraj" → Single Token → 5234

If not:
"Suraj" → Breaks into: ["Su", "raj"]
                      → [5234, 9876]

Same with "Khonde":
If common: → Single token
If rare:   → Multiple tokens

Result might be:
"Suraj Khonde" → [5234, 9876, 7621, 3344]

👦 Nephew: So less common words use more tokens?

👨‍🦳 Uncle: Exactly! This is why we count tokens, not words.

Word count: 2 words
Token count: 4 tokens

For billing and cost, we count tokens!

STEP 2: EMBEDDING LAYER

The Bridge: From Numbers to Meaning

👨‍🦳 Uncle: Now we have token IDs. But they're meaningless.

Token ID: 4521

This is just an identifier. Like an employee ID.
Employee #4521 is just a number.

The embedding layer transforms this into meaningful vectors.

👦 Nephew: How?

👨‍🦳 Uncle: Think of a giant lookup table inside the model:

Token ID    Vector (Meaning)
─────────────────────────────────────────
4521   →  [0.13, -0.46, 0.79, ... 1536 values]
318    →  [0.91, -0.21, 0.12, ... 1536 values]
9876   →  [0.44, 0.77, -0.33, ... 1536 values]

When model sees token 4521, it looks up:
4521 → [0.13, -0.46, 0.79, ...]

👦 Nephew: What are those numbers? [0.13, -0.46, 0.79]?

👨‍🦳 Uncle: Those numbers encode meaning. They describe what the token represents.

Who Created These Numbers?

👨‍🦳 Uncle: When the embedding model was first created, these numbers were mostly random.

But then something magical happened: training.

During training on billions of documents, the model learned:

React appears with JavaScript many times
↓
So React and JavaScript vectors move closer

Node.js appears with Express many times
↓
So Node.js and Express vectors move closer

Dog appears with Puppy many times
↓
So Dog and Puppy vectors move closer

After training on billions of examples, the vectors ended up in meaningful positions.

👦 Nephew: So React and JavaScript have similar vectors?

👨‍🦳 Uncle: Yes! That's why semantic search works.

KEY CONCEPT: Why 1536 Dimensions?

The Problem with Too Few Dimensions

👨‍🦳 Uncle: Suppose the embedding only used 1 number:

React   → 5
Python  → 7
JavaScript → 6

Now what? React and JavaScript are close (5 vs 6), 
but so are React and Python (5 vs 7).
All three look similar with one number!

❌ Not enough information.

Why So Many Numbers?

👨‍🦳 Uncle: Language is incredibly complex. The vector needs to capture:

Dimension 1: Frontend vs Backend
Dimension 2: Web vs Systems Programming
Dimension 3: Dynamic vs Compiled
Dimension 4: Syntax Style
Dimension 5: Ecosystem Size
...
Dimension 1536: [something the model learned]

Each dimension captures a different aspect of meaning.

👦 Nephew: So with 1536 dimensions, React can be described as:

React: [0.9 (very frontend), 0.2 (mostly web), 0.95 (dynamic), ...]
Node.js: [0.3 (somewhat frontend), 0.8 (backend), 0.95 (dynamic), ...]
Python: [0.1 (rarely frontend), 0.9 (backend), 0.5 (compiled), ...]

👨‍🦳 Uncle: Exactly! And notice:

React and Node.js both have 0.95 (dynamic) → similar there
React is 0.9 frontend, Node is 0.3 → different there
Python is 0.5 (middle) → different from both

This richness requires many dimensions.

Mental Model: Google Maps

👨‍🦳 Uncle: Think of Google Maps. Bangalore has coordinates:

Latitude: 12.9716
Longitude: 77.5946

These 2 numbers uniquely identify Bangalore.

Embeddings work similarly:

React has 1536 coordinates in "semantic space":
[0.13, -0.46, 0.79, ..., X1536]

This position uniquely identifies React's meaning.

STEP 3: SEMANTIC SEARCH

Keyword Search vs Semantic Search

👦 Nephew: So now we have vectors for all chunks. How do we search?

👨‍🦳 Uncle: Let me show you the difference:

Keyword Search:
─────────────────
User asks: "What is notice period?"
System searches for exact words: "notice" + "period"

Document contains: "resignation requirements"
Match? NO ❌

Semantic Search:
────────────────
User asks: "What is notice period?"
System converts to vector: [0.21, -0.44, 0.88, ...]

Document: "resignation requirements"
System converts to vector: [0.22, -0.40, 0.85, ...]

Compare vectors: Very similar! ✅

Cosine Similarity: How Similar Are Two Vectors?

👨‍🦳 Uncle: We need a way to measure "how similar" two vectors are.

Think of vectors as arrows pointing in a direction.

Vector A: ↗️ (pointing up-right)
Vector B: ↗️ (pointing up-right)
Similarity: HIGH (same direction)

Vector A: ↗️
Vector B: →️ (pointing right)
Similarity: MEDIUM (somewhat same)

Vector A: ↗️
Vector B: ↙️ (pointing down-left)
Similarity: LOW (opposite directions)

👦 Nephew: So cosine similarity measures "angle between vectors"?

👨‍🦳 Uncle: Exactly! In mathematics:

Cosine Similarity Score Range:
─────────────────────────────
1.0  = Vectors point in exactly same direction (identical)
0.5  = 60° angle between them (medium similarity)
0.0  = 90° angle (unrelated)
-0.5 = 120° angle (somewhat opposite)
-1.0 = Opposite directions (contradictory)

Real Example

👨‍🦳 Uncle: Suppose vector DB contains:

Question Vector: 
"What is notice period?" 
→ [0.21, -0.44, 0.88, ...]

Document Chunks:
─────────────────────────────────────
Chunk 1: "Employees must serve 30 days notice period"
→ [0.22, -0.40, 0.85, ...]
Cosine Similarity: 0.95 ✅ (Very Similar)

Chunk 2: "Resignation must be submitted in writing"
→ [0.19, -0.42, 0.87, ...]
Cosine Similarity: 0.92 ✅ (Similar)

Chunk 3: "Office timings are 9 AM to 5 PM"
→ [0.05, 0.88, -0.12, ...]
Cosine Similarity: 0.15 ❌ (Not Related)

Chunk 4: "Coffee is available in cafeteria"
→ [-0.45, 0.12, 0.33, ...]
Cosine Similarity: -0.05 ❌ (Unrelated)

The system returns the top 3 chunks with highest similarity.

👦 Nephew: Why top 3 and not all?

👨‍🦳 Uncle: Because:

Efficiency - Why send irrelevant chunks to LLM?
Context window - LLM has limited capacity
Cost - More tokens = more money
Quality - Less noise = better answers

This is called Top-K Retrieval (usually K=3-5).

Complete Flow: Query Time

👨‍🦳 Uncle: Now let's trace a complete query:

USER ASKS:
"What is our notice period?"

STEP 1: Tokenization (inside embedding model)
─────────────────────────────────────
"What is our notice period?"
→ Tokenizer
→ [789, 44, 123, 555, 321]

STEP 2: Embedding (inside embedding model)
─────────────────────────────────────
Token IDs: [789, 44, 123, 555, 321]
→ Embedding Layer looks up each
→ Combines into one vector
Query Vector: [0.21, -0.44, 0.88, ... 1536 values]

STEP 3: Vector Database Search
─────────────────────────────────────
Query Vector: [0.21, -0.44, 0.88, ...]
↓
Compare against ALL stored chunk vectors
↓
Compute cosine similarity for each
↓
Sort by score (highest first)
↓
Return Top-K (usually 5)

STEP 4: Pass to LLM
─────────────────────────────────────
Retrieved chunks:
1. "Employees receive 30 days notice period"
2. "Manager approval required for resignation"
3. "Notice period starts from submission date"

Prompt:
Context:
[chunk 1]
[chunk 2]
[chunk 3]

Question: What is our notice period?

STEP 5: Generate Answer
─────────────────────────────────────
LLM reads context and answers:
"The notice period is 30 days from submission."

The Question Everyone Asks: "Where's the Magic?"

👦 Nephew: So the magic is... the vectors?

👨‍🦳 Uncle: Yes and no. The magic is in how vectors are created.

During training on billions of documents, the model learned:

Notice period
↔
Resignation requirements
↔
Notice duration

These concepts appear together many times.
So their vectors end up close to each other.

It's not magic. It's pattern recognition at scale.

Why Different Models Give Different Vectors

👦 Nephew: If I use OpenAI embeddings, then switch to Cohere, what happens?

👨‍🦳 Uncle: Different model = different vector space.

OpenAI text-embedding-3:
"React" → [0.13, -0.46, 0.79, ...]

Cohere Embed:
"React" → [0.57, 0.12, -0.34, ...]

Different numbers!

👦 Nephew: Can I compare them?

👨‍🦳 Uncle: NO! They're in different "semantic spaces".

It's like:
Celsius: 25 degrees
Fahrenheit: 77 degrees

Different scales. Can't compare directly.

When you change embedding models, you must re-embed everything.

Production Concern: Model Migrations

👨‍🦳 Uncle: This is where RAG gets expensive.

Your company has:

100,000 documents
After chunking: 5 million chunks
After embedding: 5 million vectors stored

New, better embedding model appears:

Old Model: 92% accuracy
New Model: 95% accuracy

But cost of migration: $50,000 (re-embedding 5M vectors)

CTO's question:
"Is 3% improvement worth $50,000?"

Usually NO. So we don't upgrade.

👦 Nephew: So companies are stuck with old models?

👨‍🦳 Uncle: Yes, until the improvement is significant enough. This is why:

Many enterprises still use old APIs
Many use old databases
Migration is expensive and risky

The best technology doesn't always win. The best ROI wins.

Smart Companies: Gradual Migration

👨‍🦳 Uncle: Smart companies do:

Old documents (stored 2 years ago)
→ Use old embedding model

New documents (uploaded today)
→ Use new embedding model

Gradually migrate in background:
- When storage capacity allows
- When costs are low
- When nobody notices the performance dip

Why Store Original Chunk Text?

👦 Nephew: Uncle, we stored the vector. Why also store the chunk text?

👨‍🦳 Uncle: Great question! It's for the future.

Scenario:

Today: Using Model V1
Store:
{
  "chunk_text": "Employees receive 30 days notice",
  "embedding": [0.21, -0.44, 0.88, ...],
  "embedding_model": "text-embedding-3-small"
}

Tomorrow: Model V2 is available (better!)

Can we do:
Old Vector → New Vector?
NO ❌

But we can do:
Old Text → New Model → New Vector
YES ✅

This is why chunk_text is precious!

Without chunk_text:

You'd have to:
1. Find original PDF
2. Parse it again
3. Clean it again
4. Chunk it again
5. Embed it with new model

= Huge pain and cost

Vector Database: Making Search Fast

The Problem: 5 Million Vectors!

👦 Nephew: If we have 5 million vectors, don't we compare with all of them?

👨‍🦳 Uncle: That would be:

Query Vector
↓
Compare with 5 million vectors = 5 million calculations
↓
Takes 30 seconds

User waits 30 seconds. Bad experience.

This is where Vector Databases come in (Pinecone, Qdrant, pgvector).

How Vector Databases Work Fast

👨‍🦳 Uncle: They use indexing. Like Google Maps.

Google doesn't search the entire world to find Bangalore. It uses:

Regions → Countries → States → Cities → Exact Location

Similarly, vector databases use:

Approximate Nearest Neighbor (ANN)
↓
Divides the semantic space into regions
↓
When you search, quickly finds which region
↓
Only searches that region
↓
Returns nearest neighbors in 100ms instead of 30 seconds

Interview-Level Answers

Q1: What is Tokenization?

👨‍🦳 Uncle: Here's your answer:

"Tokenization is the process of breaking down text into smaller units called tokens and converting them into token IDs. Since embedding models work with numbers, not text, a tokenizer converts words or subwords into numeric identifiers. For example, 'React is awesome' might become [4521, 318, 9876] where each number represents a token. The granularity depends on the tokenizer's vocabulary - common words might be single tokens, while rare words could be split into multiple tokens."

Q2: What is an Embedding?

"An embedding is a vector representation of text that captures semantic meaning. When a token ID is passed to the embedding layer, the model looks it up in a learned embedding matrix, returning 1536 numbers (dimensions) that collectively represent the token's meaning. These numbers are learned during the model's training process - similar concepts end up with vectors that point in similar directions in this high-dimensional space."

Q3: Why 1536 Dimensions?

"One number cannot capture the complexity of language. The embedding model needs to represent many relationships simultaneously: whether a word relates to programming, frontend vs backend, syntax style, ecosystem size, etc. With 1536 dimensions, each dimension can capture a different aspect of meaning. This richness allows the model to distinguish between concepts that would look similar with fewer dimensions."

Q4: How Does Semantic Search Work?

"Semantic search works by converting both queries and documents into embeddings (vectors), then using cosine similarity to find documents whose vectors point in similar directions. If a user asks 'notice period' and a document discusses 'resignation requirements', both will have vectors that point in similar directions despite using different words, because they were learned to be related during the model's training on billions of documents."

Q5: What Happens When You Change Embedding Models?

"When you switch embedding models, you must re-embed all stored chunks because different models produce vectors in different vector spaces. You can't compare vectors from Model A with query vectors from Model B - it's like mixing Celsius and Fahrenheit. Companies must evaluate the accuracy improvement against the cost of re-indexing. For example, improving from 92% to 95% accuracy might not justify a $50,000 re-embedding cost."

The Complete Architecture So Far

PHASE 1: DOCUMENT INGESTION ✅
─────────────────────────────
PDF Upload
  ↓
File Hash Check
  ↓
Parse & Clean
  ↓
Chunking
  ↓
Deduplication
  ↓
Store Chunk Text

PHASE 2: EMBEDDINGS ← YOU ARE HERE
─────────────────────────────
Chunk Text
  ↓
Tokenization (internal to embedding model)
  ↓
Token IDs
  ↓
Embedding Layer
  ↓
Vector
  ↓
Store in Vector Database

PHASE 3: QUERY TIME
─────────────────────────────
User Question
  ↓
Tokenization + Embedding
  ↓
Query Vector
  ↓
Vector Database Search (ANN)
  ↓
Top-K Similar Chunks
  ↓
Send to LLM
  ↓
Generate Answer

Summary: What Phase 2 Solves

Problem	Phase 1	Phase 2
Large Documents	✅ Chunking	✅ Handled
Exact Keyword Matching	✅ Works	❌ Fails for synonyms
Similar Concepts	❌ Keyword search misses	✅ Vector similarity finds
Scale (Millions of chunks)	❌ Search is slow	✅ Vector DB indexes fast
Meaning Understanding	❌ Can't understand meaning	✅ Embeddings capture it

Key Takeaways

Tokenization = Text → Numbers (done by embedding model internally)
Embedding = Numbers → Meaningful vectors (captures semantic relationships)
Semantic Search = Find vectors pointing in similar direction (cosine similarity)
Vector Databases = Index for fast nearest-neighbor search
Re-embedding Cost = Why companies don't switch models often
Chunk Text Storage = Insurance for future model upgrades

Next: Phase 2 Node.js Code

Now that you understand:

How tokenization works
How embeddings are created
How semantic search finds similar documents
How to measure similarity
Production considerations for migrations

We'll implement Phase 2 in Node.js:

Create embeddings for chunks
Store vectors in pgvector
Implement semantic search
Handle model versioning

Ready?

Remember: Less noise, more action. Phase 2 is where the actual intelligence happens.

Phase 1: Document Ingestion - The Hidden Complexity Before Embeddings

surajrkhonde — Mon, 22 Jun 2026 10:46:44 +0000

The Complete Story: Why Most RAG Systems Fail Before They Start

The Story Begins: Why Your Upload Button Is Just The Beginning

👦 Nephew: Uncle! I finally built my RAG system. User uploads a PDF, system finds answers. Simple, right?

👨‍🦳 Uncle: (smiles knowingly) You uploaded a PDF and got an answer?

👦 Nephew: Yes! It works!

👨‍🦳 Uncle: Did you get the right answer?

👦 Nephew: Well... sometimes. Why?

👨‍🦳 Uncle: Because between "user uploads PDF" and "system creates embeddings", there are 15 critical steps. Skip even one, and your system fails silently. You get wrong answers and don't know why.

👦 Nephew: 15 steps? I just embedded the text!

👨‍🦳 Uncle: Exactly. That's the problem. Come, let me show you what production engineers actually do.

The 15 Steps of Phase 1: Document Ingestion

👨‍🦳 Uncle: Think of it like cooking biryani. You don't just dump rice and meat, right?

👦 Nephew: No, you prepare everything first!

👨‍🦳 Uncle: Exactly. Before you cook, you:

Wash rice
Soak rice
Prepare meat
Marinate meat
Chop onions
... and many more steps

Only THEN you cook.

The same with documents. Before embeddings, you must:

1. Document Upload (user action)
2. File Hashing (did we see this before?)
3. PDF Parsing (extract text)
4. Text Extraction (convert PDF → text)
5. Text Cleaning (remove junk)
6. Metadata Extraction (add context)
7. Chunking (split smartly)
8. Chunk Boundaries (don't break meaning)
9. Chunk Size (1000 tokens, not 100)
10. Overlap (context continuity)
11. Chunk Hashing (detect changes)
12. Deduplication (prevent duplicates)
13. Versioning (handle updates)
14. Incremental Ingestion (avoid re-embedding)
15. Cost Optimization (save money)

ONLY THEN:
   ↓
   Embeddings
   ↓
   Vector DB

👦 Nephew: That's a lot! Where do I start?

👨‍🦳 Uncle: With step 1. Let's go slow. The foundation must be solid.

PHASE 1, STEP 1-2: Document Upload & File Hashing

Why Does File Hashing Matter?

👦 Nephew: Why hash a file? Why not just upload it?

👨‍🦳 Uncle: Because humans are lazy. The HR person uploads the same PDF three times. Your system processes it three times. Three embeddings created. Three times the cost.

👦 Nephew: So hash prevents duplicates?

👨‍🦳 Uncle: Yes. But here's the trick: don't hash the filename.

👦 Nephew: Why not?

👨‍🦳 Uncle: Because someone could change the content and keep the same name. Look:

File: HR_Policy.pdf (Version 1)
Content: "30 days notice required"
Filename Hash: HR_Policy.pdf

File: HR_Policy.pdf (Version 2)
Content: "7 days notice required"
Filename Hash: HR_Policy.pdf (SAME!)

System thinks: Same file!
Reality: Different policies!

That's a disaster.

👦 Nephew: So we hash the content?

👨‍🦳 Uncle: Yes. The actual binary data.

HR_Policy.pdf (Version 1)
↓
[PDF binary bytes: 0xAA 0xBB 0xCC ...]
↓
SHA256
↓
A7B82C1F9D3E...

HR_Policy.pdf (Version 2)
↓
[PDF binary bytes: 0xAA 0xBB 0xDD ...]  ← Different!
↓
SHA256
↓
X9Z47M3Q2K1L...  ← Different hash!

System detects: New file, process it.

Step 1-2: Node.js Implementation

// src/ingestion/fileHasher.ts

import crypto from 'crypto';
import fs from 'fs';
import path from 'path';
import db from '../config/database';
import logger from '../utils/logger';

interface FileHashResult {
  fileName: string;
  fileHash: string;
  alreadyExists: boolean;
  fileSize: number;
}

/**
 * Step 1-2: Upload file and check if already processed
 * 
 * ⚠️ CRITICAL POINTS:
 * 1. Hash FILE CONTENT, not filename
 * 2. Same content → same hash (deterministic)
 * 3. Different content → different hash (even if same filename)
 * 4. Check database before processing
 */
export async function handleFileUpload(
  filePath: string,
  tenantId: string
): Promise<FileHashResult> {
  try {
    const fileName = path.basename(filePath);
    const fileSize = fs.statSync(filePath).size;

    // Step 1: Read file and create hash from content
    logger.info('File upload started', { fileName, fileSize });

    const fileContent = fs.readFileSync(filePath);
    const fileHash = crypto
      .createHash('sha256')
      .update(fileContent)
      .digest('hex');

    logger.debug('File hashed', { fileName, fileHash, contentBytes: fileContent.length });

    // Step 2: Check if this exact file was already processed
    const existingFile = await db.oneOrNone(
      `
      SELECT id, created_at 
      FROM documents 
      WHERE file_hash = $1 AND tenant_id = $2
      `,
      [fileHash, tenantId]
    );

    if (existingFile) {
      logger.warn('Duplicate file detected', {
        fileName,
        fileHash,
        firstUploadedAt: existingFile.created_at
      });

      return {
        fileName,
        fileHash,
        alreadyExists: true,
        fileSize
      };
    }

    // Step 3: File is new, save metadata
    const documentRecord = await db.one(
      `
      INSERT INTO documents (tenant_id, file_name, file_hash, file_size, status)
      VALUES ($1, $2, $3, $4, $5)
      RETURNING id, created_at
      `,
      [tenantId, fileName, fileHash, fileSize, 'uploaded']
    );

    logger.info('New file registered', {
      documentId: documentRecord.id,
      fileName,
      fileHash: fileHash.substring(0, 12) + '...'
    });

    return {
      fileName,
      fileHash,
      alreadyExists: false,
      fileSize
    };

  } catch (error: any) {
    logger.error('File upload error', { error: error.message });
    throw error;
  }
}

/**
 * Verify file integrity (optional but recommended)
 * If file is corrupted, don't process it
 */
export function verifyFileIntegrity(filePath: string, expectedHash: string): boolean {
  const fileContent = fs.readFileSync(filePath);
  const actualHash = crypto
    .createHash('sha256')
    .update(fileContent)
    .digest('hex');

  return actualHash === expectedHash;
}

// Database schema for documents table
export const documentsTableSQL = `
CREATE TABLE IF NOT EXISTS documents (
  id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
  tenant_id UUID NOT NULL REFERENCES tenants(id),
  file_name VARCHAR(500) NOT NULL,
  file_hash VARCHAR(64) NOT NULL,  -- SHA256 produces 64 char hex
  file_size BIGINT NOT NULL,
  status VARCHAR(50) DEFAULT 'uploaded',  -- uploaded, parsing, parsed, chunking, chunked, embedding, complete
  error_message TEXT,
  uploaded_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
  created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,

  -- Unique constraint: same file can't be uploaded twice for same tenant
  CONSTRAINT unique_file_per_tenant UNIQUE(tenant_id, file_hash),

  INDEX idx_file_hash (file_hash),
  INDEX idx_tenant_status (tenant_id, status)
);
`;

👦 Nephew: So if someone uploads the same PDF twice, we detect it and skip?

👨‍🦳 Uncle: Exactly. And we save embedding cost. Which is the most expensive step.

PHASE 1, STEP 3: PDF Parsing - Choosing the Right Tool

👦 Nephew: Now we have the PDF. How do we extract text?

👨‍🦳 Uncle: This is where the real decision happens. There are five major tools. Each has tradeoffs.

👦 Nephew: Five?! Which one should I use?

👨‍🦳 Uncle: Depends on your documents. Let me show you.

The PDF Parsing Landscape

Simple Text PDFs          →  pdf-parse (cheap, simple)
                          ↓
Mixed content (text + tables) →  PDFPlumber (better)
                          ↓
Complex documents        →  Unstructured (production-grade)
                          ↓
Advanced documents       →  LlamaParse (state-of-art)
(tables, images, OCR)
                          ↓
Enterprise documents     →  Azure Document Intelligence
(forms, invoices, scans)

Tool Comparison

Tool	Best For	Cost	Speed	Table Support	OCR	Metadata	Production Ready
pdf-parse	Simple text	₹0 (free)	⚡⚡⚡ Fast	✗ No	✗	✗	⚠️ Hobby
PDFPlumber	Text + tables	₹0	⚡⚡ Medium	✓ Basic	✗	⚠️ Limited	⚠️ Small
Unstructured	Normal docs	₹50-200/mo	⚡ Slow	✓ Good	✓ Basic	✓ Good	✓ Yes
LlamaParse	Complex docs	₹100-500/mo	⚡ Slow	✓ Excellent	✓ Advanced	✓ Excellent	✓ Yes
Azure Doc Int.	Enterprise	₹500-2000/mo	⚡ Medium	✓ Perfect	✓ Perfect	✓ Perfect	✓ Enterprise

👨‍🦳 Uncle: Let me explain each.

Tool 1: pdf-parse (Free, Simple)

// Simple approach - good for learning, bad for production

const pdf = require('pdf-parse');
const fs = require('fs');

async function extractTextFromPDF(filePath) {
  const dataBuffer = fs.readFileSync(filePath);

  const data = await pdf(dataBuffer);

  console.log(data.text);  // Raw text
  // Output: "Company Policy Leave Policy Employees receive..."
}

👨‍🦳 Uncle: Notice what we lost?

Original PDF:
═══════════════════════════════════════
COMPANY POLICY
───────────────────────────────────────
Leave Policy (HEADING)

Employees are entitled to 24 paid leaves annually.
(PARAGRAPH)

Leave Types:
- Annual Leave
- Casual Leave
(LIST)
═══════════════════════════════════════

pdf-parse Output:
"COMPANY POLICY Leave Policy Employees are entitled to 24 paid leaves annually. Leave Types: Annual Leave Casual Leave"

LOST:
✗ Heading level
✗ List structure
✗ Paragraph breaks
✗ Section organization
✗ Tables (if any)

👦 Nephew: So we just get text soup?

👨‍🦳 Uncle: Yes. And when you embed soup, you get soup answers.

Tool 2: PDFPlumber (Better, Still Python)

# PDFPlumber - extracts tables better

import pdfplumber

with pdfplumber.open("document.pdf") as pdf:
    for i, page in enumerate(pdf.pages):
        # Extract text
        text = page.extract_text()

        # Extract tables (if any)
        tables = page.extract_tables()

        print(f"Page {i}:")
        print(f"Text: {text}")
        print(f"Tables: {tables}")

👨‍🦳 Uncle: Better, but still loses structure. And it's Python, not Node.js.

Tool 3: Unstructured (Production Choice)

👨‍🦳 Uncle: This is what most companies use. It preserves structure.

// Using Unstructured via API (Node.js friendly)

import axios from 'axios';
import fs from 'fs';

/**
 * Step 3: Parse PDF using Unstructured
 * 
 * IMPORTANT: Unstructured preserves document structure
 * Returns: Array of structured elements
 */
export async function parsePDFWithUnstructured(filePath: string): Promise<any[]> {
  try {
    const fileContent = fs.readFileSync(filePath);
    const base64Content = fileContent.toString('base64');

    const response = await axios.post(
      'https://api.unstructuredapp.io/general/v0/general',
      {
        file: base64Content,
        strategy: 'hi_res',  // High resolution parsing
        coordinates: true     // Preserve coordinates (useful for tables)
      },
      {
        headers: {
          'Authorization': `Bearer ${process.env.UNSTRUCTURED_API_KEY}`,
          'Content-Type': 'application/json'
        }
      }
    );

    const elements = response.data.elements;

    // Output: Structured elements
    // [
    //   { type: "Title", text: "Leave Policy", metadata: {...} },
    //   { type: "Heading", text: "Annual Leave", metadata: {...} },
    //   { type: "Paragraph", text: "Employees are entitled...", metadata: {...} },
    //   { type: "ListItem", text: "Manager approval required", metadata: {...} }
    // ]

    logger.info('PDF parsed with structure preserved', {
      elementCount: elements.length,
      types: [...new Set(elements.map((e: any) => e.type))]
    });

    return elements;

  } catch (error: any) {
    logger.error('Unstructured parsing failed', { error: error.message });
    throw error;
  }
}

👦 Nephew: So it preserves structure?

👨‍🦳 Uncle: Yes. Look at the difference:

Unstructured Output:

[
  { 
    type: "Title",
    text: "Leave Policy",
    metadata: {
      page_number: 1,
      section: "Policies"
    }
  },
  { 
    type: "Paragraph",
    text: "Employees receive 24 leaves",
    metadata: {
      page_number: 1
    }
  },
  { 
    type: "List",
    text: "Annual Leave, Casual Leave",
    metadata: {
      page_number: 2,
      list_items: 2
    }
  }
]

Now we know:
✓ What is a title
✓ What is body text
✓ What is a list
✓ Page numbers
✓ Sections

Tool 4: LlamaParse (State of the Art)

👨‍🦳 Uncle: For really complex documents, use LlamaParse.

// LlamaParse - best for complex PDFs

import axios from 'axios';
import FormData from 'form-data';
import fs from 'fs';

/**
 * Parse PDF with LlamaParse (best for:
 * - Multi-column layouts
 * - Tables with merged cells
 * - Images with text
 * - Scanned documents (OCR)
 * - Footnotes and annotations
 */
export async function parsePDFWithLlamaParse(filePath: string): Promise<any> {
  try {
    // Step 1: Upload file
    const formData = new FormData();
    formData.append('file', fs.createReadStream(filePath));
    formData.append('parsing_instruction', `
      Extract all content including:
      - Tables with proper structure
      - Column layouts
      - Images with OCR
      - Headings and sections
      - Metadata like page numbers
    `);

    const uploadResponse = await axios.post(
      'https://api.llamaindex.ai/api/parsing/upload_file',
      formData,
      {
        headers: {
          'Authorization': `Bearer ${process.env.LLAMAPARSE_API_KEY}`,
          ...formData.getHeaders()
        }
      }
    );

    const jobId = uploadResponse.data.id;

    // Step 2: Poll for results
    let result = null;
    for (let i = 0; i < 60; i++) {
      const statusResponse = await axios.get(
        `https://api.llamaindex.ai/api/parsing/job/${jobId}/result/markdown`,
        {
          headers: {
            'Authorization': `Bearer ${process.env.LLAMAPARSE_API_KEY}`
          }
        }
      );

      if (statusResponse.status === 200) {
        result = statusResponse.data;
        break;
      }

      // Wait 2 seconds before retry
      await new Promise(resolve => setTimeout(resolve, 2000));
    }

    logger.info('LlamaParse completed', { jobId });

    return {
      markdown: result,
      parsedAt: new Date()
    };

  } catch (error: any) {
    logger.error('LlamaParse failed', { error: error.message });
    throw error;
  }
}

👦 Nephew: When do I use LlamaParse vs Unstructured?

👨‍🦳 Uncle: Simple rule:

Document type?

  ├─ Simple text policies
  │  └─→ pdf-parse (free)
  │
  ├─ Text + basic tables
  │  └─→ Unstructured (cheap, good)
  │
  ├─ Complex tables, multi-column
  │  └─→ LlamaParse (excellent)
  │
  └─ Enterprise documents, forms, invoices
     └─→ Azure Document Intelligence (best)

Tool 5: Azure Document Intelligence (Enterprise)

// Azure Document Intelligence - for enterprise documents

import { DocumentAnalysisClient, AzureKeyCredential } from "@azure/ai-form-recognizer";
import fs from 'fs';

/**
 * Parse with Azure (best for:
 * - Invoices
 * - Forms
 * - Bank documents
 * - Scanned PDFs with OCR
 */
export async function parseWithAzureDocumentIntelligence(filePath: string) {
  try {
    const client = new DocumentAnalysisClient(
      process.env.AZURE_FORM_RECOGNIZER_ENDPOINT,
      new AzureKeyCredential(process.env.AZURE_FORM_RECOGNIZER_KEY)
    );

    const fileContent = fs.readFileSync(filePath);

    // Choose model based on document type
    const poller = await client.beginAnalyzeDocument(
      "prebuilt-document",  // Or: prebuilt-invoice, prebuilt-receipt, etc.
      fileContent
    );

    const result = await poller.pollUntilDone();

    // Extract structure
    const output = {
      text: result.content,
      tables: result.tables?.map(table => ({
        rows: table.rowCount,
        columns: table.columnCount,
        cells: table.cells
      })),
      forms: result.fields,  // For form data
      confidence: result.confidence
    };

    logger.info('Azure parsing complete', {
      pages: result.pages?.length,
      tables: result.tables?.length,
      confidence: output.confidence
    });

    return output;

  } catch (error: any) {
    logger.error('Azure parsing failed', { error: error.message });
    throw error;
  }
}

👦 Nephew: These all cost money?

👨‍🦳 Uncle: Yes, except pdf-parse. And here's the secret: you should still use pdf-parse for simple documents!

👦 Nephew: Why?

👨‍🦳 Uncle: Cost. If 80% of your documents are simple policies, use pdf-parse. Use expensive parsers only for 20% that need them.

PHASE 1, STEP 4-5: Text Extraction & Cleaning

The Garbage Problem

👨‍🦳 Uncle: After parsing, you get garbage.

Raw Extracted Text:

═════════════════════════════════════════════════════════════
Company Name                          Page 1
═════════════════════════════════════════════════════════════
[CONFIDENTIAL]

LEAVE POLICY

[Document Version: 2.3]

Employees are entitled to 24 paid leaves annually.
These leaves must be approved by:
1. Direct Manager
2. HR Department

[CONFIDENTIAL]
═════════════════════════════════════════════════════════════
Company Name                          Page 2
═════════════════════════════════════════════════════════════

👦 Nephew: What's wrong with this?

👨‍🦳 Uncle: When you embed "[CONFIDENTIAL]", your system learns that every policy is confidential. When someone asks "Is this policy confidential?", the answer is always YES!

// Step 4-5: Clean the text

import logger from '../utils/logger';

/**
 * Step 4-5: Extract and Clean text
 * 
 * Remove:
 * - Page headers/footers
 * - Repeated watermarks
 * - Extra whitespace
 * - Page numbers
 * - Confidential markings (if generic)
 */
export function cleanExtractedText(rawText: string): string {
  let cleaned = rawText;

  // Remove page headers (appears at top of every page)
  cleaned = cleaned.replace(
    /^Company Name\s+Page \d+\s*$/gm,
    ''
  );

  // Remove confidential markings (if generic)
  cleaned = cleaned.replace(
    /\[CONFIDENTIAL\]\n?/g,
    ''
  );

  // Remove document version
  cleaned = cleaned.replace(
    /\[Document Version: [\d.]+\]/g,
    ''
  );

  // Remove page numbers alone on a line
  cleaned = cleaned.replace(
    /^\s*Page \d+\s*$/gm,
    ''
  );

  // Normalize whitespace (multiple spaces/newlines → single space)
  cleaned = cleaned.replace(/\s+/g, ' ');

  // Remove leading/trailing whitespace
  cleaned = cleaned.trim();

  logger.debug('Text cleaned', {
    originalLength: rawText.length,
    cleanedLength: cleaned.length,
    removed: rawText.length - cleaned.length
  });

  return cleaned;
}

// Example
const raw = `
═════════════════════
Company Name    Page 1
═════════════════════
[CONFIDENTIAL]

LEAVE POLICY

Employees get 24 leaves.

[CONFIDENTIAL]
═════════════════════
Company Name    Page 2
═════════════════════
`;

const clean = cleanExtractedText(raw);
// Output: "LEAVE POLICY Employees get 24 leaves."

PHASE 1, STEP 6: Metadata Extraction

👨‍🦳 Uncle: Metadata is information ABOUT the chunk, not IN the chunk.

👦 Nephew: Like what?

👨‍🦳 Uncle: Like:

Document: HR_Policy.pdf
Department: Human Resources
Section: Leave Policy
Page: 3
Version: 2.3
Effective Date: 2025-01-01
Author: HR Manager
Source: Internal Wiki

This metadata helps retrieval.

👦 Nephew: How?

👨‍🦳 Uncle: Suppose someone asks: "What's the HR policy for leaves?"

Without metadata:

System searches all documents
Returns leaves policies from EVERY department

With metadata:

System knows to search only HR documents
Returns only HR leave policy

Much better!

// Step 6: Extract metadata

import logger from '../utils/logger';

interface DocumentMetadata {
  source: string;
  department?: string;
  section?: string;
  version?: string;
  effectiveDate?: string;
  author?: string;
}

/**
 * Extract metadata from document structure
 * 
 * Sources:
 * 1. Folder structure: /HR/Policies/Leave_Policy.pdf
 * 2. Filename: leave_policy_v2.pdf
 * 3. Parser output: Unstructured gives headings as metadata
 * 4. LLM classification: Ask AI to classify
 */
export function extractMetadataFromStructure(
  filePath: string,
  parserElements: any[]
): DocumentMetadata {
  // Source 1: Folder structure
  const pathParts = filePath.split('/');
  const department = pathParts[pathParts.length - 2] === 'HR' ? 'Human Resources' : undefined;

  // Source 2: Filename
  const fileName = pathParts[pathParts.length - 1];
  const versionMatch = fileName.match(/v(\d+)/i);
  const version = versionMatch ? versionMatch[1] : undefined;

  // Source 3: Parser output (if using Unstructured)
  let section = undefined;
  if (parserElements && parserElements.length > 0) {
    const heading = parserElements.find((e: any) => e.type === 'Title' || e.type === 'Heading');
    if (heading) {
      section = heading.text;
    }
  }

  const metadata: DocumentMetadata = {
    source: filePath,
    department,
    section,
    version
  };

  logger.info('Metadata extracted', metadata);

  return metadata;
}

/**
 * Optional: Use LLM to classify (more powerful but costs money)
 */
export async function classifyWithLLM(text: string) {
  const client = new Anthropic();

  const response = await client.messages.create({
    model: 'claude-3-5-sonnet-20241022',
    max_tokens: 200,
    messages: [{
      role: 'user',
      content: `Classify this document. Return JSON:
{
  "department": "HR|Finance|IT|Legal",
  "topic": "Leave|Salary|Security|Contracts",
  "confidentiality": "public|internal|confidential"
}

Document: ${text.substring(0, 500)}`
    }]
  });

  return JSON.parse(response.content[0].text);
}

PHASE 1, STEP 7-10: Chunking - The Heart of Preparation

Why Chunking Matters

👨‍🦳 Uncle: This is the most important step. Everything after depends on it.

👦 Nephew: More important than embeddings?

👨‍🦳 Uncle: YES. Because bad chunks → bad embeddings → wrong answers.

Step 7: Chunking Strategy

// Step 7: Smart chunking with boundaries

/**
 * Chunking strategies:
 * 
 * 1. Fixed size (simple, dumb)
 *    ├─ Pros: Easy to understand
 *    └─ Cons: Breaks meaning
 * 
 * 2. Sentence boundaries (better)
 *    ├─ Pros: Doesn't break sentences
 *    └─ Cons: Variable sizes
 * 
 * 3. Semantic (best, most complex)
 *    ├─ Pros: Keeps meaning together
 *    └─ Cons: Expensive, slow
 * 
 * 4. Recursive (good hybrid)
 *    ├─ Pros: Smart, flexible
 *    └─ Cons: Complex implementation
 */

function slidingWindowChunk(
  text: string,
  windowTokens: number = 1000,
  overlapTokens: number = 200
): string[] {
  // 1 token ≈ 4 characters (rough estimate for English)
  const charWindow = windowTokens * 4;
  const charOverlap = overlapTokens * 4;
  const step = charWindow - charOverlap;

  const chunks: string[] = [];

  for (let i = 0; i < text.length; i += step) {
    let end = i + charWindow;

    // Find sentence boundary (don't break mid-sentence)
    if (end < text.length) {
      const periodIndex = text.lastIndexOf('.', end);
      const newlineIndex = text.lastIndexOf('\n', end);
      const boundaryIndex = Math.max(periodIndex, newlineIndex);

      if (boundaryIndex > i + (charWindow * 0.8)) {
        end = boundaryIndex + 1;
      }
    }

    const chunk = text.substring(i, end).trim();
    if (chunk.length > 0) {
      chunks.push(chunk);
    }

    if (end >= text.length) break;
  }

  return chunks;
}

// Example
const text = `
Employees receive 24 paid leaves annually. 
These leaves can be taken as:
1. Annual Leave (20 days)
2. Casual Leave (4 days)

Manager approval is required for:
- Leaves longer than 5 days
- Leave during peak season
`;

const chunks = slidingWindowChunk(text, 1000, 200);
console.log(chunks);
// [
//   "Employees receive 24 paid leaves annually...",
//   "These leaves can be taken as: 1. Annual Leave..."
// ]

Step 8-10: Chunk Boundaries, Size, and Overlap

👦 Nephew: Why does size matter?

👨‍🦳 Uncle: Look at three scenarios:

Scenario 1: Too Small (100 tokens)

Chunk 1: "Employees receive"
Chunk 2: "24 paid leaves"
Chunk 3: "annually"

Result: Lost context. AI confused.

Question: "How many leaves do employees get?"
Chunk 1 alone: "Employees receive" (incomplete)
Chunk 2 alone: "24 paid leaves" (where?)
Chunk 3 alone: "annually" (what annually?)

Wrong answer!

─────────────────────────────────────

Scenario 2: Right Size (1000 tokens)

Chunk: "Employees receive 24 paid leaves annually.
         These leaves can be taken as annual or casual.
         Manager approval required for leaves >5 days."

Result: Full context. Clear meaning.

Question: "How many leaves do employees get?"
Answer: "24 paid leaves"

Correct!

─────────────────────────────────────

Scenario 3: Too Large (5000 tokens)

Chunk: [Entire company handbook]

Result: Too much noise.

Question: "How many leaves?"
System reads 5000 tokens to find answer.
Slow. Confused by other policies.

Wrong section highlighted.

👨‍🦳 Uncle: Sweet spot: 1000-1500 tokens.

// Complete chunking service

import logger from '../utils/logger';

interface Chunk {
  text: string;
  index: number;
  tokens: number;
  metadata: {
    startChar: number;
    endChar: number;
  };
}

/**
 * Steps 7-10: Complete chunking strategy
 * 
 * - Sliding window with overlap
 * - Respects sentence boundaries  
 * - Includes overlap for context
 * - Validates chunk size
 */
export function createSmartChunks(
  text: string,
  windowTokens: number = 1000,
  overlapTokens: number = 200
): Chunk[] {
  const charWindow = windowTokens * 4;
  const charOverlap = overlapTokens * 4;
  const step = charWindow - charOverlap;

  const chunks: Chunk[] = [];
  let chunkIndex = 0;

  for (let i = 0; i < text.length; i += step) {
    let end = i + charWindow;

    // Validate not empty
    if (end - i < charWindow * 0.3) {
      // Remaining text too small, merge with previous
      if (chunks.length > 0) {
        const lastChunk = chunks[chunks.length - 1];
        lastChunk.text += ' ' + text.substring(i).trim();
        break;
      }
    }

    // Find sentence boundary
    if (end < text.length) {
      const periodIndex = text.lastIndexOf('.', end);
      if (periodIndex > i) {
        end = periodIndex + 1;
      }
    }

    const chunkText = text.substring(i, end).trim();

    if (chunkText.length === 0) continue;

    // Estimate tokens (rough)
    const estimatedTokens = Math.ceil(chunkText.length / 4);

    // Validate: chunk should be reasonable size
    if (estimatedTokens < 100) {
      logger.warn('Chunk too small', { tokens: estimatedTokens });
    }

    chunks.push({
      text: chunkText,
      index: chunkIndex++,
      tokens: estimatedTokens,
      metadata: {
        startChar: i,
        endChar: end
      }
    });

    if (end >= text.length) break;
  }

  logger.info('Chunking complete', {
    textLength: text.length,
    chunkCount: chunks.length,
    avgChunkTokens: Math.round(
      chunks.reduce((sum, c) => sum + c.tokens, 0) / chunks.length
    )
  });

  return chunks;
}

PHASE 1, STEP 11-13: Deduplication, Hashing & Versioning

Step 11: Chunk Hashing

👨‍🦳 Uncle: After chunking, we hash each chunk.

👦 Nephew: Why hash chunks? We already hashed the file!

👨‍🦳 Uncle: Because files change. When they do, we need to know WHICH chunks changed.

HR_Policy_v1.pdf (Old)
├─ Chunk 1: "Employees get 24 leaves"
├─ Chunk 2: "Manager approval required"
└─ Chunk 3: "Casual leave guidelines"

HR_Policy_v2.pdf (New)
├─ Chunk 1: "Employees get 30 leaves"  ← CHANGED!
├─ Chunk 2: "Manager approval required" ← SAME
└─ Chunk 3: "Casual leave guidelines"   ← SAME

Old system: Re-embed all 3 chunks
Smart system: Re-embed only chunk 1

Cost saving: 66%!

// Step 11: Hash each chunk

import crypto from 'crypto';

export function hashChunk(chunkText: string): string {
  return crypto
    .createHash('sha256')
    .update(chunkText)
    .digest('hex');
}

/**
 * Key insight: Hash is deterministic
 * 
 * Same text:
 * hashChunk("Employees get 24 leaves")
 * Always returns: A7B82C1F...
 * 
 * Different text:
 * hashChunk("Employees get 30 leaves")
 * Always returns: X9Z47M3Q...
 * 
 * Different hash = Document changed
 */

// Example
const chunk1 = "Employees receive 24 paid leaves annually";
const chunk1Hash = hashChunk(chunk1);
// A7B82C1F9D3E4K2...

const chunk2 = "Employees receive 24 paid leaves annually";
const chunk2Hash = hashChunk(chunk2);
// A7B82C1F9D3E4K2... (SAME!)

const chunk3 = "Employees receive 30 paid leaves annually";
const chunk3Hash = hashChunk(chunk3);
// X9Z47M3Q2K1L9... (DIFFERENT!)

Step 12: Deduplication

// Step 12: Detect and skip duplicates

/**
 * Deduplication at multiple levels:
 * 
 * Level 1: Document level
 *          Have we processed this exact file?
 *          (Already done with file hash)
 * 
 * Level 2: Chunk level
 *          Have we processed this exact chunk?
 * 
 * Level 3: Semantic level
 *          Is this chunk similar to existing chunk?
 *          (Advanced, uses embeddings)
 */

export async function deduplicateChunks(
  chunks: Chunk[],
  tenantId: string,
  documentId: string
): Promise<{ newChunks: Chunk[]; skipped: number }> {
  const newChunks: Chunk[] = [];
  let skipped = 0;

  for (const chunk of chunks) {
    const chunkHash = hashChunk(chunk.text);

    // Check if chunk exists
    const existingChunk = await db.oneOrNone(
      `
      SELECT id FROM chunks 
      WHERE chunk_hash = $1 AND tenant_id = $2
      `,
      [chunkHash, tenantId]
    );

    if (existingChunk) {
      logger.debug('Duplicate chunk skipped', { chunkHash: chunkHash.substring(0, 8) });
      skipped++;
      continue;
    }

    newChunks.push(chunk);
  }

  logger.info('Deduplication complete', {
    total: chunks.length,
    new: newChunks.length,
    skipped,
    percentage: Math.round((skipped / chunks.length) * 100)
  });

  return { newChunks, skipped };
}

👦 Nephew: What if dedup changes because we change chunk size?

👨‍🦳 Uncle: Good question! That's versioning.

Step 13: Versioning

// Step 13: Handle document versions

export async function handleDocumentVersion(
  tenantId: string,
  fileName: string,
  newFileHash: string
) {
  // Check if this document (by name) was processed before
  const oldVersion = await db.one(
    `
    SELECT id, version FROM documents 
    WHERE file_name = $1 AND tenant_id = $2
    ORDER BY version DESC LIMIT 1
    `,
    [fileName, tenantId]
  );

  let version = 1;
  if (oldVersion) {
    // Old version exists
    // Mark it as inactive
    await db.none(
      `
      UPDATE documents 
      SET active = false 
      WHERE id = $1
      `,
      [oldVersion.id]
    );

    version = oldVersion.version + 1;
  }

  // Create new version
  const newDoc = await db.one(
    `
    INSERT INTO documents 
    (tenant_id, file_name, file_hash, version, active)
    VALUES ($1, $2, $3, $4, true)
    RETURNING id
    `,
    [tenantId, fileName, newFileHash, version]
  );

  logger.info('Document versioned', {
    fileName,
    version,
    active: true
  });

  return {
    documentId: newDoc.id,
    version,
    isNewVersion: oldVersion ? true : false
  };
}

/**
 * Database schema
 */
export const versioningSchema = `
CREATE TABLE documents (
  id UUID PRIMARY KEY,
  tenant_id UUID NOT NULL,
  file_name VARCHAR(500),
  file_hash VARCHAR(64),
  version INTEGER DEFAULT 1,
  active BOOLEAN DEFAULT true,
  created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,

  INDEX idx_active (tenant_id, active)
);
`;

👨‍🦳 Uncle: Important: During retrieval, we search ONLY active documents.

-- Search only from active documents
SELECT chunk_text
FROM chunks
WHERE tenant_id = $1
  AND document_id IN (
    SELECT id FROM documents 
    WHERE active = true
  )
ORDER BY embedding <=> query_embedding
LIMIT 5

PHASE 1, STEP 14-15: Incremental Ingestion & Cost Optimization

The Big Picture: Before vs After

👨‍🦳 Uncle: Let me show you the difference between naive and smart systems.

NAIVE SYSTEM
───────────────────────────────────────

File v1 uploaded
  ├─ Parse
  ├─ Chunk (1000 chunks)
  ├─ Embed ALL 1000  ← Cost: ₹50 per 1000 = ₹50
  └─ Store

File v2 uploaded (only 1 chunk changed)
  ├─ Parse
  ├─ Chunk (1000 chunks)
  ├─ Embed ALL 1000  ← Cost: ₹50 again!
  └─ Store

Total cost: ₹100
Total waste: ₹50 (50% wasted)

───────────────────────────────────────

SMART SYSTEM
───────────────────────────────────────

File v1 uploaded
  ├─ Parse
  ├─ Chunk (1000 chunks)
  ├─ Embed ALL 1000  ← Cost: ₹50
  └─ Store with chunk hashes

File v2 uploaded (only 1 chunk changed)
  ├─ Parse
  ├─ Chunk (1000 chunks)
  ├─ Compare hashes → 999 same, 1 changed
  ├─ Embed ONLY 1 chunk  ← Cost: ₹0.05 only!
  └─ Store (update index for chunk 1)

Total cost: ₹50.05
Total waste: ₹0 (99% efficient!)

Step 14: Incremental Ingestion

// Step 14: Incremental ingestion (smart re-embedding)

/**
 * The key insight:
 * 
 * Don't re-embed unchanged chunks.
 * Reuse old embeddings.
 * 
 * This is how production systems save money.
 */

export async function incrementalIngestion(
  tenantId: string,
  documentId: string,
  newChunks: Chunk[],
  documentVersion: number
) {
  // Get old chunks from previous version
  const oldDoc = await db.one(
    `
    SELECT id FROM documents 
    WHERE file_name = (SELECT file_name FROM documents WHERE id = $1)
      AND version = $2 - 1
      AND tenant_id = $3
    `,
    [documentId, documentVersion, tenantId]
  );

  let embeddingsNeeded = newChunks;
  let embeddingsReused = 0;

  if (oldDoc) {
    // Get old chunks
    const oldChunks = await db.manyOrNone(
      `
      SELECT id, chunk_hash, embedding FROM chunks 
      WHERE document_id = $1
      `,
      [oldDoc.id]
    );

    // Compare: which chunks are new/changed?
    embeddingsNeeded = newChunks.filter(newChunk => {
      const newHash = hashChunk(newChunk.text);
      const oldChunk = oldChunks.find(
        c => c.chunk_hash === newHash
      );

      if (oldChunk) {
        // Chunk unchanged: reuse embedding
        embeddingsReused++;

        // Copy old embedding to new chunk
        newChunk.embedding = oldChunk.embedding;

        return false; // Don't need to re-embed
      }

      return true; // Need to embed
    });
  }

  logger.info('Incremental ingestion', {
    totalChunks: newChunks.length,
    newChunks: embeddingsNeeded.length,
    reused: embeddingsReused,
    efficiency: Math.round((embeddingsReused / newChunks.length) * 100) + '%'
  });

  return {
    chunksToEmbed: embeddingsNeeded,
    chunksCached: embeddingsReused,
    costSaving: embeddingsReused * 0.00001  // ₹ saved
  };
}

Step 15: Cost Optimization Strategy

// Step 15: Track costs and optimize

export async function optimizeIngestionCosts() {
  // Strategy 1: Batch embeddings
  // Don't embed one by one. Batch them.
  // Cost: 1000 embeddings in 1 API call = cheaper than 1000 calls

  // Strategy 2: Use cheaper models for simple chunks
  // Use Gemini for obvious policy text
  // Use Claude only for complex analysis

  // Strategy 3: Cache embeddings aggressively
  // If same chunk text appears in document 1 and 2
  // Don't embed twice. Reuse.

  // Strategy 4: Incremental updates
  // Don't re-embed when document slightly changes
  // Only embed changed chunks

  /**
   * Real example: HR uploads 100 policies
   * 
   * Policy 1: 100 chunks
   * Policy 2: 100 chunks
   * Policy 3: 100 chunks (99 similar to Policy 1)
   * 
   * Naive: Embed 300 chunks = ₹15
   * Smart: Embed 200 chunks = ₹10 (saved ₹5)
   * 
   * At scale (1000 policies):
   * Naive: ₹500/month
   * Smart: ₹100/month (saved ₹400!)
   */

  const costBreakdown = {
    parsing: 0.50,        // ₹0.50 per document
    embedding: 50,        // ₹50 per 1000 chunks
    vectorStorage: 10,    // ₹10 per GB/month
    llmClassification: 1, // ₹1 per document (optional)
    monthlyInfrastructure: 1000
  };

  logger.info('Cost breakdown', costBreakdown);

  return costBreakdown;
}

Complete Production Ingestion Flow

👨‍🦳 Uncle: Let me show you the complete flow as code.

// Complete ingestion pipeline

import db from '../config/database';
import logger from '../utils/logger';

/**
 * Phase 1: Complete Document Ingestion Pipeline
 * 
 * This is what production systems do before embeddings.
 * 
 * Cost tracking included.
 */
export async function completeIngestionPipeline(
  filePath: string,
  tenantId: string
): Promise<{
  documentId: string;
  chunksCreated: number;
  chunksEmbedded: number;
  costEstimate: number;
  duration: number;
}> {
  const startTime = Date.now();
  const metrics = {
    documentId: '',
    chunksCreated: 0,
    chunksEmbedded: 0,
    costEstimate: 0,
    duration: 0
  };

  try {
    // STEP 1-2: Upload & File Hashing
    logger.info('Step 1-2: File upload and hashing...');
    const fileHash = await handleFileUpload(filePath, tenantId);

    if (fileHash.alreadyExists) {
      logger.warn('File already processed. Skipping.');
      return {
        ...metrics,
        duration: Date.now() - startTime
      };
    }

    // STEP 3: Choose parser
    logger.info('Step 3: Parsing PDF...');
    const parserType = selectParser(filePath); // pdf-parse, unstructured, etc.
    const parsedElements = await parsePDF(filePath, parserType);

    // STEP 4-5: Extract & Clean
    logger.info('Step 4-5: Extracting and cleaning text...');
    let cleanText = extractText(parsedElements);
    cleanText = cleanExtractedText(cleanText);

    // STEP 6: Metadata
    logger.info('Step 6: Extracting metadata...');
    const metadata = extractMetadataFromStructure(filePath, parsedElements);

    // STEP 7-10: Chunking
    logger.info('Step 7-10: Creating chunks...');
    const chunks = createSmartChunks(cleanText, 1000, 200);
    metrics.chunksCreated = chunks.length;

    // STEP 11-12: Deduplication
    logger.info('Step 11-12: Deduplication...');
    const { newChunks, skipped } = await deduplicateChunks(
      chunks,
      tenantId,
      fileHash.fileHash
    );

    // STEP 13: Versioning
    logger.info('Step 13: Handling versions...');
    const versionInfo = await handleDocumentVersion(
      tenantId,
      fileHash.fileName,
      fileHash.fileHash
    );
    metrics.documentId = versionInfo.documentId;

    // STEP 14-15: Incremental & Optimize
    logger.info('Step 14-15: Incremental ingestion...');
    const { chunksToEmbed } = await incrementalIngestion(
      tenantId,
      versionInfo.documentId,
      newChunks,
      versionInfo.version
    );
    metrics.chunksEmbedded = chunksToEmbed.length;

    // Calculate cost
    metrics.costEstimate = (chunksToEmbed.length / 1000) * 50; // ₹50 per 1000

    // Save chunks to database
    for (const chunk of newChunks) {
      const chunkHash = hashChunk(chunk.text);

      await db.none(
        `
        INSERT INTO chunks 
        (document_id, tenant_id, chunk_text, chunk_index, chunk_hash, metadata)
        VALUES ($1, $2, $3, $4, $5, $6)
        `,
        [
          versionInfo.documentId,
          tenantId,
          chunk.text,
          chunk.index,
          chunkHash,
          JSON.stringify(metadata)
        ]
      );
    }

    logger.info('✓ Phase 1 complete!', metrics);

    // Return ready for Phase 2: Embeddings
    return {
      ...metrics,
      duration: Date.now() - startTime
    };

  } catch (error: any) {
    logger.error('Ingestion pipeline failed', { error: error.message });
    throw error;
  }
}

/**
 * Helper: Choose the right parser
 */
function selectParser(filePath: string) {
  const extension = filePath.split('.').pop()?.toLowerCase();

  if (extension === 'pdf') {
    // Use document size to decide
    const size = fs.statSync(filePath).size;

    if (size < 1000000) {
      // < 1MB: simple PDF
      return 'pdf-parse';
    } else if (size < 10000000) {
      // 1-10MB: normal PDF
      return 'unstructured';
    } else {
      // > 10MB: complex PDF
      return 'llamaparse';
    }
  }

  if (extension === 'docx') {
    return 'docx-parse';
  }

  return 'pdf-parse'; // default
}

The Complete Journey: From Upload to Ready-for-Embedding

👨‍🦳 Uncle: Let me show you the entire flow visually.

USER UPLOADS PDF
       ↓
   [STEP 1-2]
   File Hash Check
       ↓
   Already exists? ──YES──→ STOP (save cost)
       │
       NO
       ↓
   [STEP 3]
   Parse PDF ──→ Choose parser
                 ├─ pdf-parse (free)
                 ├─ Unstructured (cheap)
                 └─ LlamaParse (expensive)
       ↓
   [STEP 4-5]
   Extract & Clean ──→ Remove garbage
                       (headers, footers, etc.)
       ↓
   [STEP 6]
   Extract Metadata ──→ Add context
                        (department, section, etc.)
       ↓
   [STEP 7-10]
   Smart Chunking ──→ 1000-1500 tokens
                      with overlap & boundaries
       ↓
   [STEP 11-12]
   Chunk Hashing ──→ Detect duplicates
       ↓
   Skip duplicates? ──YES──→ Don't embed
       │
       NO
       ↓
   [STEP 13]
   Versioning ──→ Mark old version inactive
                  Keep new version active
       ↓
   [STEP 14-15]
   Incremental ──→ Only embed changed chunks
   Ingestion       Reuse cached embeddings
       ↓
   ✓ READY FOR EMBEDDINGS

Interview-Level Summary

👦 Nephew: If someone asks me in an interview about RAG ingestion?

👨‍🦳 Uncle: Say this:

"Phase 1 Document Ingestion has 15 critical steps before embeddings:

File Upload & Hashing - Hash file content (not filename) to detect duplicates
PDF Parsing - Choose tool based on document type (pdf-parse for simple, Unstructured for normal, LlamaParse for complex)
Text Extraction - Get raw text from parser output
Text Cleaning - Remove headers, footers, watermarks, junk
Metadata Extraction - Extract department, section, version from structure or LLM
Chunking - Split into 1000-1500 tokens with smart boundaries (sentence/paragraph)
Chunk Size - Balance between context (large) and precision (small)
Chunk Boundaries - Don't break mid-sentence; respect document structure
Overlap - 200 tokens overlap to maintain context continuity
Chunk Hashing - Hash each chunk to detect changes
Deduplication - Skip duplicate chunks at semantic level
Versioning - Handle document updates (v1 inactive, v2 active)
Incremental Ingestion - Only re-embed changed chunks, reuse old embeddings
Cost Optimization - Track costs, batch requests, use cheaper models
Index Update - Store chunks and prepare for vector embedding

The key insight: Avoid embedding unnecessary content. At enterprise scale, avoiding 100K unnecessary embeddings saves thousands per month."

Practical Checklist: Before You Deploy

Pre-Deployment Questions

const preDeploymentChecklist = {
  // Parsing
  "Have you tested parsing with your actual documents?": false,
  "Do you handle tables, images, multi-column layouts?": false,
  "Did you choose the right parser for your documents?": false,

  // Deduplication
  "Are you hashing file contents, not filenames?": false,
  "Do you check for existing files before parsing?": false,
  "Do you check for duplicate chunks before embedding?": false,

  // Chunking
  "Is your chunk size 800-1500 tokens?": false,
  "Do chunks respect sentence/paragraph boundaries?": false,
  "Do you have 100-200 token overlap?": false,

  // Metadata
  "Do you extract metadata from document structure?": false,
  "Is metadata used during retrieval?": false,

  // Versioning
  "Do you handle document updates (v1, v2, v3)?": false,
  "Do old versions get marked inactive?": false,

  // Cost
  "Are you tracking embedding costs per document?": false,
  "Do you reuse embeddings for unchanged chunks?": false,
  "Are you batching embedding requests?": false,

  // Production
  "Do you have error handling and retries?": false,
  "Are you logging every step?": false,
  "Can you rollback a bad ingestion?": false
};

Key Takeaways

👨‍🦳 Uncle: Remember:

Before embeddings, 15 steps matter
- Skip even one, quality suffers
The right parser saves time
- pdf-parse: Free, simple
- Unstructured: Balanced
- LlamaParse: Best quality
Chunking is everything
- Size: 1000-1500 tokens
- Overlap: 200 tokens
- Boundaries: Sentence/paragraph
Deduplication saves money
- Hash file content, not filename
- Hash chunks, reuse embeddings
- 50%+ cost savings possible
Metadata enables better retrieval
- Extract from structure
- Use during search
- Improve accuracy
Production systems are incremental
- Don't re-embed everything
- Reuse cached embeddings
- Only process what changed

👦 Nephew: So this is what "production-grade RAG" really means?

👨‍🦳 Uncle: Exactly. It's not just about embeddings and vector search. It's about doing the hard work BEFORE you search. That's what separates hobby projects from systems that work at scale.

Next Steps

You now understand Phase 1: Document Ingestion.

Next comes Phase 2: Retrieval & Ranking (vector search, reranking, query expansion)

Then Phase 3: Safety & Production (hallucination prevention, monitoring, cost tracking)

Ready to learn Phase 2?

👦 Nephew: Uncle, definitely! But first... can you show me a working example?

👨‍🦳 Uncle: That's next. We'll build a real ingestion pipeline together.

Reference Code Repository

# Complete implementation files:

1. fileHasher.ts        - File hashing, duplicate detection
2. pdfParsers.ts        - pdf-parse, Unstructured, LlamaParse
3. textCleaning.ts      - Remove junk, normalize whitespace
4. metadataExtractor.ts - Extract from structure, LLM classify
5. chunking.ts          - Sliding window, smart boundaries
6. deduplication.ts     - Chunk hashing, duplicate detection
7. versioning.ts        - Handle updates, version tracking
8. incrementalIngestion.ts - Reuse embeddings, smart updates
9. ingestionPipeline.ts - Complete flow orchestration
10. costOptimizer.ts    - Track costs, identify savings

Total: ~2000 lines of production-ready Node.js code

Remember: Less noise, more action. Phase 1 is the foundation.
Build it right.
written by -SurajK

RAG Pipeline: Complete Node.js Implementation Guide

surajrkhonde — Sat, 20 Jun 2026 10:27:48 +0000

Build production RAG systems in Node.js - Know where it breaks, why it works, and when to use it

Introduction: Why Node.js for RAG?

👦 Nephew: Uncle, why would I build RAG in Node.js? I thought this was AI stuff?

👨‍🦳 Uncle: Good question. Node.js is perfect for RAG because:

Fast I/O (API calls to Claude, PostgreSQL queries)
Real-time capable (WebSockets for streaming)
Easy deployment (Vercel, Railway, Render)
Async/await makes complex flows clean

Plus, you probably already have Node.js running your backend. Why add Python?

👦 Nephew: So I can build the whole thing in JavaScript?

👨‍🦳 Uncle: Yes. Frontend, backend, RAG - all JavaScript. That's the beauty.

But we need to be honest about limitations. Let's talk about that too.

PART 1: PROJECT SETUP

Initialize Your Project

# Create project
mkdir rag-system
cd rag-system

# Initialize Node.js
npm init -y

# Install dependencies
npm install express dotenv @anthropic-ai/sdk pg pg-promise cors body-parser
npm install --save-dev nodemon typescript @types/node

# Optional but recommended for production
npm install winston helmet compression

Project Structure

rag-system/
├── src/
│   ├── config/
│   │   ├── database.ts         # PostgreSQL + pgvector setup
│   │   └── embedding.ts         # Claude embeddings
│   ├── services/
│   │   ├── retrieval.ts         # Vector search logic
│   │   ├── reranking.ts         # Two-stage ranking
│   │   ├── queryProcessing.ts   # Query expansion
│   │   └── safety.ts            # Hallucination prevention
│   ├── routes/
│   │   └── rag.ts               # API endpoints
│   ├── utils/
│   │   ├── logger.ts            # Logging (critical for debugging)
│   │   └── metrics.ts           # Track recall, precision
│   └── index.ts                 # Main server
├── .env                          # Secrets
├── package.json
└── tsconfig.json

TypeScript Configuration

{
  "compilerOptions": {
    "target": "ES2020",
    "module": "commonjs",
    "lib": ["ES2020"],
    "outDir": "./dist",
    "rootDir": "./src",
    "strict": true,
    "esModuleInterop": true,
    "skipLibCheck": true,
    "forceConsistentCasingInFileNames": true
  }
}

PART 2: DATABASE SETUP (PostgreSQL + pgvector)

1. Create PostgreSQL Database with pgvector

👨‍🦳 Uncle: This is your foundation. Get it wrong, everything breaks.

-- Connect to PostgreSQL
psql -U postgres

-- Create database
CREATE DATABASE rag_system;

-- Connect to the database
\c rag_system

-- Install pgvector extension
CREATE EXTENSION IF NOT EXISTS vector;

-- Create resumes table
CREATE TABLE resumes (
  id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
  tenant_id UUID NOT NULL,
  candidate_name VARCHAR(255) NOT NULL,
  raw_text TEXT NOT NULL,
  created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,

  -- CRITICAL: tenant isolation
  CONSTRAINT tenant_isolation UNIQUE(tenant_id, id)
);

-- Create chunks table (where vectors live)
CREATE TABLE resume_chunks (
  id SERIAL PRIMARY KEY,
  resume_id UUID NOT NULL REFERENCES resumes(id) ON DELETE CASCADE,
  tenant_id UUID NOT NULL,
  chunk_text TEXT NOT NULL,
  chunk_index INTEGER NOT NULL,

  -- The vector: 1536 dimensions for Claude embeddings
  embedding vector(1536) NOT NULL,

  -- Metadata for debugging
  created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
  updated_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,

  -- CRITICAL: Always check tenant
  CONSTRAINT tenant_isolation_chunks 
    FOREIGN KEY (tenant_id) REFERENCES tenants(id)
);

-- Create indexes
-- 1. Vector index for fast search (MOST IMPORTANT)
CREATE INDEX idx_resume_chunks_embedding 
ON resume_chunks USING ivfflat (embedding vector_cosine_ops)
WITH (lists = 100);

-- 2. Tenant index (security)
CREATE INDEX idx_resume_chunks_tenant 
ON resume_chunks(tenant_id, resume_id);

-- 3. Text search index (keyword search)
CREATE INDEX idx_resume_chunks_text 
ON resume_chunks USING GIN (to_tsvector('english', chunk_text));

-- Create tenants table (multi-tenancy)
CREATE TABLE tenants (
  id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
  name VARCHAR(255) NOT NULL,
  api_key VARCHAR(255) UNIQUE NOT NULL,
  created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
);

-- Create query logs (for metrics)
CREATE TABLE query_logs (
  id SERIAL PRIMARY KEY,
  tenant_id UUID NOT NULL REFERENCES tenants(id),
  query TEXT NOT NULL,
  latency_ms INTEGER NOT NULL,
  recall DECIMAL(3,2),
  precision DECIMAL(3,2),
  cost_cents INTEGER,
  created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
);

-- Create index on query logs for analytics
CREATE INDEX idx_query_logs_tenant 
ON query_logs(tenant_id, created_at DESC);

2. Database Connection Service

👨‍🦳 Uncle: This is where your first failure point lives.

// src/config/database.ts

import pgPromise from 'pg-promise';
import dotenv from 'dotenv';
import logger from '../utils/logger';

dotenv.config();

const initOptions = {
  // Detailed error info (critical for debugging)
  error(error: any, context: any) {
    logger.error('Database Error', {
      error: error.message,
      query: context.query,
      params: context.params
    });
  },

  // Connection events
  connect(client: any) {
    logger.info('Database connected');
  },

  disconnect(client: any) {
    logger.info('Database disconnected');
  }
};

const pgp = pgPromise(initOptions);

const db = pgp({
  host: process.env.DB_HOST || 'localhost',
  port: parseInt(process.env.DB_PORT || '5432'),
  database: process.env.DB_NAME || 'rag_system',
  user: process.env.DB_USER || 'postgres',
  password: process.env.DB_PASSWORD,

  // Connection pooling
  max: 20,

  // Timeout after 5 seconds
  connectionTimeoutMillis: 5000,

  // Idle timeout
  idleTimeoutMillis: 30000,
});

// Test connection on startup
export async function initializeDatabase() {
  try {
    await db.one('SELECT 1');
    logger.info('✓ Database connection verified');
  } catch (error) {
    logger.error('✗ Database connection failed', { error });
    process.exit(1);
  }
}

export default db;

3. Environment Variables

# .env
DB_HOST=localhost
DB_PORT=5432
DB_NAME=rag_system
DB_USER=postgres
DB_PASSWORD=your_secure_password

ANTHROPIC_API_KEY=sk-ant-...
ANTHROPIC_MODEL=claude-3-5-sonnet-20241022

NODE_ENV=production
PORT=3000

# Security
ADMIN_API_KEY=super-secret-key-change-this

PART 3: EMBEDDING SERVICE

Create Embeddings with Claude

👨‍🦳 Uncle: This is where the first real cost happens. Know what can fail here.

// src/config/embedding.ts

import Anthropic from '@anthropic-ai/sdk';
import logger from '../utils/logger';

const client = new Anthropic({
  apiKey: process.env.ANTHROPIC_API_KEY
});

interface EmbeddingResult {
  text: string;
  embedding: number[];
}

/**
 * Get embeddings for text chunks.
 * 
 * ⚠️ FAILURE POINTS:
 * 1. API rate limit (429) - implements exponential backoff
 * 2. Token too long (4096 tokens max) - chunks pre-validated
 * 3. Network timeout - retry logic built in
 * 4. Cost tracking - logs cost per embedding
 */
export async function getEmbeddings(texts: string[]): Promise<EmbeddingResult[]> {
  const startTime = Date.now();

  try {
    // VALIDATION: Prevent token overrun
    // Claude's text embeddings: ~1 token = 4 chars average
    const validTexts = texts.map(text => {
      if (text.length > 16000) {  // ~4000 tokens
        logger.warn('Text truncated for embedding', { 
          originalLength: text.length,
          truncatedTo: 16000
        });
        return text.substring(0, 16000);
      }
      return text;
    });

    // Call Claude API for embeddings
    const response = await client.messages.create({
      model: 'claude-3-5-sonnet-20241022',
      max_tokens: 1024,
      messages: [{
        role: 'user',
        content: `Generate embeddings for the following texts. Return ONLY valid JSON array with "embeddings" key containing array of number arrays.

Texts:
${validTexts.map((t, i) => `${i}: ${t}`).join('\n\n')}

Return format: {"embeddings": [[...], [...], ...]}`
      }]
    });

    // Parse response
    const responseText = response.content[0].type === 'text' 
      ? response.content[0].text 
      : '';

    let embeddings: number[][];
    try {
      const parsed = JSON.parse(responseText);
      embeddings = parsed.embeddings || [];
    } catch (parseError) {
      logger.error('Failed to parse embeddings response', { 
        response: responseText.substring(0, 500) 
      });
      throw new Error('Invalid embeddings response format');
    }

    // Validate embeddings
    if (embeddings.length !== validTexts.length) {
      throw new Error(
        `Embedding count mismatch: got ${embeddings.length}, expected ${validTexts.length}`
      );
    }

    // Calculate cost (Claude 3.5 Sonnet: $0.003 per 1M input tokens)
    const inputTokens = response.usage.input_tokens;
    const costCents = (inputTokens / 1_000_000) * 0.003 * 100;

    const latency = Date.now() - startTime;
    logger.info('Embeddings generated', { 
      count: embeddings.length,
      latency,
      inputTokens,
      costCents: costCents.toFixed(4)
    });

    return validTexts.map((text, i) => ({
      text,
      embedding: embeddings[i]
    }));

  } catch (error: any) {
    logger.error('Embedding API error', {
      error: error.message,
      status: error.status
    });

    // Retry logic for rate limits
    if (error.status === 429) {
      logger.warn('Rate limited. Waiting before retry...');
      await new Promise(resolve => setTimeout(resolve, 5000));
      return getEmbeddings(texts); // Exponential backoff in real system
    }

    throw error;
  }
}

/**
 * Embed a single text (convenience function)
 */
export async function embedText(text: string): Promise<number[]> {
  const results = await getEmbeddings([text]);
  return results[0].embedding;
}

Chunking Function

👨‍🦳 Uncle: Remember: 1000-1500 tokens, 200-token overlap.

// src/utils/chunking.ts

import logger from './logger';

interface Chunk {
  text: string;
  index: number;
  tokens: number;
}

/**
 * Break text into chunks with sliding window overlap.
 * 
 * ⚠️ FAILURE POINTS:
 * 1. Overlap larger than chunk size
 * 2. Single chunk can't hold meaningful text
 */
export function chunkText(
  text: string,
  windowTokens: number = 1000,
  overlapTokens: number = 200
): Chunk[] {
  // Simple tokenization (1 token ≈ 4 chars for English)
  const estimatedTokens = Math.ceil(text.length / 4);

  if (estimatedTokens < windowTokens) {
    // Text is smaller than chunk size
    logger.debug('Text smaller than chunk window', { 
      estimatedTokens,
      windowTokens 
    });
    return [{
      text,
      index: 0,
      tokens: estimatedTokens
    }];
  }

  // Calculate character window (1 token ≈ 4 chars)
  const charWindow = windowTokens * 4;
  const charOverlap = overlapTokens * 4;
  const step = charWindow - charOverlap;

  const chunks: Chunk[] = [];
  let index = 0;

  for (let i = 0; i < text.length; i += step) {
    let end = i + charWindow;

    // Find sentence boundary to avoid splitting mid-sentence
    if (end < text.length) {
      const periodIndex = text.lastIndexOf('.', end);
      const newlineIndex = text.lastIndexOf('\n', end);
      const boundaryIndex = Math.max(periodIndex, newlineIndex);

      if (boundaryIndex > i + (charWindow * 0.8)) {
        // Found good boundary
        end = boundaryIndex + 1;
      }
    } else {
      end = text.length;
    }

    const chunk = text.substring(i, end).trim();

    if (chunk.length > 0) {
      chunks.push({
        text: chunk,
        index,
        tokens: Math.ceil(chunk.length / 4)
      });
      index++;
    }

    // Stop if we've reached the end
    if (end >= text.length) break;
  }

  logger.debug('Text chunked', {
    originalLength: text.length,
    chunkCount: chunks.length,
    avgChunkTokens: Math.round(
      chunks.reduce((sum, c) => sum + c.tokens, 0) / chunks.length
    )
  });

  return chunks;
}

PART 4: RETRIEVAL SERVICE

Vector Search with Hybrid (Vector + Keyword)

👨‍🦳 Uncle: This is the heart. Where everything lives or dies.

// src/services/retrieval.ts

import db from '../config/database';
import { embedText } from '../config/embedding';
import logger from '../utils/logger';

interface RetrievalResult {
  chunkText: string;
  chunkIndex: number;
  vectorDistance: number;
  keywordScore: number;
  combinedScore: number;
}

/**
 * Retrieve relevant chunks using hybrid search.
 * 
 * ⚠️ FAILURE POINTS:
 * 1. Missing tenant_id check → DATA BREACH
 * 2. Vector index not built → Slow queries (10s+ instead of 100ms)
 * 3. Query too long → API error
 * 4. No results → Need to handle gracefully
 * 5. Typos in query → Keyword search might fail
 */
export async function hybridSearch(
  tenantId: string,
  resumeId: string,
  query: string,
  topK: number = 5
): Promise<RetrievalResult[]> {
  const startTime = Date.now();

  try {
    // Validate inputs
    if (!tenantId || !resumeId) {
      throw new Error('tenant_id and resume_id are required');
    }

    if (query.length === 0) {
      throw new Error('Query cannot be empty');
    }

    if (query.length > 500) {
      logger.warn('Query truncated', { originalLength: query.length });
      query = query.substring(0, 500);
    }

    // Step 1: Get query embedding
    logger.debug('Embedding query', { query });
    const queryEmbedding = await embedText(query);

    // Step 2: Vector search (fast)
    // Convert embedding to PostgreSQL format: [0.1, 0.2, ...]
    const embeddingString = `[${queryEmbedding.join(',')}]`;

    const vectorResults = await db.manyOrNone(`
      SELECT 
        chunk_text,
        chunk_index,
        embedding <=> $1::vector AS vector_distance
      FROM resume_chunks
      WHERE 
        tenant_id = $2
        AND resume_id = $3
      ORDER BY vector_distance ASC
      LIMIT $4
    `, [embeddingString, tenantId, resumeId, topK * 2]); // Get 2x to filter

    if (vectorResults.length === 0) {
      logger.warn('No vector results found', { query, resumeId });
      return [];
    }

    // Step 3: Keyword filter (precision)
    // Only keep chunks that also match the query
    const keywordResults = await db.manyOrNone(`
      SELECT 
        chunk_text,
        chunk_index,
        ts_rank(
          to_tsvector('english', chunk_text), 
          plainto_tsquery('english', $1)
        ) AS keyword_score
      FROM resume_chunks
      WHERE 
        tenant_id = $2
        AND resume_id = $3
        AND to_tsvector('english', chunk_text) @@ 
            plainto_tsquery('english', $1)
      ORDER BY keyword_score DESC
      LIMIT $4
    `, [query, tenantId, resumeId, topK]);

    // Step 4: Combine results
    // Chunks that appear in both vector AND keyword search are best
    const combined = vectorResults
      .map(vr => {
        const kr = keywordResults.find(k => k.chunk_text === vr.chunk_text);
        return {
          ...vr,
          keywordScore: kr ? kr.keyword_score : 0,
          // Weighted score: 60% vector, 40% keyword
          combinedScore: (1 - vr.vector_distance) * 0.6 + (kr?.keyword_score || 0) * 0.4
        };
      })
      .sort((a, b) => b.combinedScore - a.combinedScore)
      .slice(0, topK);

    const latency = Date.now() - startTime;

    logger.info('Hybrid search complete', {
      query,
      resultsCount: combined.length,
      latency,
      vectorResultsCount: vectorResults.length,
      keywordResultsCount: keywordResults.length
    });

    // Log for metrics
    if (combined.length > 0) {
      await db.none(`
        INSERT INTO query_logs (tenant_id, query, latency_ms)
        VALUES ($1, $2, $3)
      `, [tenantId, query.substring(0, 255), latency]);
    }

    return combined as RetrievalResult[];

  } catch (error: any) {
    logger.error('Retrieval error', {
      error: error.message,
      query,
      resumeId,
      tenantId
    });
    throw error;
  }
}

/**
 * Multi-query retrieval - search with multiple variations.
 * 
 * Better recall, but slower and more expensive.
 */
export async function multiQueryRetrieval(
  tenantId: string,
  resumeId: string,
  queries: string[],
  topK: number = 5
): Promise<RetrievalResult[]> {
  try {
    const allResults: RetrievalResult[] = [];

    for (const query of queries) {
      const results = await hybridSearch(tenantId, resumeId, query, topK * 2);
      allResults.push(...results);
    }

    // Deduplicate by chunk text, keep highest score
    const unique = Array.from(
      allResults
        .reduce((map, item) => {
          const existing = map.get(item.chunkText);
          if (!existing || item.combinedScore > existing.combinedScore) {
            map.set(item.chunkText, item);
          }
          return map;
        }, new Map<string, RetrievalResult>())
        .values()
    );

    return unique
      .sort((a, b) => b.combinedScore - a.combinedScore)
      .slice(0, topK);

  } catch (error) {
    logger.error('Multi-query retrieval error', { error });
    throw error;
  }
}

PART 5: RERANKING SERVICE

👨‍🦳 Uncle: Two-stage is where quality happens. First stage is fast, second is accurate.

// src/services/reranking.ts

import Anthropic from '@anthropic-ai/sdk';
import logger from '../utils/logger';

interface RerankedResult {
  text: string;
  score: number;
  rank: number;
}

/**
 * Rerank chunks using Claude (more accurate but slower).
 * 
 * ⚠️ FAILURE POINTS:
 * 1. Claude API timeout (fix with timeout wrapper)
 * 2. Chunks too long (truncate before sending)
 * 3. Response parsing fails
 * 4. Cost explosion (reranking costs money - track it)
 */
export async function rerank(
  query: string,
  chunks: string[],
  topK: number = 5
): Promise<RerankedResult[]> {
  const startTime = Date.now();

  try {
    if (chunks.length === 0) {
      return [];
    }

    const client = new Anthropic({
      apiKey: process.env.ANTHROPIC_API_KEY,
      timeout: 30 * 1000, // 30 second timeout
    });

    // Truncate chunks to prevent token overflow
    const truncatedChunks = chunks.map(c => 
      c.length > 2000 ? c.substring(0, 2000) + '...' : c
    );

    // Build reranking prompt
    const chunksFormatted = truncatedChunks
      .map((chunk, i) => `[${i}] ${chunk}`)
      .join('\n\n---\n\n');

    const prompt = `You are a search relevance expert. Rank the following chunks by relevance to the query.

Query: "${query}"

Chunks to rank:
${chunksFormatted}

Return ONLY valid JSON with this format:
{
  "rankings": [
    {"index": 0, "relevance_score": 0.95},
    {"index": 1, "relevance_score": 0.72}
  ]
}

Relevance score: 0.0 (irrelevant) to 1.0 (highly relevant)
Sort by relevance_score descending.`;

    const response = await client.messages.create({
      model: 'claude-3-5-sonnet-20241022',
      max_tokens: 1024,
      messages: [{
        role: 'user',
        content: prompt
      }]
    });

    // Parse response
    const responseText = response.content[0].type === 'text'
      ? response.content[0].text
      : '';

    let rankings: any[];
    try {
      // Extract JSON from response (might be wrapped in markdown)
      const jsonMatch = responseText.match(/\{[\s\S]*\}/);
      const jsonStr = jsonMatch ? jsonMatch[0] : responseText;
      const parsed = JSON.parse(jsonStr);
      rankings = parsed.rankings || [];
    } catch (parseError) {
      logger.error('Failed to parse reranking response', {
        response: responseText.substring(0, 500)
      });
      // Fallback: return original order
      return chunks.slice(0, topK).map((text, i) => ({
        text,
        score: 1.0 - (i * 0.1),
        rank: i + 1
      }));
    }

    // Convert to results
    const results = rankings
      .filter(r => r.index >= 0 && r.index < chunks.length)
      .map((r, rank) => ({
        text: chunks[r.index],
        score: r.relevance_score,
        rank: rank + 1
      }))
      .slice(0, topK);

    const latency = Date.now() - startTime;

    logger.info('Reranking complete', {
      query,
      inputCount: chunks.length,
      outputCount: results.length,
      latency,
      topScore: results[0]?.score
    });

    return results;

  } catch (error: any) {
    logger.error('Reranking error', {
      error: error.message,
      chunksCount: chunks.length,
      query: query.substring(0, 100)
    });

    // Fallback: return original order
    return chunks.slice(0, topK).map((text, i) => ({
      text,
      score: 1.0 - (i * 0.1),
      rank: i + 1
    }));
  }
}

PART 6: QUERY PROCESSING

👨‍🦳 Uncle: Expand the query so you find more relevant chunks.

// src/services/queryProcessing.ts

import Anthropic from '@anthropic-ai/sdk';
import logger from '../utils/logger';

/**
 * Expand a query into related search terms.
 * 
 * ⚠️ FAILURE POINTS:
 * 1. LLM generates irrelevant expansions
 * 2. Original query lost in expansion
 * 3. Too many expansions → slow retrieval
 */
export async function expandQuery(originalQuery: string): Promise<string[]> {
  try {
    const client = new Anthropic({
      apiKey: process.env.ANTHROPIC_API_KEY
    });

    const response = await client.messages.create({
      model: 'claude-3-5-sonnet-20241022',
      max_tokens: 200,
      messages: [{
        role: 'user',
        content: `Given this query about a job candidate, generate 2-3 alternative phrasings or related concepts that would help find relevant information.

Original query: "${originalQuery}"

Return ONLY a JSON array of strings:
["alternative1", "alternative2", "alternative3"]

These should help find the same information using different keywords.`
      }]
    });

    const responseText = response.content[0].type === 'text'
      ? response.content[0].text
      : '';

    let expansions: string[];
    try {
      expansions = JSON.parse(responseText);
    } catch (e) {
      logger.warn('Failed to parse query expansion', { response: responseText });
      return [originalQuery];
    }

    // Always include original query
    const allQueries = [originalQuery, ...expansions].filter(Boolean);

    logger.debug('Query expanded', {
      original: originalQuery,
      expansions: allQueries.length
    });

    return allQueries;

  } catch (error) {
    logger.error('Query expansion error', { error });
    return [originalQuery]; // Fallback
  }
}

/**
 * Normalize query (remove typos, standardize terms).
 */
export function normalizeQuery(query: string): string {
  return query
    .toLowerCase()
    .trim()
    // Remove extra spaces
    .replace(/\s+/g, ' ')
    // Remove special characters (keep alphanumeric and spaces)
    .replace(/[^\w\s]/g, '');
}

PART 7: SAFETY & HALLUCINATION PREVENTION

👨‍🦳 Uncle: This is where you prevent the AI from lying. Critical.

// src/services/safety.ts

import Anthropic from '@anthropic-ai/sdk';
import logger from '../utils/logger';

interface SafeAnswer {
  answer: string;
  confidence: number;
  evidence: string[];
  isSafe: boolean;
  reason?: string;
}

/**
 * Get answer from AI with multiple safety layers.
 * 
 * ⚠️ FAILURE POINTS:
 * 1. AI answer not in JSON format
 * 2. Confidence calculation wrong
 * 3. Evidence doesn't exist in chunks
 * 4. Excessive cost for failed attempts
 */
export async function safeAnswer(
  query: string,
  chunks: string[],
  confidenceThreshold: number = 0.7
): Promise<SafeAnswer> {
  const startTime = Date.now();

  try {
    if (chunks.length === 0) {
      return {
        answer: 'No relevant information found.',
        confidence: 0,
        evidence: [],
        isSafe: false,
        reason: 'No source chunks provided'
      };
    }

    const client = new Anthropic({
      apiKey: process.env.ANTHROPIC_API_KEY
    });

    // Layer 1: Retrieval boundaries
    // Show ONLY the chunks, nothing from training
    const chunksText = chunks
      .map((c, i) => `[Chunk ${i}]\n${c}`)
      .join('\n\n---\n\n');

    const prompt = `You are evaluating a candidate resume based on specific chunks.

INSTRUCTIONS:
1. Answer ONLY based on the provided chunks
2. Do NOT use any knowledge from training data
3. If information is not in chunks, say "Unknown"
4. Always cite which chunk supports your answer
5. Return valid JSON ONLY - no other text

Query: "${query}"

Chunks provided:
${chunksText}

Return JSON in this exact format:
{
  "answer": "your answer here",
  "confidence": 0.0 to 1.0,
  "evidence_chunks": [0, 1, 2],
  "explanation": "why you're confident"
}`;

    // Layer 2: Structured output
    const response = await client.messages.create({
      model: 'claude-3-5-sonnet-20241022',
      max_tokens: 500,
      messages: [{
        role: 'user',
        content: prompt
      }]
    });

    const responseText = response.content[0].type === 'text'
      ? response.content[0].text
      : '';

    // Parse response
    let parsed: any;
    try {
      const jsonMatch = responseText.match(/\{[\s\S]*\}/);
      const jsonStr = jsonMatch ? jsonMatch[0] : responseText;
      parsed = JSON.parse(jsonStr);
    } catch (e) {
      logger.error('Failed to parse safety response', {
        response: responseText.substring(0, 300)
      });
      return {
        answer: 'Error processing answer',
        confidence: 0,
        evidence: [],
        isSafe: false,
        reason: 'Invalid response format'
      };
    }

    // Layer 3: Validation
    // Check evidence chunks actually exist
    const validEvidenceIndices = (parsed.evidence_chunks || [])
      .filter((i: number) => i >= 0 && i < chunks.length);

    if (validEvidenceIndices.length === 0 && parsed.answer !== 'Unknown') {
      logger.warn('No valid evidence for answer', { 
        answer: parsed.answer,
        requestedIndices: parsed.evidence_chunks,
        chunksCount: chunks.length
      });
    }

    const evidence = validEvidenceIndices.map((i: number) => chunks[i]);

    // Layer 4: Confidence gating
    const isSafe = parsed.confidence >= confidenceThreshold;

    if (!isSafe) {
      logger.warn('Low confidence answer', {
        answer: parsed.answer,
        confidence: parsed.confidence,
        threshold: confidenceThreshold
      });
    }

    const latency = Date.now() - startTime;

    logger.info('Safe answer generated', {
      query: query.substring(0, 50),
      confidence: parsed.confidence,
      isSafe,
      latency,
      evidenceCount: evidence.length
    });

    return {
      answer: parsed.answer,
      confidence: parsed.confidence,
      evidence,
      isSafe,
      reason: isSafe ? 'Confident' : 'Low confidence'
    };

  } catch (error: any) {
    logger.error('Safety check error', { error: error.message });
    return {
      answer: 'Error',
      confidence: 0,
      evidence: [],
      isSafe: false,
      reason: error.message
    };
  }
}

/**
 * Validate that answer is faithful to evidence.
 * Post-check: does answer match the chunks?
 */
export async function validateFaithfulness(
  answer: string,
  evidence: string[],
  threshold: number = 0.8
): Promise<{ isFaithful: boolean; score: number }> {
  try {
    // Simple check: are key terms from answer in evidence?
    const answerTerms = answer.toLowerCase().split(/\s+/);
    const evidenceText = evidence.join(' ').toLowerCase();

    const matchedTerms = answerTerms.filter(term => 
      term.length > 3 && evidenceText.includes(term)
    );

    const score = answerTerms.length > 0 
      ? matchedTerms.length / answerTerms.length 
      : 0;

    return {
      isFaithful: score >= threshold,
      score
    };

  } catch (error) {
    logger.error('Faithfulness validation error', { error });
    return { isFaithful: false, score: 0 };
  }
}

PART 8: API ROUTES & ENDPOINTS

👨‍🦳 Uncle: This is what the client calls. Make it robust.

// src/routes/rag.ts

import express, { Router, Request, Response } from 'express';
import db from '../config/database';
import { hybridSearch, multiQueryRetrieval } from '../services/retrieval';
import { rerank } from '../services/reranking';
import { expandQuery } from '../services/queryProcessing';
import { safeAnswer, validateFaithfulness } from '../services/safety';
import { chunkText } from '../utils/chunking';
import { getEmbeddings } from '../config/embedding';
import logger from '../utils/logger';

const router = Router();

// Middleware: Check authentication
function authMiddleware(req: Request, res: Response, next: Function) {
  const apiKey = req.headers['x-api-key'] as string;

  if (!apiKey) {
    return res.status(401).json({ error: 'Missing API key' });
  }

  // In production, validate against database
  if (apiKey !== process.env.ADMIN_API_KEY) {
    return res.status(401).json({ error: 'Invalid API key' });
  }

  next();
}

router.use(authMiddleware);

/**
 * Upload and process a resume.
 * POST /rag/upload
 */
router.post('/upload', async (req: Request, res: Response) => {
  try {
    const { tenantId, candidateName, resumeText } = req.body;

    if (!tenantId || !candidateName || !resumeText) {
      return res.status(400).json({ 
        error: 'Missing required fields: tenantId, candidateName, resumeText' 
      });
    }

    // Step 1: Save resume
    const resumeResult = await db.one(`
      INSERT INTO resumes (tenant_id, candidate_name, raw_text)
      VALUES ($1, $2, $3)
      RETURNING id
    `, [tenantId, candidateName, resumeText]);

    const resumeId = resumeResult.id;

    // Step 2: Chunk the resume
    const chunks = chunkText(resumeText, 1000, 200);
    logger.info('Resume chunked', { resumeId, chunkCount: chunks.length });

    // Step 3: Get embeddings for all chunks
    const chunkTexts = chunks.map(c => c.text);
    const embeddingResults = await getEmbeddings(chunkTexts);

    // Step 4: Save chunks with embeddings
    for (let i = 0; i < chunks.length; i++) {
      const chunk = chunks[i];
      const embedding = embeddingResults[i].embedding;
      const embeddingArray = `[${embedding.join(',')}]`;

      await db.none(`
        INSERT INTO resume_chunks 
        (resume_id, tenant_id, chunk_text, chunk_index, embedding)
        VALUES ($1, $2, $3, $4, $5::vector)
      `, [resumeId, tenantId, chunk.text, chunk.index, embeddingArray]);
    }

    logger.info('Resume uploaded successfully', { resumeId, chunkCount: chunks.length });

    res.json({
      success: true,
      resumeId,
      chunkCount: chunks.length,
      message: `Resume for ${candidateName} processed successfully`
    });

  } catch (error: any) {
    logger.error('Upload error', { error: error.message });
    res.status(500).json({ error: error.message });
  }
});

/**
 * Query a resume.
 * POST /rag/query
 */
router.post('/query', async (req: Request, res: Response) => {
  try {
    const { tenantId, resumeId, question, useExpansion = false } = req.body;

    if (!tenantId || !resumeId || !question) {
      return res.status(400).json({
        error: 'Missing required fields: tenantId, resumeId, question'
      });
    }

    const startTime = Date.now();

    // Step 1: Expand query if requested
    let queries = [question];
    if (useExpansion) {
      queries = await expandQuery(question);
      logger.debug('Query expanded', { count: queries.length });
    }

    // Step 2: Retrieve chunks (multi-query if expanded)
    const retrieved = useExpansion
      ? await multiQueryRetrieval(tenantId, resumeId, queries, 10)
      : await hybridSearch(tenantId, resumeId, question, 10);

    if (retrieved.length === 0) {
      return res.json({
        answer: 'No relevant information found in resume.',
        confidence: 0,
        evidence: [],
        isSafe: false,
        latency: Date.now() - startTime
      });
    }

    // Step 3: Rerank for accuracy
    const chunks = retrieved.map(r => r.chunkText);
    const reranked = await rerank(question, chunks, 5);
    const topChunks = reranked.map(r => r.text);

    // Step 4: Get safe answer with evidence
    const safeAns = await safeAnswer(question, topChunks, 0.7);

    // Step 5: Validate faithfulness (optional)
    const faithfulness = await validateFaithfulness(safeAns.answer, safeAns.evidence);

    const latency = Date.now() - startTime;

    res.json({
      success: true,
      answer: safeAns.answer,
      confidence: safeAns.confidence,
      evidence: safeAns.evidence,
      isSafe: safeAns.isSafe,
      faithfulness: faithfulness.score,
      latency,
      chunksRetrieved: retrieved.length,
      chunksReranked: reranked.length
    });

  } catch (error: any) {
    logger.error('Query error', { error: error.message });
    res.status(500).json({ error: error.message });
  }
});

/**
 * Get metrics for a tenant.
 * GET /rag/metrics/:tenantId
 */
router.get('/metrics/:tenantId', async (req: Request, res: Response) => {
  try {
    const { tenantId } = req.params;

    // Query logs aggregation
    const metrics = await db.one(`
      SELECT 
        COUNT(*) as query_count,
        AVG(latency_ms) as avg_latency,
        MAX(latency_ms) as max_latency,
        MIN(latency_ms) as min_latency,
        AVG(recall) as avg_recall,
        AVG(precision) as avg_precision,
        SUM(cost_cents) / 100.0 as total_cost_dollars
      FROM query_logs
      WHERE tenant_id = $1
    `, [tenantId]);

    res.json({
      success: true,
      metrics
    });

  } catch (error: any) {
    logger.error('Metrics error', { error: error.message });
    res.status(500).json({ error: error.message });
  }
});

export default router;

PART 9: MAIN SERVER

// src/index.ts

import express from 'express';
import cors from 'cors';
import compression from 'compression';
import helmet from 'helmet';
import dotenv from 'dotenv';
import ragRoutes from './routes/rag';
import { initializeDatabase } from './config/database';
import logger from './utils/logger';

dotenv.config();

const app = express();
const PORT = process.env.PORT || 3000;

// Middleware
app.use(helmet()); // Security headers
app.use(compression()); // Compress responses
app.use(cors());
app.use(express.json());

// Health check
app.get('/health', (req, res) => {
  res.json({ status: 'ok', timestamp: new Date().toISOString() });
});

// RAG routes
app.use('/rag', ragRoutes);

// Error handler
app.use((err: any, req: express.Request, res: express.Response, next: express.NextFunction) => {
  logger.error('Unhandled error', { error: err.message });
  res.status(500).json({ error: 'Internal server error' });
});

// Start server
async function start() {
  try {
    // Initialize database
    await initializeDatabase();

    app.listen(PORT, () => {
      logger.info(`Server running on port ${PORT}`);
    });
  } catch (error) {
    logger.error('Failed to start server', { error });
    process.exit(1);
  }
}

start();

Logging Service

// src/utils/logger.ts

import winston from 'winston';

const logger = winston.createLogger({
  level: process.env.LOG_LEVEL || 'info',
  format: winston.format.combine(
    winston.format.timestamp(),
    winston.format.json()
  ),
  transports: [
    // Console in development
    new winston.transports.Console({
      format: winston.format.combine(
        winston.format.colorize(),
        winston.format.printf(({ timestamp, level, message, ...meta }) => {
          return `${timestamp} [${level}] ${message} ${
            Object.keys(meta).length ? JSON.stringify(meta, null, 2) : ''
          }`;
        })
      )
    }),
    // File for production
    new winston.transports.File({ 
      filename: 'logs/error.log',
      level: 'error'
    }),
    new winston.transports.File({ 
      filename: 'logs/combined.log'
    })
  ]
});

export default logger;

PART 10: FAILURE POINTS & SOLUTIONS

Summary Table

Failure Point	Symptom	Root Cause	Solution
Missing tenant_id	Data leak between companies	No isolation check	Add WHERE tenant_id = X to EVERY query
Vector index missing	Queries take 10+ seconds	Sequential scan of 500K vectors	Create IVFFLAT index on embedding column
Query too long	API error 4096 tokens exceeded	Question >16000 chars	Truncate queries to 500 chars
No results	Empty array returned	Chunks don't exist or embeddings wrong	Check if chunks were saved, verify vector distance threshold
Hallucination	AI invents information	No retrieval boundaries in prompt	Use safety layers (5 layers as described)
Rate limit (429)	API call fails	Too many requests to Claude	Implement exponential backoff, queue requests
Database connection lost	"Cannot connect to server"	Network issue, DB down, wrong credentials	Add retry logic, connection pooling, health checks
Embedding dimension mismatch	"Vector dimension 1536 != 768"	Using different embedding model	Ensure consistent model (claude-3-5-sonnet)
Memory overload	Node.js crashes	Trying to embed entire 100MB file	Chunk before embedding, process in batches
Cost explosion	Unexpected $10k bill	Each embedding/rerank/answer costs money	Track costs, log them, set spending limits

PART 11: PROS AND CONS OF RAG

PROS ✓

1. Accuracy

✓ AI answers from evidence, not training data
✓ No hallucinations (with proper safety layers)
✓ Can trace every answer to source

Without RAG: "Does John know Docker?" → Guess → "maybe, looks like it"
With RAG: "Does John know Docker?" → Evidence → "Yes. His resume says: 'Docker, Kubernetes, 4 years'"

2. Up-to-Date

✓ Works with information from last month
✓ No retraining needed
✓ Can add 1000 new resumes per day

3. Explainability

✓ Users see why the system answered
✓ Can audit decisions
✓ Legally defensible

4. Cost

✓ Cheaper than retraining models
✓ Only pay for what you use
✓ PostgreSQL is free (open source)

5. Flexibility

✓ Can use with ANY language
✓ Works with specialized domains
✓ Easy to update documents

CONS ✗

1. Complexity

✗ More moving parts: embeddings, database, retrieval, reranking, safety
✗ More things to break
✗ More things to debug

Single LLM: Simple
RAG: Embeddings → Vector DB → Retrieval → Reranking → Safety checks
Each stage can fail independently

2. Retrieval Quality

✗ If wrong chunks retrieved, answer is wrong
✗ Embeddings are fuzzy (not perfect matches)
✗ Rare topics might not be in training data

Example problem:

Resume: "Worked with distributed ledger technology"
Search: "blockchain"
Result: Miss (DLT ≠ blockchain in embeddings)

3. Cost of Operations

✗ Each embedding: $0.00001
✗ Each rerank: $0.0001
✗ Each LLM answer: $0.001
✗ Adds up fast with scale

100K queries/month = $100/month just for operations
(Not including infrastructure, salaries, etc)

4. Latency

✗ Vector search: 100ms
✗ Reranking: 300-500ms
✗ Safety check: 200ms
✗ Total: 600-800ms

User expects <200ms. RAG adds delay.

5. Data Quality Matters

✗ If source documents are bad, answers are bad
✗ Garbage in = garbage out
✗ No amount of engineering fixes bad data

Resume full of typos: "React" → "Rreact" → Embeddings confused
→ System can't find React skills
→ Answer is wrong

6. Scaling is Hard

✗ Vector index maintenance complex
✗ Multi-tenancy requires isolation at every level
✗ Database queries become slow with 10M+ chunks

When to Use RAG

Scenario	Use RAG?	Why
Customer support	YES	Up-to-date, explainable, no hallucinations
Medical diagnosis	YES	Safety-critical, needs evidence
Resume screening	YES	Domain-specific, needs accuracy
General chatbot	NO	Training data sufficient, latency matters
Quick facts	NO	Simple lookup is faster
Creative writing	NO	Hallucinations are features, not bugs
Code search	MAYBE	Depends on code freshness
Legal documents	YES	Must cite sources, no mistakes

When NOT to Use RAG

1. Simple lookup → Use database
2. Conversational → Use base LLM
3. Speed critical → Too slow (600ms+)
4. Data quality poor → Garbage in/out
5. Training data sufficient → No value-add
6. Cost-sensitive → Each query costs money

PART 12: COST ANALYSIS

Break Down of Costs

Per-Query Costs (approximate):

1. Embedding query:
   - 50 tokens @ $0.000003/token = $0.00015

2. Vector search + keyword filter:
   - Database operation ≈ $0 (hosted: ~$0.00001)

3. Reranking (Claude):
   - 500 tokens input + 100 output @ $0.003/$0.015 = $0.0018

4. Final answer:
   - 500 tokens input + 200 output @ $0.003/$0.015 = $0.004

Total per query: ≈ $0.0075 (~0.75 cents)

At scale:
- 1K queries/month = $7.50
- 100K queries/month = $750
- 1M queries/month = $7,500

How to Reduce Costs

Cache results
- Same question asked twice? Use Redis cache
- Saves: 80% of cost
Batch processing
- Don't embed one by one
- Embed 100 at a time
- Saves: 20% (batch discounts)
Smart reranking
- Only rerank if needed
- Don't rerank for obvious matches
- Saves: 30-50%
Cheaper models
- Use Claude Haiku for simple tasks
- Claude Opus only for complex reasoning
- Saves: 80%

PART 13: DEPLOYMENT & PRODUCTION

Docker Deployment

# Dockerfile

FROM node:18-alpine

WORKDIR /app

COPY package*.json ./
RUN npm ci --only=production

COPY dist ./dist

EXPOSE 3000

CMD ["node", "dist/index.js"]

Docker Compose (Local Dev)

# docker-compose.yml

version: '3.8'
services:
  postgres:
    image: postgres:15-alpine
    environment:
      POSTGRES_PASSWORD: postgres
      POSTGRES_DB: rag_system
    volumes:
      - postgres_data:/var/lib/postgresql/data
    ports:
      - "5432:5432"

  pgvector:
    build:
      context: .
      dockerfile: Dockerfile.pgvector
    environment:
      POSTGRES_PASSWORD: postgres
    depends_on:
      - postgres

  app:
    build: .
    environment:
      DB_HOST: postgres
      DB_USER: postgres
      DB_PASSWORD: postgres
      DB_NAME: rag_system
      ANTHROPIC_API_KEY: ${ANTHROPIC_API_KEY}
    ports:
      - "3000:3000"
    depends_on:
      - postgres

volumes:
  postgres_data:

Deployment Checklist

[ ] Environment variables set correctly
[ ] Database indexes created
[ ] API key securely stored (use AWS Secrets Manager)
[ ] Logging configured (CloudWatch or similar)
[ ] Monitoring set up (CPU, memory, latency)
[ ] Rate limiting enabled
[ ] Database backups enabled
[ ] Error alerts configured
[ ] Cost monitoring enabled
[ ] Security scan passed
[ ] Load test passed (1000 req/sec)
[ ] Graceful shutdown implemented

PART 14: EXAMPLE QUERIES

Complete End-to-End Example

# 1. Upload a resume
curl -X POST http://localhost:3000/rag/upload \
  -H "Content-Type: application/json" \
  -H "X-API-Key: super-secret-key" \
  -d '{
    "tenantId": "company-1",
    "candidateName": "John Doe",
    "resumeText": "John has 5 years React experience, built e-commerce platforms with Node.js..."
  }'

# Response:
# {
#   "success": true,
#   "resumeId": "uuid-123",
#   "chunkCount": 8
# }

# 2. Query the resume
curl -X POST http://localhost:3000/rag/query \
  -H "Content-Type: application/json" \
  -H "X-API-Key: super-secret-key" \
  -d '{
    "tenantId": "company-1",
    "resumeId": "uuid-123",
    "question": "Does John have React experience?",
    "useExpansion": true
  }'

# Response:
# {
#   "success": true,
#   "answer": "Yes, John has 5 years of React experience.",
#   "confidence": 0.95,
#   "evidence": ["John has 5 years React experience, built e-commerce..."],
#   "isSafe": true,
#   "faithfulness": 0.92,
#   "latency": 720,
#   "chunksRetrieved": 10,
#   "chunksReranked": 5
# }

# 3. Get metrics
curl -X GET http://localhost:3000/rag/metrics/company-1 \
  -H "X-API-Key: super-secret-key"

# Response:
# {
#   "success": true,
#   "metrics": {
#     "query_count": 42,
#     "avg_latency": 680,
#     "max_latency": 1200,
#     "avg_recall": 0.91,
#     "avg_precision": 0.88,
#     "total_cost_dollars": 0.32
#   }
# }

FINAL CHECKLIST: BEFORE YOU SHIP

Functionality

[ ] Embeddings working (call Claude API successfully)
[ ] PostgreSQL storing vectors correctly
[ ] Hybrid search returning results
[ ] Reranking improving quality
[ ] Safety layers preventing hallucinations
[ ] Multi-tenancy isolated (tenant1 can't see tenant2)

Performance

[ ] Vector search <200ms
[ ] Full query <1000ms
[ ] No N+1 queries
[ ] Database queries indexed
[ ] Caching working

Security

[ ] API keys not logged
[ ] Tenant isolation verified (manual test)
[ ] SQL injection prevented (parameterized queries)
[ ] Rate limiting enabled
[ ] HTTPS enforced

Operations

[ ] Logging configured
[ ] Error alerts set up
[ ] Cost tracking enabled
[ ] Database backups tested
[ ] Graceful error handling

Monitoring

[ ] Query latency tracked
[ ] Recall/precision measured
[ ] Cost per query logged
[ ] Error rate tracked
[ ] Uptime monitored

Quick Reference: Common Issues & Fixes

Issue: Queries Take 10+ Seconds

Cause: Missing vector index
Fix:

CREATE INDEX ON resume_chunks 
USING ivfflat (embedding vector_cosine_ops) 
WITH (lists = 100);

Issue: Same Query Returns Different Results

Cause: Stale embeddings or non-deterministic reranking
Fix: Always embed with same model, use fixed random seed for reranking

Issue: AI Answers with Wrong Information

Cause: Insufficient safety layers
Fix: Add confidence thresholding, require citations, validate faithfulness

Issue: "Tenant1 sees Tenant2 data"

Cause: Missing tenant_id check in WHERE clause
Fix: Add WHERE tenant_id = $X to EVERY query

Issue: API calls get rate limited (429)

Cause: Too many rapid requests to Claude
Fix: Implement exponential backoff, queue requests, batch operations

Issue: Cost explodes unexpectedly

Cause: No cost tracking, inefficient queries, excessive reranking
Fix: Log cost per operation, implement budgets, use cheaper models for easy tasks

The Bottom Line

👨‍🦳 Uncle's Final Word:

RAG is powerful but complex. Every layer serves a purpose:

Embeddings = understand meaning
Vector DB = store and search fast
Retrieval = find relevant information
Reranking = improve quality
Safety = prevent lies
Operations = track costs and metrics

You don't need all of this on day one. Start simple:

Day 1: PostgreSQL + embeddings + basic search
Week 1: Add reranking
Month 1: Add safety layers
Month 3: Add monitoring and optimization

Each layer buys you something. Know what you're buying.

👦 Nephew: When should I NOT use RAG?

👨‍🦳 Uncle: When:

Data is stable and doesn't change
You need sub-200ms latency
You're paying $10/month and can't afford $100+
You're building a simple chatbot
Your source data has low quality

Otherwise? RAG is the way.

Now go build. Start simple. Measure everything. Ship fast.

Good luck. ---SurajK

RAG Pipeline: The Uncle-Nephew Complete Learning Guide

surajrkhonde — Sat, 20 Jun 2026 09:44:17 +0000

How to Build Systems That Actually Know Your Data (Not Hallucinate About It)

Introduction: The Story Begins

👦 Nephew: Uncle, I keep hearing "RAG this, RAG that" in tech interviews. When I ask what it means, people throw around words like "Retrieval-Augmented Generation" and I just nod like I understand. But honestly? I'm lost.

👨‍🦳 Uncle: (laughing) That's the best honest question I've heard all week. Let me ask you something first. If I gave you a question right now - "What year did India win the World Cup?" - how would you answer?

👦 Nephew: Well... I'd pull up Google, search for it, read the answer, then tell you.

👨‍🦳 Uncle: Exactly. You don't answer from memory alone. You go fetch the information first, then answer based on what you found. That's RAG in real life. And that simple idea - fetch first, answer after - fixes almost every problem we face with AI today.

👦 Nephew: But uncle, AI can remember things from its training. Why does it need to fetch?

👨‍🦳 Uncle: Ah! That's where we land in trouble. Come, sit...

SECTION 1: RAG FUNDAMENTALS - The Core Concept

The Problem We're Actually Solving

👨‍🦳 Uncle: Imagine you're hiring for a tech company. You receive 500 resumes for a Senior React Developer role. Now tell me - how would you actually process them?

👦 Nephew: I'd... probably make a spreadsheet? List all the candidates with key skills?

👨‍🦳 Uncle: Right. But here's the catch - you can't read all 500 resumes deeply. So what do you really do?

👦 Nephew: Skim for keywords like "React", "JavaScript", "5 years"?

👨‍🦳 Uncle: Exactly. You skim and hope you don't miss anyone good. Now, here's the problem: what if a candidate wrote "React.js" instead of "React"? Your eyes might still catch it. But a dumb computer doing exact string matching? It says "no match".

What if someone wrote "Built real-time user interfaces with the React framework"? The candidate clearly knows React, but the word "React" appears nowhere in that sentence. The computer misses them.

This is exactly what happens with AI. When you ask an AI a question, it tries to answer purely from what it memorized during training. And it makes mistakes - sometimes big ones.

👦 Nephew: So RAG stops these mistakes?

👨‍🦳 Uncle: Precisely. RAG is simple: instead of asking the AI to guess, you hand it the document first. You say, "Here's the resume, here's the job description - now answer my question."

The AI stops guessing. It reads the evidence. It answers correctly.

What is RAG Really?

👦 Nephew: Okay, so RAG = give the AI documents, then ask questions?

👨‍🦳 Uncle: Yes, but we need to be precise about how we give it documents. RAG has three steps:

Retrieval - Find the right documents
Augmentation - Add those documents to the AI's prompt
Generation - Let AI answer based on what it read

That's it. R-A-G.

👦 Nephew: But if it's that simple, why is everyone talking about it like it's complicated?

👨‍🦳 Uncle: Because "finding the right documents" is the hard part. You have 500 resumes. When someone asks "Does John have Docker experience?", you can't search all 500 linearly. That's slow.

And you can't use simple keyword search either - what if John wrote "containerization" instead of "Docker"? What if he wrote "I work with Kubernetes" - which means he knows Docker too?

So the real question becomes: How do you find the right documents fast, and how do you understand when two different words mean the same thing?

This is where everything else - embeddings, vector databases, chunking - comes in. They're all in service of solving that one problem.

👦 Nephew: Okay, I think I get the big picture. But uncle, why can't we just make the AI smarter?

👨‍🦳 Uncle: Two reasons. First - you train an AI once. After that, it doesn't learn new information. If your company has 100 internal policies created last month, the AI knows nothing about them. It can't learn them instantly.

Second - even if you could retrain it, hallucinations would still happen. AIs are pattern-matching machines. They're brilliant at patterns. But they sometimes see patterns that aren't there. RAG forces the AI to cite its sources, to point at evidence.

It's the difference between:

"I think John knows Docker" (guess)
"John knows Docker because his resume clearly says 'Docker, Kubernetes, 4 years'" (evidence)

The second one is what you want. That's RAG.

SECTION 2: EMBEDDINGS - Understanding Meaning

The Fundamental Problem with Words

👦 Nephew: Uncle, I have a question. How does a computer know that "React" and "React.js" are the same thing?

👨‍🦳 Uncle: That's the right question. Let me explain with a story.

I go to a fruit market. I see apples, mangoes, oranges. How do I know which is which? I look at them - color, shape, smell. My brain recognizes patterns and says "that's a mango".

Now, how does a computer do that? It doesn't have eyes. All it has are numbers.

👦 Nephew: Numbers?

👨‍🦳 Uncle: Yes. Here's the trick: every word - "React", "React.js", "JavaScript", "Python" - is just a number to a computer. Or actually, a list of numbers. We call this list an "embedding".

👦 Nephew: Like a list of... what kind of numbers?

👨‍🦳 Uncle: Imagine I'm describing a person to you. I might say:

Height: 180 cm
Age: 30
Skin tone: medium
Hair color: black

That's 4 numbers describing 1 person. Now imagine I use 1536 numbers instead. Each number describes a different quality - not just physical things, but hidden things like "how technical is this word", "is this related to web development", "how often is this used", and so on.

So "React" becomes a point in a 1536-dimensional space. "React.js" becomes another point in that same space. And because they mean the same thing, those two points are very close to each other.

But "React" and "Python"? Those points are far apart.

👦 Nephew: Okay, so closeness = similarity?

👨‍🦳 Uncle: Exactly. And here's the beautiful part: you don't have to manually create these embeddings. An AI model does it for you. You feed it the word "React" and it says, "This word should be represented as [0.12, -0.45, 0.78, ..., 1536 numbers]".

Different models might represent it differently, but the same model always represents similar concepts closely. That's what matters.

Why This Is Genius

👦 Nephew: So embeddings let computers understand meaning?

👨‍🦳 Uncle: They let computers represent meaning as a position in space. And when you represent things as positions, you can do math with them.

For example, if I tell you: "Engineer" - "Java" + "React" = ?

👦 Nephew: That's... weird math?

👨‍🦳 Uncle: Right? But with embeddings, you can actually do this. And you know what the answer is?

The vector closest to that result is usually "web developer" or something similar. The math captures meaning.

Now here's why this matters for RAG:

When someone asks "Does John have Docker experience?", we convert that question to an embedding - a point in 1536D space. Then we search for resume chunks that are close to that point. The chunks mentioning Docker - whether they say "Docker", "containerization", "container orchestration", or "I deploy with Docker" - are all close to the question's embedding.

So we find them all. And the AI reads them. And the AI answers correctly.

Without embeddings, simple keyword search would miss half of them.

SECTION 3: VECTOR DATABASES - Storing at Scale

The Storage Problem Nobody Talks About

👦 Nephew: Okay, so we have embeddings. But uncle, if I have a company with 10,000 resumes, and each resume has 50 chunks, that's 500,000 embeddings. Each embedding is 1536 numbers. How do I store and search that?

👨‍🦳 Uncle: This is where most people choose poorly. They think: "I'll use Pinecone! I'll use Milvus! I'll use Weaviate!"

They add three systems. One for storage, one for search, one for cache. Now they have 3 things that could break. 3 things to debug. 3 bills to pay.

There's a better way.

👦 Nephew: What?

👨‍🦳 Uncle: PostgreSQL.

👦 Nephew: The database? Just... PostgreSQL?

👨‍🦳 Uncle: With one addition: the pgvector extension. PostgreSQL is already reliable, it's already running your data, it's already scaling. We just add vector support.

Think about it this way: 500,000 vectors stored in PostgreSQL as data. When you have a question, you convert it to an embedding, and you say:

SELECT * FROM resume_chunks 
ORDER BY embedding <=> question_embedding 
LIMIT 5

That <=> operator means "find the 5 vectors closest to my question vector". PostgreSQL finds them in milliseconds. You're done.

No new infrastructure. No vendor lock-in. Just one database doing one job well.

👦 Nephew: But doesn't the <=> operator need an index to be fast?

👨‍🦳 Uncle: Good catch. Yes. PostgreSQL's pgvector creates an IVFFLAT index by default. Think of it like this:

When you have 500,000 resumes and need to find the 5 closest to a question, searching all 500,000 linearly is slow. The index divides the 1536D space into regions (like neighborhoods). When you search, it says "The question's embedding is closest to this neighborhood" and only searches that neighborhood. Much faster.

A linear search of 500K vectors? 5 seconds. With the index? 50-100ms.

That's the magic of the right index.

Setting It Up

👦 Nephew: How do I actually create this?

👨‍🦳 Uncle: Simple. PostgreSQL + pgvector. Here's what you do:

-- Create extension
CREATE EXTENSION IF NOT EXISTS vector;

-- Create table with vector column
CREATE TABLE resume_chunks (
  id SERIAL PRIMARY KEY,
  resume_id UUID,
  chunk_text TEXT,
  embedding vector(1536),
  created_at TIMESTAMP DEFAULT NOW()
);

-- Create index for fast search
CREATE INDEX ON resume_chunks USING ivfflat (embedding vector_cosine_ops)
  WITH (lists = 100);

Now you have a table. You insert chunks with their embeddings. You query with the <=> operator. Done.

👦 Nephew: That's it?

👨‍🦳 Uncle: That's it. No API calls to external services. No paying per query. Just a single, reliable database.

This is the beauty of choosing the right tool.

SECTION 4: BASIC RETRIEVAL - Finding What You Need

How Retrieval Works

👦 Nephew: Uncle, we have embeddings, we have a database. How does actual retrieval happen?

👨‍🦳 Uncle: Let me walk you through a real scenario. A hiring manager asks: "Does Priya have blockchain experience?"

Step 1: Understand the question
We need to find chunks about blockchain skills.

Step 2: Convert question to embedding
"blockchain experience" becomes a vector: [0.34, -0.12, ..., 1536 numbers]

Step 3: Search the database
We query:

SELECT chunk_text, similarity FROM resume_chunks 
WHERE resume_id = 'priya-uuid'
ORDER BY embedding <=> question_embedding
LIMIT 5

Step 4: Get results (might be):

"Ethereum smart contracts development"
"Bitcoin protocol understanding"
"Web3 and DeFi platform expertise"
"Solidity programming language"
"Cryptocurrency platform design"

Step 5: Send to AI
We put these 5 chunks in the prompt:

Based on the following resume excerpts:
[chunk 1]
[chunk 2]
[chunk 3]
[chunk 4]
[chunk 5]

Question: Does Priya have blockchain experience?

Step 6: AI answers
"Yes, Priya has strong blockchain experience. She has worked with Ethereum smart contracts, understands Bitcoin protocol, and has built on Web3/DeFi platforms using Solidity."

With evidence.

Why Basic Retrieval Fails Sometimes

👦 Nephew: This seems perfect. What's the problem?

👨‍🦳 Uncle: The problem is subtle. Resume has this line:

"I worked on distributed ledger technology and consensus protocols"

This person knows blockchain. But the word "blockchain" is not there.

You search for "blockchain". Embeddings are smart, but not magic. "Distributed ledger technology" and "blockchain" are close but not identical.

Sometimes the vector search returns this chunk. Sometimes it doesn't.

Basic retrieval works maybe 70% of the time. For production, you need 95%+.

That's why we need advanced techniques.

👦 Nephew: So what makes it better?

👨‍🦳 Uncle: Multiple techniques working together. But we'll get there. First, let's talk about preparation.

SECTION 5: CHUNKING STRATEGIES - Smart Breaking

The Chunking Dilemma

👦 Nephew: Uncle, let me ask something. Why do we break resumes into chunks at all? Why not just embed the whole resume?

👨‍🦳 Uncle: Good question. Two reasons:

Cost - An embedding costs money (usually). A 50-page resume broken into chunks might cost $0.50 instead of $5 if you embed it whole.
Precision - If you embed the entire resume as one chunk, and you search for "Docker experience", the system returns the whole 50-page resume. The AI has to find the Docker mention itself. But if you break it into chunks, the system returns only the chunks about Docker. The AI reads less noise.

But here's the problem: how small should chunks be?

Size Matters More Than You Think

👦 Nephew: Can't I just pick a number? Like 500 tokens per chunk?

👨‍🦳 Uncle: Let's see what happens with different sizes.

Attempt 1: Too small (100 tokens)
Chunk 1: "John has"
Chunk 2: "5 years of"
Chunk 3: "React experience"

Problem: When we read Chunk 3, we've lost context. 5 years of what? The AI is confused.

Attempt 2: Right size (1000 tokens)
"John has 5 years of React experience, built e-commerce platforms, Redux for state management, and mentored junior developers."

Perfect. Context is clear.

Attempt 3: Too large (5000 tokens)
Entire work history section, with all jobs, all skills, everything mixed together.

Problem: Too much noise. When the AI searches for "React", it gets 5000 tokens to read, but only 2 sentences mention React. It's slow and confusing.

👨‍🦳 Uncle: The sweet spot is usually 1000-1500 tokens per chunk. Not less, not more.

Overlap is Important

👦 Nephew: What's overlap? Overlapping chunks?

👨‍🦳 Uncle: Yes. Imagine this:

Without overlap:
Chunk 1 ends: "...John used React..."
Chunk 2 starts: "...for building e-commerce..."

When you read Chunk 2 alone, you don't know John used React for this. Context is broken.

With 200-token overlap:
Chunk 1: "...John has React experience. He built e-commerce..."
Chunk 2: "...He built e-commerce with Redux and WebSockets..."

Now each chunk contains enough context. You can read them individually and still understand.

👦 Nephew: So overlap = context continuity?

👨‍🦳 Uncle: Exactly. It's like reading a book. Each page overlaps the previous one slightly. You always have context.

Chunking Strategies

👦 Nephew: Are there different ways to chunk?

👨‍🦳 Uncle: Yes. Here are the main ones:

Strategy	Chunk Size	Overlap	Best For	Trade-off
Naive/Fixed	512 tokens	None	Simple code	Loses context
Sliding Window	1000-1500	200	Resumes, documents	Balanced
Semantic	Variable	Variable	Technical docs	Complex to implement
Recursive	1000 first, then smaller	200	Large documents	More processing

For resumes? Sliding window, 1000 tokens, 200-token overlap. It's what works best.

Here's example code:

def sliding_window_chunk(text, window_size=1000, overlap=200):
    """
    Break text into chunks with overlap.
    window_size = tokens per chunk
    overlap = tokens of overlap between chunks
    """
    tokens = text.split()  # Simplified tokenization
    chunks = []

    for i in range(0, len(tokens), window_size - overlap):
        chunk = tokens[i:i + window_size]
        chunks.append(" ".join(chunk))

        if i + window_size >= len(tokens):
            break

    return chunks

# Example
resume_text = "John has 5 years React... [long text]"
chunks = sliding_window_chunk(resume_text, 1000, 200)
# chunks[0] = "John has 5 years React... [1000 tokens]"
# chunks[1] = "[last 200 tokens of chunk 0] ... [next 800 tokens]"

Simple. Effective. This is what production systems use.

SECTION 6: HYBRID SEARCH - Vector + Keywords

The Limitation of Pure Vector Search

👦 Nephew: Uncle, embeddings are smart. But are they perfect?

👨‍🦳 Uncle: No. Here's a real problem:

Resume has: "Kubernetes 1.26"
You search for: "Kubernetes 1.26"

Vector search thinks:

"Kubernetes 1.26" is a version
"Kubernetes 1.25" is also a version
They're similar technologies
Distance: 0.85 (similar!)

FALSE MATCH! The versions are different, but vectors say they're similar.

👦 Nephew: So vectors miss exact matches?

👨‍🦳 Uncle: Not miss - they're fuzzy. They blur differences. Sometimes that's good (finding "React" when you search "React.js"). Sometimes it's bad (confusing version numbers).

This is why you need both: vectors AND keywords.

Keyword Search: The Simple Part

👦 Nephew: How does keyword search work?

👨‍🦳 Uncle: Simple. Exact string matching.

Resume: "Kubernetes 1.26"
Search: "1.26"
Result: EXACT MATCH. Yes.

Resume: "Kubernetes 1.25"
Search: "1.26"
Result: NO MATCH. No.

It's binary. No fuzzy. Just right or wrong.

Hybrid = Best of Both

👦 Nephew: So we use both?

👨‍🦳 Uncle: Exactly. Here's how:

Search "Kubernetes 1.26"

Step 1: Vector search on all chunks

Returns top 20 chunks that mention versions/container orchestration

Step 2: Keyword filter

On those 20 chunks, filter by exact keyword match: "1.26"
Returns maybe 3 chunks with exact version

Step 3: Combine and rerank

Return the 3 high-precision results

Result: Perfect accuracy. Both semantic understanding AND exact precision.

PostgreSQL can do this:

SELECT chunk_text 
FROM resume_chunks 
WHERE resume_id = 'john-uuid'
  AND to_tsvector(chunk_text) @@ plainto_tsquery('Kubernetes 1.26')
  AND embedding <=> question_embedding < 0.2
ORDER BY embedding <=> question_embedding
LIMIT 5

This query says:

Find chunks where text contains "Kubernetes" AND "1.26" (keyword search)
AND the embedding is close to the question (vector search)
Order by vector similarity

Both working together.

When Each Excels

👦 Nephew: When do I use pure vector?

👨‍🦳 Uncle: When you're searching for concepts:

"Tell me about this candidate's experience with modern development"
"What's their background in cloud systems?"

Vectors are great here. They find related concepts even with different words.

👦 Nephew: And pure keyword?

👨‍🦳 Uncle: For exact facts:

"What certifications does John have?"
"What's their GPA?"
"When did they graduate?"

Keywords are perfect. You don't need fuzzy matching.

👦 Nephew: And hybrid?

👨‍🦳 Uncle: For most real scenarios. Maximum accuracy. That's what production uses.

SECTION 7: RERANKING - Ranking Results Better

The Quality Problem

👦 Nephew: Uncle, we retrieve 5 chunks. But are they the best 5?

👨‍🦳 Uncle: That's a great question. Let me show you a problem:

Search: "Docker experience"

Vector search returns:

"I have 5 years Docker production experience" (similarity: 0.92)
"Container orchestration with Docker expertise" (similarity: 0.91)
"My Docker projects all failed miserably" (similarity: 0.90) ← PROBLEM!
"Docker mentioned in course material" (similarity: 0.85)
"Web dev includes Docker basics" (similarity: 0.82)

Items 3-5 shouldn't be in the top 5! Item 3 is negative, items 4-5 are weak.

Basic vector search ranks by similarity score. But similarity doesn't capture quality.

👦 Nephew: So we need a better ranker?

👨‍🦳 Uncle: Yes. And here's the trick: we use two models.

Two-Stage Retrieval

👦 Nephew: Two models? That sounds expensive.

👨‍🦳 Uncle: Exactly. So we use them smartly.

Stage 1: Fast retrieval (milliseconds)
Vector search returns 20 candidates. Speed matters here, so we use a fast model.

Time: 100ms
Quality: 70%

Stage 2: Accurate ranking (milliseconds)
We rerank those 20 with a better, slower model.

Time: 300-500ms
Quality: 95%

Total time: 500ms. Still fast. But now the top 5 are good.

The key insight: We don't apply the accurate model to all 500,000 chunks. That's too slow. We apply it only to 20. That's feasible.

Available Rerankers

👦 Nephew: What models should I use?

👨‍🦳 Uncle: A few options:

Reranker	Source	Quality	Cost
BGE-Reranker-Base	Open source	Very Good	Free
Cohere Rerank	API	Excellent	$$
Claude/GPT-4	API	Best	$$$

For production? Use BGE. It's free, open-source, and performs almost as well as expensive options.

Here's how:

from sentence_transformers import CrossEncoder

# Load open-source reranker
reranker = CrossEncoder('BAAI/bge-reranker-base')

# We have top 20 chunks from vector search
top_20_chunks = [...list of chunks...]

# Rerank them
scores = reranker.predict([[query, chunk] for chunk in top_20_chunks])

# Sort by score
ranked = sorted(zip(top_20_chunks, scores), key=lambda x: x[1], reverse=True)

# Return top 5
return ranked[:5]

That's it. Takes 300ms. Makes huge quality difference.

SECTION 8: QUERY PROCESSING - Smart Questions

Queries Are Ambiguous

👦 Nephew: Uncle, when someone asks "Does John have frontend skills?", does the AI understand that means React, Vue, JavaScript, HTML?

👨‍🦳 Uncle: Not automatically. Here's the problem:

Resume has: "React, JavaScript, HTML, CSS expertise"
Search for: "frontend skills"

Vector search looks for the exact embedding of "frontend skills". But the resume doesn't have those words. It has the components.

A human reads "React, JavaScript, HTML, CSS" and immediately thinks "frontend". A computer doesn't make that leap.

👦 Nephew: So how do we fix it?

👨‍🦳 Uncle: Query rewriting. You expand the query.

Query Expansion

👦 Nephew: Expand it how?

👨‍🦳 Uncle: Instead of searching for "frontend skills", you search for:

(frontend OR React OR Vue OR Angular OR JavaScript OR HTML OR CSS)

Now you find any mention of these. Much better.

Here's code:

def expand_query(query):
    """
    Expand a query into related terms.
    """
    # Simple approach using a dict
    expansions = {
        'frontend': ['React', 'Vue', 'Angular', 'JavaScript', 'HTML', 'CSS'],
        'backend': ['Node.js', 'Python', 'Java', 'databases', 'APIs'],
        'devops': ['Docker', 'Kubernetes', 'AWS', 'CI/CD', 'deployment'],
    }

    for key, synonyms in expansions.items():
        if key in query.lower():
            return query + " OR " + " OR ".join(synonyms)

    return query

# Example
original = "frontend skills"
expanded = expand_query(original)
# Result: "frontend skills OR React OR Vue OR Angular OR JavaScript OR HTML OR CSS"

Multi-Query Retrieval

👦 Nephew: Is there a smarter way?

👨‍🦳 Uncle: Yes! Instead of manually expanding, let an LLM generate variations.

User asks: "Real-time system experience?"

LLM thinks: "That means WebSockets, Socket.IO, real-time updates, pub/sub, message queues"

LLM generates multiple queries:

"WebSockets and real-time communication"
"Socket.IO implementation"
"Real-time data streaming"
"Redis pub/sub usage"
"Message queue systems"

You search all 5 queries. You get chunks from all angles. You combine results.

Coverage: 10x better.

from anthropic import Anthropic

def generate_queries(original_query):
    """
    Generate multiple search queries from one.
    """
    client = Anthropic()

    response = client.messages.create(
        model="claude-3-5-sonnet-20241022",
        max_tokens=500,
        messages=[{
            "role": "user",
            "content": f"""
Given this question, generate 3-5 related search queries 
that would find relevant information. Return only the queries, 
one per line.

Original question: {original_query}

Queries:
"""
        }]
    )

    queries = response.content[0].text.strip().split('\n')
    return [q.strip() for q in queries if q.strip()]

# Example
original = "Real-time system experience?"
queries = generate_queries(original)
# Returns:
# ["WebSockets real-time", "Socket.IO", "Redis pub/sub", "Message queues", "Real-time streaming"]

# Now search each
all_results = []
for q in queries:
    results = vector_search(q)
    all_results.extend(results)

# Deduplicate and return top 5
unique_results = list({r['id']: r for r in all_results}.values())[:5]

This is elegant. The LLM understands what "real-time" means and generates good search queries.

SECTION 9: EVALUATION & METRICS - Measuring Success

The Measurement Problem

👦 Nephew: Uncle, I built a RAG system. It works. But how do I know if it's good?

👨‍🦳 Uncle: This is where most teams fail. They build, ship, and hope. No metrics.

But you can measure. Here are the important metrics:

Key Metrics

👦 Nephew: What should I measure?

👨‍🦳 Uncle: Four main things:

Metric	Measures	Target	Impact
Recall	Of all relevant docs, how many found?	>90%	Missing information
Precision	Of what found, how relevant?	>85%	Wrong context
Latency	Response time	<2 seconds	User experience
Faithfulness	AI stays factual	>95%	Hallucinations

Let me explain each.

Recall vs Precision

👦 Nephew: What's the difference?

👨‍🦳 Uncle: Let me give you an example. Say there are 10 Docker-related chunks in John's resume.

Recall: Of those 10, how many did you find?

You find 9? Recall = 9/10 = 90%
You find 6? Recall = 6/10 = 60%

High recall means: "I found most of the relevant information"

Precision: Of what you returned, how many were relevant?

You return 15 chunks. 10 are about Docker. Precision = 10/15 = 67%
You return 9 chunks. 8 are about Docker. Precision = 8/9 = 89%

High precision means: "Most of what I returned was useful"

Usually there's a tradeoff:

Return everything (high recall, low precision)
Return only the most confident (low recall, high precision)

You want both high.

Measuring

👦 Nephew: How do I actually measure this?

👨‍🦳 Uncle: You need labeled data. A test set.

Take 10 real questions
For each, have a human list all relevant chunks
Run your system
Compare: did it find all relevant chunks? (Recall)
Were all returned chunks relevant? (Precision)

Here's code:

def calculate_recall_precision(retrieved, relevant):
    """
    retrieved: chunks your system found
    relevant: chunks a human labeled as relevant
    """
    retrieved_ids = set(r['id'] for r in retrieved)
    relevant_ids = set(r['id'] for r in relevant)

    # Recall: of all relevant, how many found?
    if len(relevant_ids) == 0:
        recall = 1.0
    else:
        recall = len(retrieved_ids & relevant_ids) / len(relevant_ids)

    # Precision: of what found, how many relevant?
    if len(retrieved_ids) == 0:
        precision = 0.0
    else:
        precision = len(retrieved_ids & relevant_ids) / len(retrieved_ids)

    return recall, precision

# Example
retrieved = [
    {'id': '1', 'text': 'Docker experience'},
    {'id': '2', 'text': 'Kubernetes'},
    {'id': '3', 'text': 'Machine learning'},
]

relevant = [
    {'id': '1', 'text': 'Docker experience'},
    {'id': '2', 'text': 'Kubernetes'},
    {'id': '4', 'text': 'Container orchestration'},
]

recall, precision = calculate_recall_precision(retrieved, relevant)
# Recall = 2/3 = 67% (found 2 of 3 relevant)
# Precision = 2/3 = 67% (2 of 3 returned were relevant)

Latency

👦 Nephew: How do I measure speed?

👨‍🦳 Uncle: Simple:

import time

start = time.time()
results = rag_system.query("Does John have Docker experience?")
latency = (time.time() - start) * 1000  # Convert to ms

print(f"Latency: {latency}ms")

Target: <2000ms (2 seconds). Anything faster is bonus.

Cost

👦 Nephew: What about cost?

👨‍🦳 Uncle: Important for production:

# Cost per query
# Embedding: $0.00001 per 1K tokens
# Reranking: $0.0001 per query (if using paid service)
# LLM answer: $0.001 per query (if using Claude)
# Total: ~$0.0011 per query

# If you do 1M queries/month:
monthly_cost = 1_000_000 * 0.0011  # $1100/month

This is what you need to know for business decisions.

SECTION 10: HALLUCINATION PREVENTION - Keeping AI Honest

Hallucinations: The Biggest Problem

👦 Nephew: Uncle, what's a hallucination?

👨‍🦳 Uncle: Here's a real example:

Resume says: "Worked with React for 2 years"
Hiring manager asks: "Experience with Kubernetes?"
AI responds: "Yes, John has Kubernetes with orchestration."

WRONG! The resume never mentions Kubernetes.

The AI invented an answer. It saw "Docker" or "containers" (maybe), thought "Kubernetes is related to containers", and hallucinated.

This is a hallucination. The AI is lying, but confidently.

Why Hallucinations Happen

👦 Nephew: But why does this happen?

👨‍🦳 Uncle: AIs are pattern-matching machines. They learned during training: "Docker" often appears near "Kubernetes". So the AI's brain has connected them.

Even if you show the AI only a resume mentioning Docker, its training whispers: "And Kubernetes is usually near Docker!"

So the AI thinks: "Probably Kubernetes too."

It's an artifact of how AIs learn. Not a bug. A feature used wrongly.

The Solution: Force the AI to Use Only Given Information

👦 Nephew: How do you stop it?

👨‍🦳 Uncle: Five layers of protection:

Layer 1: Retrieval Boundaries
Show the AI ONLY the retrieved chunks. Nothing from training.

In your prompt:

Based ONLY on the following information:

[Retrieved chunks]

Answer: Does John have Kubernetes experience?

The word "ONLY" is critical.

Layer 2: Structured Output
Force JSON format. Constrain the answer shape.

response = client.messages.create(
    model="claude-3-5-sonnet-20241022",
    messages=[...],
    system="""
You MUST respond in JSON format:
{
  "answer": "yes" or "no",
  "confidence": 0.0 to 1.0,
  "evidence": ["chunk 1", "chunk 2"]
}
""",
)

This forces the AI to cite evidence or it can't answer.

Layer 3: Validation
Check the output against the original chunks:

answer = json.loads(response.content[0].text)

# Check: does evidence exist in chunks?
for evidence in answer['evidence']:
    if evidence not in chunks:
        # Hallucination detected!
        answer['confidence'] = 0  # Reject

return answer

Layer 4: Confidence Gating
Low confidence? Escalate to human:

if answer['confidence'] < 0.7:
    # Hand off to human
    return {
        'answer': 'Unknown - escalated to human',
        'reason': 'Low confidence'
    }

Layer 5: Required Citations
Every claim must point to a chunk:

Question: Does John know Kubernetes?
Retrieved chunks:
[1] "Docker and container experience"
[2] "Kubernetes cluster management"

Answer: Yes, John knows Kubernetes (from chunk 2).

The AI must cite. No citation? No answer.

Together

👦 Nephew: Do you use all five?

👨‍🦳 Uncle: In production? All five. At different stages.

Retrieval boundaries + structured output stop most hallucinations.

Validation + confidence gating catch the rest.

Citations let users verify.

Together? Hallucinations become nearly impossible.

SECTION 11: PRODUCTION ARCHITECTURE - Building for Real Users

Prototype vs Production

👦 Nephew: Uncle, my RAG system works on my laptop. But production is different?

👨‍🦳 Uncle: Completely different.

Prototype version:

Works for me
Might crash
Takes 5 seconds to respond
Processes 1 user at a time

Production version:

Works for everyone
99.9% uptime
Responds in <2 seconds
Handles 1000 concurrent users
Secure (no data leaks)
Monitored (alerts when something breaks)
Compliant (legal requirements)

These are not the same system.

Core Architecture for Production

👦 Nephew: How do I design it?

👨‍🦳 Uncle: Here's a proven architecture:

                    Users
                      ↓
              Load Balancer (Nginx)
             /          |          \
         API-1      API-2      API-3  (scale horizontally)
             \          |          /
                      ↓
                 Cache Layer (Redis)
                      ↓
           PostgreSQL + pgvector + pgvector Index
                      ↓
              External APIs (Claude, etc.)

Load Balancer

Distributes user traffic
No single API gets overwhelmed
One dies? Traffic routes to others

Multiple APIs

Handle load together
Fault tolerant
Can update one without stopping others

Cache

Redis stores recent results
Same question asked twice? Answer from cache
90% faster

Database

Single source of truth
Persistent
Indexed for speed

External APIs

Claude for answers
Anthropic handles scaling there

Each layer has a job. Each layer is replaceable.

Tenant Isolation - Critical

👦 Nephew: Wait, what if I'm hosting this for multiple companies?

👨‍🦳 Uncle: Then you have a SERIOUS security issue if you don't handle isolation.

Company A's data must NEVER be visible to Company B.

Every single query must include:

SELECT * FROM resume_chunks 
WHERE resume_id = 'john-123'
  AND tenant_id = 'company-a'  ← THIS IS CRITICAL

Without the tenant_id check, a hacker or bug could leak data.

This is not optional. This is "you'll be sued" level of important.

SECTION 12: ATS DEEP DIVE - Resume Scoring

What is ATS (Applicant Tracking System)?

👦 Nephew: Uncle, I've heard of ATS. What is it?

👨‍🦳 Uncle: Old ATS systems just searched for keywords. "Does the resume have the word React?" Yes/No.

Modern ATS - the kind you're building with RAG - is intelligent. It understands:

Does the candidate have the skill?
How many years?
At what level?
Do they have relevant projects?

Resume gets a score. Top scorers get interviewed. Bad scorers get rejected.

The difference between old and new? Old ATS rejects good candidates because they phrased things differently. New ATS finds them anyway.

The Scoring Algorithm

👦 Nephew: How do you score a resume?

👨‍🦳 Uncle: With components. Job requirements × candidate abilities.

Job description requires:

React (5+ years)
JavaScript (required)
Node.js (preferred)
AWS (preferred)

Let's say this resume has:

React (6 years) ← exceeds requirement
JavaScript (7 years) ← exceeds requirement
Node.js (4 years) ← has it
AWS (not mentioned)

Here's the scoring:

def score_resume(candidate, job_requirements):
    """
    Score a resume against job requirements.
    """
    scores = {}

    # Required skills: 60% weight
    required_score = 0
    required_count = 0
    for skill, years_required in job_requirements['required'].items():
        required_count += 1
        candidate_years = candidate.get(skill, 0)

        if candidate_years >= years_required:
            required_score += 100  # Full points
        elif candidate_years > 0:
            # Partial credit
            required_score += (candidate_years / years_required) * 100
        # else: 0 points

    required_avg = required_score / required_count if required_count > 0 else 0

    # Preferred skills: 25% weight
    preferred_score = 0
    preferred_count = 0
    for skill in job_requirements['preferred']:
        preferred_count += 1
        if skill in candidate:
            preferred_score += 50  # Half credit for preferred

    preferred_avg = preferred_score / preferred_count if preferred_count > 0 else 0

    # Projects: 15% weight
    project_score = len(candidate.get('projects', [])) * 25
    project_avg = min(project_score, 100)  # Cap at 100

    # Final score
    final_score = (
        required_avg * 0.60 +
        preferred_avg * 0.25 +
        project_avg * 0.15
    )

    return final_score

# Example
candidate = {
    'React': 6,
    'JavaScript': 7,
    'Node.js': 4,
    'projects': ['E-commerce app', 'Real-time chat']
}

requirements = {
    'required': {'React': 5, 'JavaScript': 3},
    'preferred': ['Node.js', 'AWS']
}

score = score_resume(candidate, requirements)
# React: (6 >= 5) = 100 pts
# JavaScript: (7 >= 3) = 100 pts
# Required avg: 100
# Node.js: has it = 50 pts
# Preferred avg: 25
# Projects: 2 × 25 = 50 pts
# Final: (100 × 0.60) + (25 × 0.25) + (50 × 0.15) = 60 + 6.25 + 7.5 = 73.75/100

That's scoring. But wait - how do you extract the data?

Skill Normalization

👦 Nephew: Resume might say "React" but another might say "React.js". How do you normalize?

👨‍🦳 Uncle: Build a dictionary.

skill_aliases = {
    'React': ['React', 'React.js', 'ReactJS', 'react'],
    'Node.js': ['Node.js', 'node.js', 'nodejs', 'Node'],
    'Docker': ['Docker', 'docker', 'containers'],
}

def normalize_skill(mentioned_skill):
    """Find the canonical skill name."""
    mentioned_lower = mentioned_skill.lower()

    for canonical, aliases in skill_aliases.items():
        if mentioned_lower in [a.lower() for a in aliases]:
            return canonical

    return mentioned_skill  # Unknown skill

# Example
print(normalize_skill("React.js"))  # Output: React
print(normalize_skill("nodejs"))    # Output: Node.js

Now "React" and "React.js" are treated as the same skill.

For Unknown Skills: Use Embeddings

👦 Nephew: But what if the resume mentions a skill not in your dictionary?

👨‍🦳 Uncle: Use embeddings to find similar skills.

Resume mentions: "Built trading platform with real-time updates"
Job requires: "WebSockets experience"

These aren't in your dictionary. But you can check similarity:

def find_skill_match(mentioned_skill, required_skill):
    """
    Use embeddings to find matches for unknown skills.
    """
    # Get embeddings
    mentioned_embedding = get_embedding(mentioned_skill)
    required_embedding = get_embedding(required_skill)

    # Calculate distance
    distance = cosine_distance(mentioned_embedding, required_embedding)

    # If similar enough, count as a match
    if distance > 0.75:  # Threshold
        return True

    return False

# Example
if find_skill_match("real-time data updates", "WebSockets"):
    # Candidate probably has WebSockets experience
    score += 100

This is powerful. It catches variations and similar technologies.

SECTION 13: ADVANCED PATTERNS - Agents & Future

What Happens After Basic RAG

👦 Nephew: Uncle, we have RAG working. What comes next?

👨‍🦳 Uncle: Basic RAG answers questions. Advanced RAG (agents) decides what to do.

Difference:

Basic RAG: "Does John know React?" → Find resume → Answer

Agent RAG: "Is John qualified for Senior Developer role?" →

Find resume
Find job description
Compare skills
Check experience level
Look for projects
Verify certifications
Reason through all evidence
Answer with reasoning

The agent is intelligent about what to search.

Agentic RAG with Claude

👦 Nephew: How do I build an agent?

👨‍🦳 Uncle: Using Claude's tool use (function calling):

import anthropic

client = anthropic.Anthropic()

def search_resume(resume_id, query):
    """Search candidate's resume."""
    # Your RAG search logic
    return rag_system.query(resume_id, query)

def search_job_description(job_id, query):
    """Search job description."""
    return job_db.query(job_id, query)

def analyze_candidate(resume_id, job_id):
    """
    Use Claude as an agent to analyze candidate fit.
    Claude decides what to search, what to compare.
    """
    tools = [
        {
            "name": "search_resume",
            "description": "Search a candidate's resume for information",
            "input_schema": {
                "type": "object",
                "properties": {
                    "resume_id": {"type": "string"},
                    "query": {"type": "string"}
                },
                "required": ["resume_id", "query"]
            }
        },
        {
            "name": "search_job",
            "description": "Search job requirements",
            "input_schema": {
                "type": "object",
                "properties": {
                    "job_id": {"type": "string"},
                    "query": {"type": "string"}
                },
                "required": ["job_id", "query"]
            }
        }
    ]

    messages = [{
        "role": "user",
        "content": f"""
Analyze if candidate {resume_id} is qualified for job {job_id}.

Use the available tools to:
1. Find required skills in the job description
2. Check if the candidate has those skills
3. Compare years of experience
4. Look for relevant projects
5. Make a recommendation

Provide detailed reasoning.
"""
    }]

    # Agent loop
    while True:
        response = client.messages.create(
            model="claude-3-5-sonnet-20241022",
            max_tokens=1000,
            tools=tools,
            messages=messages
        )

        # Check if Claude wants to use tools
        if response.stop_reason == "tool_use":
            # Process tool calls
            tool_results = []
            for content in response.content:
                if content.type == "tool_use":
                    if content.name == "search_resume":
                        result = search_resume(
                            content.input["resume_id"],
                            content.input["query"]
                        )
                    else:  # search_job
                        result = search_job_description(
                            content.input["job_id"],
                            content.input["query"]
                        )

                    tool_results.append({
                        "type": "tool_result",
                        "tool_use_id": content.id,
                        "content": str(result)
                    })

            # Add Claude's response and tool results to messages
            messages.append({"role": "assistant", "content": response.content})
            messages.append({"role": "user", "content": tool_results})

        else:
            # Claude is done - extract final answer
            final_answer = ""
            for content in response.content:
                if hasattr(content, 'text'):
                    final_answer = content.text

            return final_answer

# Use it
result = analyze_candidate("john-resume", "senior-dev-job")
print(result)

This is elegant. Claude is the brain. It decides what to search, in what order, and how to reason about it.

The Future of RAG

👦 Nephew: What's next? After agents?

👨‍🦳 Uncle: A few directions:

1. Adaptive RAG
System learns what works. If a question fails, it tries different search strategies automatically.

2. Multi-modal RAG
Right now we handle text. Soon: images, tables, videos. A resume with a photo of their projects. A video walkthrough.

3. Real-time RAG
System continuously updates knowledge. Candidate updates LinkedIn → RAG system knows instantly.

4. Collaborative RAG
Multiple agents reasoning together. One searches, one evaluates, one questions.

5. Explainable RAG
System shows its work. "I rejected John because he doesn't have 5 years React, only 3 years, and React 5+ was required."

These are coming. RAG is only beginning.

CONCLUSION: You're a RAG Architect Now

👨‍🦳 Uncle: Let me summarize what you've learned:

RAG Fundamentals - Fetch documents, then answer. Don't guess.
Embeddings - Represent meaning as positions in space.
Vector Databases - Store embeddings fast. PostgreSQL + pgvector.
Retrieval - Find relevant documents using vector search.
Chunking - Break documents smartly (1000-1500 tokens, 200 overlap).
Hybrid Search - Combine vectors and keywords for accuracy.
Reranking - Use two models: fast retrieval, accurate ranking.
Query Processing - Expand queries to find more relevant docs.
Metrics - Measure recall, precision, latency, faithfulness.
Hallucination Prevention - Five layers: boundaries, structure, validation, confidence, citations.
Production - Load balancing, caching, security, monitoring.
ATS Scoring - Intelligent resume evaluation.
Agents - AI decides what to search and how.

👦 Nephew: That's a lot, uncle. But I think I understand. RAG isn't magic - it's just engineering.

👨‍🦳 Uncle: Exactly. RAG is:

Retrieval (fast lookup)
+ Augmentation (add context)
+ Generation (let AI answer)
= Reliable AI

Every layer serves one purpose: make AI useful for real people.

👦 Nephew: And production?

👨‍🦳 Uncle: Remember: Reliable for 10 users >> Perfect for nobody.

Start simple. Build on PostgreSQL. Measure everything. Iterate fast.

You don't need fancy systems. You need good engineering.

👦 Nephew: Uncle, one last question.

👨‍🦳 Uncle: Go ahead.

👦 Nephew: How do I actually start? Like, code?

👨‍🦳 Uncle: That's next lesson, beta. First, you understand the system. Then you build it.

But I'll give you one gift: a simple starter stack:

Frontend: Next.js + React
Backend: Node.js + Express
Database: PostgreSQL + pgvector
Embedding: Claude API (embeddings endpoint)
LLM: Claude API (messages endpoint)
Hosting: AWS Lightsail or Render

Simple. Proven. Scales.

Now go build something amazing.

Quick Reference: Key Code Snippets

Embedding and Storage

from anthropic import Anthropic

client = Anthropic()

def create_embeddings(texts):
    """Get embeddings from Claude."""
    response = client.messages.create(
        model="claude-3-5-sonnet-20241022",
        max_tokens=1024,
        system="You are an embedding service. Return ONLY JSON.",
        messages=[{
            "role": "user",
            "content": f"Get embeddings for: {texts}"
        }]
    )
    return response.content[0].text

PostgreSQL Hybrid Search

SELECT chunk_text, 
       embedding <=> query_embedding AS vector_distance,
       ts_rank(to_tsvector(chunk_text), 
               plainto_tsquery('search_term')) AS keyword_rank
FROM resume_chunks
WHERE resume_id = 'candidate-uuid'
  AND to_tsvector(chunk_text) @@ plainto_tsquery('search_term')
ORDER BY vector_distance ASC, keyword_rank DESC
LIMIT 5;

Scoring Algorithm

def score_candidate(candidate_skills, required_skills, weight=0.6):
    """Simple scoring: % of required skills matched."""
    matched = len([s for s in candidate_skills if s in required_skills])
    score = (matched / len(required_skills)) * 100
    return score

Hallucination Prevention

def safe_answer(query, chunks, confidence_threshold=0.7):
    """Answer only if confident."""
    response = client.messages.create(
        model="claude-3-5-sonnet-20241022",
        max_tokens=500,
        system="""
You MUST:
1. Answer ONLY using the provided chunks
2. Cite evidence
3. Return JSON with answer, confidence, evidence
""",
        messages=[{
            "role": "user",
            "content": f"""
Chunks: {chunks}
Question: {query}
"""
        }]
    )

    answer = json.loads(response.content[0].text)

    if answer['confidence'] < confidence_threshold:
        return "Uncertain - escalated to human"

    return answer['answer']

Final Thought

RAG isn't magic. It's engineering:

Clear data structures
Proven patterns
Careful measurement
Honest assessment

You've learned the foundations. Now use them.

Build something reliable.
Build something honest.
Build something useful.

You've got this.

Created for developers who want to understand how RAG actually works, not just use it as a black box.

Remember: Less noise, more action.

Appendix: Useful Resources

Tools We Discussed

PostgreSQL with pgvector - Local vector database
Claude API - Embeddings and LLM
Python - Implementation examples
Redis - Caching
Nginx - Load balancing

Testing Checklist

[ ] Embedding quality (check cosine distances)
[ ] Retrieval recall >90%
[ ] Retrieval precision >85%
[ ] Latency <2 seconds
[ ] Hallucination rate <5%
[ ] Cost tracking in place
[ ] Monitoring alerts configured
[ ] Data isolation verified (multi-tenant)
[ ] Cache hit rate >50%
[ ] User feedback collected

Architecture Checklist

[ ] Load balancer in front
[ ] Multiple API instances
[ ] Redis caching layer
[ ] PostgreSQL with indexes
[ ] Tenant isolation on every query
[ ] Error logging
[ ] Uptime monitoring
[ ] Graceful degradation
[ ] Security scanning
[ ] Backup strategy

Go build. Good luck. 🚀

Event-Driven Architecture: Uncle Explains Like You're Five 👦👨‍🦳

surajrkhonde — Sun, 14 Jun 2026 09:21:11 +0000

A conversation between Uncle (a backend architect) and Nephew (a curious developer) about events, publishers, subscribers, Redis Pub/Sub, RabbitMQ, and Kafka

The Beginning: What is an Event?

👦 Nephew: Uncle, I see words like "Redis Pub/Sub", "RabbitMQ", "Kafka" everywhere in job descriptions. What do they all do? Are they the same thing?

👨‍🦳 Uncle: (smiles) No, they're different. But before I confuse you with names, let me ask you something. Have you ever watched the news on TV?

👦 Nephew: Yes uncle, every morning!

👨‍🦳 Uncle: Perfect! So when the news channel says "Breaking News: India won the cricket match" - what happened?

👦 Nephew: Something important occurred... and they announced it!

👨‍🦳 Uncle: Exactly! That "something important occurred" is called an Event. In software, when something happens - like a user placing an order, a payment succeeding, or a file uploading - that's an event.

👦 Nephew: So event = something that happened?

👨‍🦳 Uncle: Yes! Think of it as news. When you place an order at Zomato, that's an event. When you get a payment notification from Google Pay, that's an event. When someone follows you on Instagram, that's an event.

👦 Nephew: Okay, I get it. But uncle, why is this "event" concept important? I can just write code directly, right?

👨‍🦳 Uncle: Ah! That's where the real story begins...

The Problem: Spaghetti Code

👨‍🦳 Uncle: Imagine you own a food delivery company. A customer places an order. Now, what all needs to happen?

👦 Nephew: Umm... save the order, send email confirmation, alert the restaurant?

👨‍🦳 Uncle: Good! Let me write the code without events:

function placeOrder(orderId) {
  saveOrder(orderId);
  sendEmailConfirmation(orderId);
  sendSMSNotification(orderId);
  notifyRestaurant(orderId);
  updateAnalyticsDashboard(orderId);
  addLoyaltyPoints(orderId);
  updateInventory(orderId);
}

Now, one day your boss says "Also send WhatsApp notification". What do you do?

👦 Nephew: Add another function call in the same code?

👨‍🦳 Uncle: Yes, but now the function becomes bigger. Tomorrow boss says "Also notify admin", you add more. Your Order Service becomes this giant blob of code that does 10 different things. It becomes:

Hard to maintain - Change one thing, something else breaks
Tightly coupled - Every service knows about every other service
Hard to scale - One service dying affects everything

This is called spaghetti code. Everyone is tangled together.

👦 Nephew: That sounds awful! So what's the solution?

👨‍🦳 Uncle: What if the Order Service just announces "Order Created" and doesn't care who's listening?

Enter: Publisher & Subscriber

👨‍🦳 Uncle: Remember the news channel? The channel broadcasts news. The viewers listen if they want. The channel doesn't know who's watching - could be 1 person or 1 million.

👦 Nephew: Yes, like YouTube!

👨‍🦳 Uncle: Perfect! YouTube is the Publisher - they create content. You are the Subscriber - you listen. The video upload is the Event.

Now in our food delivery system:

Order Service = Publisher (creates the event "Order Created")
Email Service, SMS Service, Analytics = Subscribers (listen to that event)

The Order Service just announces:

📢 "Order Created - Order #123"

And doesn't care who listens!

👦 Nephew: So if tomorrow boss wants WhatsApp notification, I just add another subscriber?

👨‍🦳 Uncle: Exactly! Order Service doesn't change. You just create a WhatsApp Service that subscribes to "Order Created" events. Beautiful decoupling!

👦 Nephew: Amazing! But uncle, how do they communicate? There must be something in the middle passing messages around?

👨‍🦳 Uncle: Yes! That's where Redis Pub/Sub comes in...

Redis Pub/Sub: The Simple Messenger

👨‍🦳 Uncle: Think of it like a WhatsApp group. You have a group called "Order Events". When someone posts a message, everyone in the group sees it.

👦 Nephew: Oh! So Order Service is like one person posting in the group, and others are reading?

👨‍🦳 Uncle: Exactly! Here's how it works:

Order Service (Publisher)
        |
        | Publishes to channel: "order-created"
        v
      Redis
        |
   -----+-----
   |    |    |
   v    v    v
Email  SMS  Analytics
Service Service Service
(Subscribers)

Each service subscribes to a "channel" - like joining a WhatsApp group.

Order Service publishes:

redis.publish('order-created', {
  orderId: 123,
  userId: 456,
  amount: 500
});

Email Service listens:

redis.subscribe('order-created', (message) => {
  sendEmail(message.userId);
});

👦 Nephew: So Redis is like the WhatsApp server?

👨‍🦳 Uncle: Exactly! Redis is the messenger sitting in the middle, broadcasting messages.

👦 Nephew: This seems perfect! Why would we need anything else?

👨‍🦳 Uncle: Ah, great question! Here comes the problem...

The Crash: Redis Pub/Sub's Dirty Secret

👨‍🦳 Uncle: Imagine it's 10:00 AM. Order Service publishes:

"Order Created - Order #123"

But at that exact moment, Email Service has crashed (network issue). Redis broadcasts the message. Email Service isn't listening.

Message is gone. Forever.

👦 Nephew: Oh no! So the customer never gets the confirmation email?

👨‍🦳 Uncle: Exactly! At 10:01 AM, Email Service comes back online. Can it ask Redis "Hey, what messages did I miss?"

👦 Nephew: Can't it?

👨‍🦳 Uncle: No! Redis says "Sorry, that message was 1 minute ago. I already sent it. I don't save messages."

Think of it like FM Radio:

Radio broadcasts live music
If you're listening: ✅ You hear it
If you're not listening: ❌ You miss it
Radio doesn't replay old songs

👦 Nephew: That's a problem for important stuff like payment notifications!

👨‍🦳 Uncle: Exactly! What if the event is "Payment Success"? We CANNOT lose that message. A customer should get confirmation even if they restart their phone.

For such critical events, we need something that:

Stores messages (doesn't lose them)
Retries (if service is down, deliver later)
Confirms (acknowledges delivery)

Redis Pub/Sub can't do any of this. And this pain gave birth to the next hero...

RabbitMQ: The Reliable Messenger

👦 Nephew: What can RabbitMQ do that Redis can't?

👨‍🦳 Uncle: Let me draw you a scenario. Same 10:00 AM order.

Order Service sends to RabbitMQ:

Order #123 Created

RabbitMQ says "Got it! I'm storing this message in a Queue."

Now, even if Email Service is down, the message stays in the queue.

When Email Service comes back online at 10:01 AM, RabbitMQ says:

"Hey! You have 1 message. Here it is."

Email Service reads and processes it.

👦 Nephew: So RabbitMQ is like a post office?

👨‍🦳 Uncle: Perfect analogy! The post office stores your letters even if the recipient isn't home. When they come back, they get the letter.

But there's a twist...

Queues & Workers

👨‍🦳 Uncle: In RabbitMQ, we don't have "subscribers listening". Instead, we have workers pulling messages.

Imagine a pizza shop:

Kitchen has a Queue of pizza orders written on paper
Worker 1 pulls an order: "Margherita Pizza"
Worker 1 makes it and says "ACK" (acknowledgement) - "I got it!"
Order is removed from queue
Worker 2 pulls next order: "Pepperoni Pizza"

This ensures:

No order is lost
No order is made twice
If Worker 1 crashes mid-making, the order goes back to queue for another worker

👦 Nephew: What does ACK mean?

👨‍🦳 Uncle: ACK = "Acknowledgement". It means "I processed this message, you can delete it from queue now."

Example:

// Email Service pulling messages
consumer.consume('order-created', async (message) => {
  const orderId = message.content;

  try {
    await sendEmail(orderId); // Process
    consumer.ack(message); // ✅ Success, delete from queue
  } catch (error) {
    consumer.nack(message); // ❌ Failed, put back in queue
    // Another worker will retry
  }
});

👦 Nephew: Oh! So if Email Service crashes before sending email, the message goes back to queue?

👨‍🦳 Uncle: Yes! Another Email Service instance picks it up. The message isn't lost.

👦 Nephew: So RabbitMQ is perfect now?

👨‍🦳 Uncle: Not quite... It has its own limitations...

RabbitMQ's Limitation: Queue is Consumed

👨‍🦳 Uncle: Here's the thing. In RabbitMQ, when a message is consumed and ACK'd, it's deleted forever.

Imagine you have:

Email Service
SMS Service
Analytics Service

All processing the same event "Order Created".

Email Service: ✅ Processed, ACK'd
SMS Service: ✅ Processed, ACK'd
Analytics Service: Hmm, I crashed. I never got to process it!

In RabbitMQ, the message is gone. Analytics loses the data.

👦 Nephew: So we can't have multiple independent consumers reading the same message?

👨‍🦳 Uncle: Not really. Each message is meant for ONE consumer.

RabbitMQ is good for:

Tasks: "Send this email", "Generate this PDF", "Resize this image"
Once done, forget about it

But it's bad for:

Events/Facts: "User Purchased", "Order Placed"
Where history matters
Where multiple services need to analyze

👦 Nephew: So what do we do?

👨‍🦳 Uncle: Here comes the king... Kafka.

Kafka: The Event Store

👨‍🦳 Uncle: Imagine instead of a trash can (like RabbitMQ), events are stored in a notebook that never gets thrown away.

Topic: orders (a notebook)

Order 101 ← Event 1 (stored forever)
Order 102 ← Event 2 (stored forever)
Order 103 ← Event 3 (stored forever)
Order 104 ← Event 4 (stored forever)

Now:

Email Service reads page 1, 2, 3
SMS Service reads page 1, 2, 3 (same pages!)
Analytics reads page 1, 2, 3 (same pages!)

The notebook never gets destroyed. Everyone can read whenever they want!

👦 Nephew: Can they reread the same page again?

👨‍🦳 Uncle: Yes! That's the magic of Kafka. Let me show you...

The Magic of Offsets & Replay

👨‍🦳 Uncle: Kafka numbers every event:

Offset 0 → Order 101
Offset 1 → Order 102
Offset 2 → Order 103
Offset 3 → Order 104

Analytics Service reads Offsets 0, 1, 2. Then it crashes.

It remembers: "I've read up to Offset 2."

When it comes back online, Kafka says: "Where did you leave off?"

Analytics: "Offset 2"

Kafka: "Here's Offset 3, 4, 5..." (only new ones)

No duplicates! No missed messages!

👦 Nephew: And the notebook never got destroyed?

👨‍🦳 Uncle: Never! It stays. This enables something magical called Replay.

New team arrives: "Recommendation Team". They need to analyze last 30 days of purchases.

In RabbitMQ: "Sorry, we already processed and deleted those messages."

In Kafka: "No problem! Start reading from Offset 0."

The Recommendation Team reads all 30 days of historical data. Done!

👦 Nephew: That's amazing! So Kafka is always better than RabbitMQ?

👨‍🦳 Uncle: It's different, not better. RabbitMQ is simpler if you just need task queues. Kafka is complex but powerful if you need event history.

But wait, there's more to Kafka...

The Scaling Problem: Partitions

👦 Nephew: Uncle, what if Amazon generates 10 million events per second? Can one notebook handle it?

👨‍🦳 Uncle: No! That's why Kafka invented Partitions.

Instead of:

1 notebook (one notebook is slow)

It creates:

10 notebooks (10 notebooks in parallel)

Example:

Topic: orders

Partition 0         Partition 1         Partition 2
------          ------          ------
Order 101       Order 102       Order 103
Order 105       Order 106       Order 104
Order 108       Order 109       Order 107

Events are distributed across partitions. Each partition can be on a different machine.

Partition 0 → Machine A (writing 3M events/sec)
Partition 1 → Machine B (writing 3M events/sec)
Partition 2 → Machine C (writing 3M events/sec)

Total: 9M events/sec (in parallel!)

👦 Nephew: So how does Kafka decide which event goes to which partition?

👨‍🦳 Uncle: It uses a Key. Example:

producer.send({
  topic: 'orders',
  key: userId,        // All events for same user go to same partition
  value: { orderId: 123 }
});

All events for User 101 go to the same partition. This ensures ordering.

👦 Nephew: Ordering? Why does that matter?

👨‍🦳 Uncle: Imagine User 101:

Logs in
Purchases iPhone
Logs out

If these go to different partitions, they might be processed out of order:

"Logs out" (done first)
"Logs in" (second)
"Purchases" (last)

That's wrong! The user hasn't even logged in yet when they purchase!

By sending all User 101 events to the same partition, they stay ordered:

Partition 2
  |
  v
Login (User 101)
Purchase (User 101)
Logout (User 101)

Always in order!

👦 Nephew: Brilliant! And what about multiple Email Services reading the same partition?

👨‍🦳 Uncle: Ah! Here's where Consumer Groups come in...

Consumer Groups: The Team

👨‍🦳 Uncle: Kafka says: "One partition can be read by only ONE consumer at a time, but multiple consumers can form a Consumer Group."

Think team of workers:

Topic: orders (3 partitions)

Consumer Group: Email Service
  - Worker A
  - Worker B
  - Worker C

Kafka assigns:
  - Worker A reads Partition 0
  - Worker B reads Partition 1
  - Worker C reads Partition 2

Now:

No duplicate processing (each partition owned by one worker)
Parallel processing (3 workers working simultaneously)
Load balanced (if one worker dies, another takes over)

👦 Nephew: What if Worker B crashes?

👨‍🦳 Uncle: Kafka detects the crash and rebalances:

Before:
  - Worker A → Partition 0
  - Worker B → Partition 1 (💥 CRASHED)
  - Worker C → Partition 2

After:
  - Worker A → Partition 0, 1 (rebalanced)
  - Worker C → Partition 2

Processing continues!

Redis vs RabbitMQ vs Kafka: The Comparison

👦 Nephew: Uncle, now I'm confused. When should I use what?

👨‍🦳 Uncle: Let me create a mental model for each:

Redis Pub/Sub

👨‍🦳 Uncle: Think FM Radio.

Radio Station broadcasts → Listeners hear (if tuned in)
                       → If not listening, message lost

Good for:

Live notifications ("User is typing...")
Live updates ("Stock price changed")
Realtime dashboards

Bad for:

Reliable message delivery
Message history

RabbitMQ

👨‍🦳 Uncle: Think Delivery Service.

Sender → Package → Delivery Boy → Receiver
         (stored temporarily in warehouse)

Once delivered and signed, package is gone.

Good for:

Task queues ("Send Email", "Generate PDF")
Work distribution
Simple message processing

Bad for:

Event history
Multiple independent consumers
Replay capability

Kafka

👨‍🦳 Uncle: Think Library Book.

Author writes → Book stored → Many readers can read
              (stays on shelf forever)

Anyone can read anytime. History preserved.

Good for:

Event history ("User purchased", "Payment completed")
Multiple independent services analyzing same events
Replay and timeline analysis
High-throughput event streaming

Bad for:

Simple task queues (too complex)
Fire-and-forget scenarios

👦 Nephew: Perfect! So they're for different use cases?

👨‍🦳 Uncle: Exactly! Choose based on your need:

Real-time, lose-if-missed? → Redis
Reliable task queue? → RabbitMQ
Event history & multiple consumers? → Kafka

Real World Example: E-Commerce Order

👦 Nephew: Can you give me a complete real-world example?

👨‍🦳 Uncle: Sure! Amazon customer purchases iPhone.

Event: "UserPurchased"

Using Kafka, multiple services independently process this:

Order Service publishes:
  Topic: user-events
  Partition: 0 (because all User #123 events go here)
  Event: { userId: 123, productId: iPhone, amount: 50000 }

        ↓

Kafka stores this forever

        ↓

Now multiple services can read:

Email Service:
  Offset: 0 → Send confirmation email

Analytics Service:
  Offset: 0 → Update dashboard, revenue metrics

Fraud Service:
  Offset: 0 → Check suspicious purchase pattern

Recommendation Service:
  Offset: 0 → Suggest related products

Inventory Service:
  Offset: 0 → Update stock count

Loyalty Service:
  Offset: 0 → Add reward points

All reading the same event independently. All storing their own offsets.

If Analytics crashes and comes back after 2 hours:

It reads from its last offset
Gets all events from the last 2 hours
Processes them

The event still exists on Kafka's Topic, unchanged.

👦 Nephew: This is like a perfect audit trail!

👨‍🦳 Uncle: Exactly! Netflix keeps Kafka events for 30 days. Any new team can analyze historical data. That's the power!

Key Takeaways: What to Remember

👦 Nephew: Uncle, so I should just remember...?

👨‍🦳 Uncle: Remember these mental models:

Event = Something happened (like news)
Publisher = One who creates events (like news channel)
Subscriber = One who listens to events (like viewer)
Redis Pub/Sub = FM Radio (instant, no storage)
RabbitMQ = Delivery Service (reliable, consumed/deleted)
Kafka = Library Books (permanent, many can read)
Partition = Multiple books handling millions of events in parallel
Consumer Group = Team of workers reading partitions
Offset = Page number, remember where you left off
Replay = Read from beginning again (only possible in Kafka)

👦 Nephew: This makes so much sense now!

👨‍🦳 Uncle: That's the goal! Once you understand the "why" of events, the tools become simple. They all solve the same problem (decoupling services) with different tradeoffs.

When to Use What: A Decision Tree

👦 Nephew: Any quick tips for choosing?

👨‍🦳 Uncle: Sure!

Do you need message history?
├─ YES → Use Kafka
├─ NO → Are multiple independent services reading the same message?
         ├─ YES → Use Kafka (trust me)
         ├─ NO → Do you need reliable delivery?
                  ├─ YES → Use RabbitMQ
                  ├─ NO → Use Redis Pub/Sub

Quick examples:

Chat app (new message notification) → Redis Pub/Sub
Job queue (send 1000 emails) → RabbitMQ
Order processing (email, analytics, fraud, inventory) → Kafka
Live dashboard updates → Redis Pub/Sub
Banking (payment settlement) → RabbitMQ or Kafka
User analytics pipeline → Kafka

Final Word

👨‍🦳 Uncle: Remember, it all starts with understanding the problem. None of these tools matter if you don't know:

What events matter to your business?
Who needs to know about them?
Can we afford to lose a message?
Do we need history?
How many events per second?

Once you answer these, the choice becomes obvious.

👦 Nephew: Thank you uncle! This was super clear!

👨‍🦳 Uncle: You're welcome! Now go build something with events! Start simple with Redis Pub/Sub, then graduate to RabbitMQ, then maybe Kafka when you need it.

👦 Nephew: Will do, uncle! Less noise, more action! 😄

👨‍🦳 Uncle: That's my boy! 👏

Happy event-driven coding!

Dawa Saathi: I finally finished the part of my medicine bot that actually mattered

surajrkhonde — Fri, 05 Jun 2026 12:47:51 +0000

AI medicine-awareness bot worked — but only in English, the one language most of the people I built it for can't read. Here's the problem, the one-day sprint to fix it, and how GitHub Copilot got me through it.

GitHub Finish-Up-A-Thon submission. I shipped this project months ago, but it was never truly finished — it was missing the one feature that decided whether a real person in my own community could use it. This challenge was the push that made me sit down and close that gap in a single night.

The problem we face

In India, you can walk into most pharmacies and buy prescription drugs without a prescription. We've quietly normalized something genuinely dangerous, and the research is blunt about it.

A simulated-patient study in Bengaluru sent two researchers into 261 pharmacies with fake symptoms. Antibiotics were handed over without any prescription at roughly two-thirds (66.7%) of them. Not a single pharmacy warned about side effects. Only about one in five even mentioned that a doctor's prescription was needed. The "guidance" most people walked out with was "take it twice a day" — and nothing else. (study)

Here is the part that kept me up at night: it's not because people are careless. The shopkeeper isn't a doctor. The label is printed in tiny English. A proper consultation costs time and money many families simply don't have. So people take what they're handed — and hope it's fine.

Why it's important

This isn't a small inconvenience. Taking the wrong medicine, or the right medicine the wrong way, is one of the biggest drivers of antibiotic resistance — the slow disaster where medicines stop working.

The landmark 2019 Lancet GRAM study estimated 1.27 million deaths worldwide were directly attributable to drug-resistant infections in a single year. In India alone, an estimated 297,000 deaths were directly attributable to AMR, and about 1.04 million were associated with it. (Lancet GRAM 2019, India figures)

I'm not a doctor and I can't change how medicines are sold. But I'm a developer, and I kept circling one small thing I might actually be able to build:

What if a person could at least understand what they're holding — before they take it?

So I built Dawa Saathi. You send a photo of a medicine label, and it reads it with AI and explains, in plain language: what the medicine is commonly used for, its common and serious side effects, and the key warnings (pregnancy, liver, kidney, alcohol, children, the elderly). It does not diagnose. It does not give a dose. It never says a medicine is "safe to take." Because with medicine, a confident wrong answer is more dangerous than no answer at all.

The gap I shipped with (the "before")

Here's the uncomfortable truth about the first version: it only spoke English.

I built Dawa Saathi for people in rural and small-town India — the elderly, people with little formal schooling, people who grew up reading Kannada, Hindi, or Marathi, not English. I come from that world myself; I went to a Marathi-medium government school in a village before I ever wrote a line of code. And yet the tool I built for those people answered every single question in fluent English.

It worked. The AI read labels accurately, the safety guardrails held, caching and rate limits were solid. But it was "finished" the way a bridge is finished when it stops halfway across the river. The one feature that decided whether a real person in my community could actually use it — language — was the thing I'd left undone.

The Finish-Up-A-Thon prompt ("revive a project you started but never completed") landed on exactly that nerve.

One day, under pressure

I gave myself the length of the challenge to fix it, and it came down to a single overnight session — about six hours straight.

That kind of pressure cuts both ways. On one hand, a deadline is the best forcing function I know; the project had sat "good enough" for months precisely because nothing forced me to finish it. On the other hand, language support is not a cosmetic change — it touches the welcome flow, every button, every error message, the medicine response itself, and the storage layer. At 2 a.m., with a medical app where a sloppy mistake isn't just a bug, that's a lot of surface area to get right while tired.

There were stretches where I went down the wrong path — I first started building a translation dictionary, realized it would silently drop medical words, and had to back it out and rethink the whole approach. Hours like that are where solo projects usually die.

How GitHub Copilot helped me through it

I enabled GitHub Copilot specifically for this sprint, and I'll be honest about exactly what it did — good and bad — because that's the point of this challenge.

Where it genuinely sped me up: the repetitive scaffolding. The new i18n layer has dozens of string entries across two languages; the Telegram handler needed every reply rewired to read from it; the callback router needed a whole new branch for language selection. From a comment and the surrounding code, Copilot autocompleted big chunks of that pattern-heavy work. Tasks that would have been an hour of careful typing became a few minutes of accepting and fixing.

Where it helped me think under fatigue: when the bot stubbornly kept replying in English even after I'd changed the code, Copilot Chat helped me reason through the deploy pipeline and spot that I was running an old compiled build (npm start runs dist/, not src/ — I'd forgotten to rebuild). At 3 a.m., that saved me from chasing a ghost for an hour. It also caught a sloppy callback branch where my "language" and "purpose" buttons could collide.

Where I had to stay in charge: Copilot is confidently wrong sometimes. It suggested code that looked right but quietly dropped a safety guardrail, and once invented an API shape that didn't exist. In a normal app that's an annoyance; in a medical app, an unreviewed AI suggestion is exactly the "confident wrong answer" my whole project exists to prevent. So I read every line it gave me. The speed was real — but the judgment stayed mine.

Honestly, Copilot didn't replace that six-hour night. It made the difference between finishing it and giving up around hour three.

The major update: what actually changed

The first thing the bot now does on /start is ask which language you're comfortable in:

🌐 Please choose your language
ದಯವಿಟ್ಟು ನಿಮ್ಮ ಭಾಷೆಯನ್ನು ಆಯ್ಕೆಮಾಡಿ
[ English ]   [ ಕನ್ನಡ ]

Tap a button, and everything afterward — welcome, buttons, questions, and the full medicine explanation — comes back in that language. The choice is remembered, and you can change it anytime with /language.

Under the hood, I split the text into two kinds and treated them very differently:

Fixed UI text (buttons, labels, errors, and the all-important medical disclaimer) lives in a small i18n table. It's instant, costs zero tokens, and — critical for health content — the disclaimer in each language can be reviewed word-for-word instead of machine-guessed.
The medicine explanation is generated by the LLM directly in the user's language. I deliberately did not translate the English afterward. The awareness model writes its answer in Kannada from the start, while keeping JSON keys in English (so the parser never changes) and drug names in Latin script (so "Tranexamic Acid 500mg" is never mangled). Every safety rule — no dosage, no diagnosis, no "safe to take," always "consult a doctor" — is repeated for the target language.

That choice means one model call instead of two, no brittle dictionary dropping medical words, and a reply that reads like it was written for a Kannada speaker rather than awkwardly converted into Kannada. Each language is cached separately, so the next person who scans the same strip gets an instant answer.

What this means for people in Karnataka

This is the part I actually care about.

Kannada is the first language of around 44 million people, and roughly two-thirds of Karnataka's ~68 million residents speak it natively (2011 census / overview). Meanwhile, by census-based estimates only about one in ten Indians use English at all — and a far smaller share read it comfortably enough to parse a medicine label or a list of side effects.

Put those two facts together and the old version's flaw is stark: an English-only health tool quietly excluded the majority of the exact people it was meant to help. The grandmother in a Belagavi village holding a strip of tablets, the daily-wage worker in Kalaburagi who left school early — they could open the bot, but they couldn't read it.

With this update, that same grandmother can now ask in Kannada and get an answer she can actually understand — what the medicine is for, what to watch out for, and a clear, repeated nudge to see a doctor. It doesn't make the bot a doctor. It makes the bot legible to the people who need it most. For a medicine-awareness tool, legibility in the user's own language isn't a feature — it's the whole job.

The before → after, at a glance

	Before	After
Languages	English only	English + ಕನ್ನಡ (more in ~2 lines)
First interaction	Dumped an English welcome	Asks your language, then speaks it
Who could actually use it	English readers (~1 in 10)	Tens of millions of Kannada speakers
Translation approach	—	LLM generates in-language; i18n for fixed UI
Reachable by my own family	No	Yes

Future plan

Hindi and Marathi next — the architecture already supports them; each new language is roughly a two-line change plus a reviewed disclaimer.
Voice output, for users who can't read at all — arguably the people who need this most.
Reviewed translations of the fixed disclaimer with help from native-speaker volunteers, so the safety text is never just my best guess.
Pharmacy and NGO partnerships — the long-term home for something like this is alongside health workers and CSR programs, not as a lone bot.

The honest impact

I want to be careful not to overclaim. This won't fix the bigger problem. That needs doctors, pharmacists, and real regulation. One Telegram bot does not move a national health statistic.

But the gap it closes is real and specific: a person can now understand, in their own language, what they're about to put in their body — before they take it. If that makes even a few people pause and think, or take one question to a doctor they'd otherwise have skipped, it was worth the night.

If you work in public health or medicine, I'd genuinely value your guidance on doing this responsibly.

Try it / code

Bot: TELEGRAM_BOT_LINK — https://t.me/dawasaathi_bot
Source: https://github.com/SurajKhonde/Dawa_saathi_mediAssist (MIT licensed)
Demo:
- Before_challenge:-https://drive.google.com/file/d/1UW5HsSfBZKWx93P-flJ9sUKVrg03JflS/view?usp=sharing
- After_challenge :- https://drive.google.com/file/d/1pDaeMIelnb9ymHF9UygBmR4DY6lT7Vh1/view?usp=sharing

Built solo. Not a substitute for a doctor — by design.

Sources

Non-prescription antimicrobial dispensing in Bangalore pharmacies (261-pharmacy simulated-client study): https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4676185/
Global burden of bacterial AMR, 2019 (Lancet, GRAM): https://www.thelancet.com/journals/lancet/article/PIIS0140-6736(21)02724-0/fulltext
India-specific AMR mortality (297,000 attributable / 1,042,500 associated, 2019): https://journals.lww.com/pmrr/fulltext/2024/01040/antimicrobial_resistance__an_emerging_global.1.aspx
Kannada speakers and Karnataka share (2011 census overview): https://en.wikipedia.org/wiki/Kannada

Uncle, I burned ₹1000 in 4 runs — what did I do wrong?

surajrkhonde — Fri, 05 Jun 2026 07:17:23 +0000

"An uncle-nephew conversation about the new line item on every developer's bill: LLM tokens."

Uncle, I burned ₹1000 in 4 runs

An uncle-nephew conversation on why some AI calls cost paise, and others cost thousands.

👦 Nephew: Uncle, I have a problem. I put ₹1000 of my own money into the AI API this month and it finished in four runs. I don't even know what happened.

👨‍🦳 Uncle: Sit down, beta. Drink chai first. Then tell me everything you built this month, from the beginning.

👦 Nephew: Okay. I built four things. Three of them were fine. One of them ate the whole month.

👨‍🦳 Uncle: Good. We'll go one by one. Start with the easy one.

Project one: image text extraction

👦 Nephew: First one was simple. User uploads a photo — invoice, prescription, screenshot, whatever. I send it to the vision model. It returns the text. Done.

👨‍🦳 Uncle: How many calls per user?

👦 Nephew: One. Maybe two if the image is bad and they retry.

👨‍🦳 Uncle: And the cost?

👦 Nephew: Honestly, kaka, almost nothing. A few paise per image. I scanned around 200 images during testing and barely touched ₹50.

👨‍🦳 Uncle: Now stop and think. Why was it so cheap?

👦 Nephew: Aha… because it's just one image in, some text out. One round trip.

👨‍🦳 Uncle: Exactly. One input, one output, one transaction. Like sending one SMS. You know what you're paying before you press send. Keep this model in your head. It's the cheapest pattern that exists.

Project two: a customer support bot

👦 Nephew: Second one was a support bot for a client. User types a question. Bot replies.

👨‍🦳 Uncle: Where does the bot get its answers from?

👦 Nephew: I wrote a system prompt — around 800 words — explaining the product, common questions, tone of voice. Every user message goes in with that prompt. Bot replies.

👨‍🦳 Uncle: And the cost?

👦 Nephew: A bit higher than OCR. Each conversation was maybe ₹2 to ₹3. Total for the month, around ₹200 across hundreds of conversations.

👨‍🦳 Uncle: Tell me — why is this one costlier than the image project?

👦 Nephew: (thinking) …because the system prompt is going every time?

👨‍🦳 Uncle: Yes. You're paying for the same 800 words of instructions on every single message. The user pays for one message, you pay for 800 + their message. But conversations are short, so it doesn't get out of hand.

👦 Nephew: And the bot replies are also small — 100, 150 words.

👨‍🦳 Uncle: Small input, small output, slightly bigger envelope than OCR. Still safe. Now the next project.

Project three: document Q&A with RAG

👦 Nephew: Third one was a document Q&A. User uploads a 50-page PDF, asks questions, gets answers from inside the document.

👨‍🦳 Uncle: Smart. How did you handle the document?

👦 Nephew: I chunked the PDF, embedded each chunk, stored them in a vector database. When the user asks, I pull the top 5 relevant chunks and send them with the question to the LLM.

👨‍🦳 Uncle: Beta, that's actually a sensible design. Cost?

👦 Nephew: ₹3 to ₹4 per question. The project as a whole cost me around ₹300 across some testing and a few real users.

👨‍🦳 Uncle: Why higher than the support bot?

👦 Nephew: Because each question carries 5 chunks of document context. Maybe 3000 words of input on every call.

👨‍🦳 Uncle: Correct. Bigger input on every call. But notice what you didn't do — you didn't send the entire 50-page PDF every time. You sent only the 5 most relevant pieces. That's a smart RAG.

👦 Nephew: What's a dumb one?

👨‍🦳 Uncle: A dumb RAG sends the whole 50-page document every time the user asks anything. Some people actually build this. They pay for 50 pages of input when they could have paid for 5. Don't be that person.

👦 Nephew: Noted.

Project four: the disaster

👨‍🦳 Uncle: Now tell me about the project that ate the rest of your money.

👦 Nephew: (sighs) Job search agent. Idea was — user uploads their resume, my agent searches the web for matching jobs, returns the top ones with reasoning. Like a personal recruiter.

👨‍🦳 Uncle: How does the agent search?

👦 Nephew: It has a web search tool. Searches job boards, company career pages — whatever comes up. Clicks into postings, reads them, decides if they match the resume, then either suggests them or searches again with refined keywords.

👨‍🦳 Uncle: How many of these search-and-read cycles per user?

👦 Nephew: I didn't think about it carefully. The agent does around 5 searches. Each search returns 10 results. Then it fetches the most promising 3 or 4 pages in full. Reasons over them. Maybe searches again.

👨‍🦳 Uncle: Cost per user?

👦 Nephew: …₹200 to ₹250 per user, kaka.

👨‍🦳 Uncle: (silence)

👦 Nephew: I tested it four times to debug. ₹1000 gone. And it didn't even get anyone a job. It just suggested some.

Where the money actually went

👨‍🦳 Uncle: Beta, listen carefully now. I want you to understand what really happened, because it will change how you think about every AI project from now on.

👦 Nephew: Tell me.

👨‍🦳 Uncle: Look at the four projects we discussed.

OCR:                [image] → [text]                          one call
Support bot:        [prompt + query] → [answer]               one call
RAG:                [prompt + chunks + query] → [answer]      one call
Web search agent:   [prompt + query] → [search results]
                  → [prompt + results] → [decide to fetch]
                  → [prompt + results + FULL PAGE] → [decide next]
                  → [prompt + results + page + ANOTHER PAGE] → ...
                  → and on, and on

👦 Nephew: Oh… the last one is a loop.

👨‍🦳 Uncle: Yes. And here is the part that breaks beginners. Every call inside that loop includes everything that came before it.

👦 Nephew: Wait, what?

👨‍🦳 Uncle: When the agent makes its second decision, it must see what the first search returned. So the second call carries the first search's results. When it makes the third decision, it must see the first two searches plus whatever pages were fetched in between. By the fifth step, the input on each call has grown to thousands and thousands of words.

👦 Nephew: So the context keeps stacking.

👨‍🦳 Uncle: And web pages are huge. One stripped-down job posting page is easily 2000 to 5000 words. Multiply by 3 or 4 pages, your agent is reasoning over 15,000 to 20,000 words on every step. Five steps of looping — you're at 80,000+ words of input billed for one user session.

👦 Nephew: Compared to OCR…

👨‍🦳 Uncle: Compared to OCR, your web search agent cost you roughly 400 times more per user. For the same kind of API.

👦 Nephew: (quietly) I see.

The one rule to remember

👨‍🦳 Uncle: Pick up your pen. Write this down. Stick it above your laptop.

The cost of an LLM application is not the cost of one call.

It's the cost of one call × how many times you call × how big the input has grown by then.

👨‍🦳 Uncle: Three things multiply together. Most beginner cost disasters come from forgetting one of them.

👦 Nephew: Which one bit me?

👨‍🦳 Uncle: All three at once. Many calls because of the agent loop. Growing input because of accumulated page content. And each call seemed cheap when you tested in isolation, so you never saw the multiplication coming.

👦 Nephew: Kaka, this is like electricity. Each appliance seems small. The bill at the end of the month tells the truth.

👨‍🦳 Uncle: Yes! Exactly that. AI tokens are electricity, not water connection. Variable, not fixed. Some operations are fans. Some are air conditioners running all day.

How to build the same agent without going broke

👦 Nephew: So how do I build a job search agent that doesn't bankrupt me?

👨‍🦳 Uncle: I'll give you five rules. Learn them properly.

Rule 1: Match the model to the job

👨‍🦳 Uncle: Not every step needs your most expensive model. There's a hierarchy.

👦 Nephew: Like what?

👨‍🦳 Uncle: Extracting text from an image? Use a smaller, cheaper vision model. The task is mechanical. Classifying whether a job posting matches a resume? A small model is more than enough. Writing the final personalized recommendation letter to the user? Now use your best model. That's the part where quality is visible.

👦 Nephew: I was using the smartest, most expensive model for every step.

👨‍🦳 Uncle: I know. Most people do. It's like using a JCB to plant a tulsi plant.

Rule 2: Don't pass full pages — pass extracts

👨‍🦳 Uncle: If your agent fetches a 5000-word page, do not shove all 5000 words into the next reasoning step.

👦 Nephew: Then what?

👨‍🦳 Uncle: Run a cheap model first to pull only the relevant 200 words. Pass those 200 words to your expensive reasoning model. Two-stage processing. You can cut 80–90% of cost this way.

Bad:    [agent — best model] reads 5000-word page → decides
Good:   [extractor — cheap model] pulls 200 relevant words
     → [agent — best model] reads 200 words → decides

Rule 3: Cap the loop

👨‍🦳 Uncle: Set a maximum number of iterations on your agent. Three searches. Four. Not "keep going until satisfied."

👦 Nephew: Why?

👨‍🦳 Uncle: Because an agent with no cap will happily run forever if you let it. Polite, hard-working, expensive servants need supervision.

Rule 4: Cache like your wallet depends on it

👨‍🦳 Uncle: If two users have similar resumes, you're searching for the same jobs and reading the same pages. Cache the search results. Cache the extracted summaries. Cache the model's interpretations.

👦 Nephew: Disk is cheaper than tokens?

👨‍🦳 Uncle: Disk is cheaper than tokens by a factor of a thousand. Always.

Rule 5: Use specialized APIs, not generic crawling

👨‍🦳 Uncle: This was the biggest mistake in your job project.

👦 Nephew: Tell me.

👨‍🦳 Uncle: You were doing general web search and crawling job board HTML. That is the most expensive path that exists. Job boards have APIs. LinkedIn, Naukri, Indeed, Glassdoor — many of them have official APIs or structured data feeds.

👦 Nephew: Even paid?

👨‍🦳 Uncle: Even paid. Pay ₹1000 once for an API subscription instead of ₹250 per user for crawling. The math turns positive at the fourth user.

👦 Nephew: I thought building my own crawler was the clever, free way.

👨‍🦳 Uncle: It was the expensive way wearing a free costume.

The bigger lesson

👦 Nephew: Kaka, I think I've been treating the AI API like a database. Like a fixed cost.

👨‍🦳 Uncle: And what is it actually?

👦 Nephew: More like electricity. The more you do, the more it draws. Some operations draw a lot more than others.

👨‍🦳 Uncle: Now you're thinking like an engineer, not just a developer.

👦 Nephew: What's the difference?

👨‍🦳 Uncle: A developer asks, "does this work?" An engineer asks, "does this work, and what does it cost to run for ten thousand users?" Both questions matter. But only one of them will tell you whether your product survives past the third month.

The math before you build

👦 Nephew: So before I build my next AI project, what do I do?

👨‍🦳 Uncle: Estimate four numbers for your worst-case user.

👦 Nephew: Tell me.

👨‍🦳 Uncle:

How many model calls per session.

How big the input is on each call (in words — roughly 1 word ≈ 1.3 tokens).

How big the output is on each call.

What you're paying per million input and per million output tokens for your chosen model.

👨‍🦳 Uncle: Multiply them out. If the per-user cost is more than what the user pays you — or more than you can absorb as a free tier — you don't have a product. You have a charity that will close in two months.

👦 Nephew: Or worse, a bill that goes to my father.

👨‍🦳 Uncle: (laughs) Don't make me explain this conversation to him.

A short summary for the next nephew

👦 Nephew: Kaka, can you give me a small table I can keep in my notes?

👨‍🦳 Uncle: Take this.

Project type	Pattern	Cost behaviour
Image OCR	One image → one output	Cheap, predictable
Support bot	Prompt + question → answer	Cheap, mildly accumulating
Document RAG	Retrieved chunks + question	Moderate, scales with chunk size
Agent + web search	Many loops with growing context	Expensive, scales explosively

👨‍🦳 Uncle: The hierarchy is not about technology being "advanced" or "basic." It is about how many tokens flow through the model per user, per session. Some patterns naturally produce a small flow. Others produce a flood.

👦 Nephew: Know which one I'm building. Do the math first. Pick the right model for each step. Cache aggressively.

👨‍🦳 Uncle: And one last thing.

👦 Nephew: What?

👨‍🦳 Uncle: Next time, build a small calculator before you build the agent. One spreadsheet cell would have saved you a thousand rupees.

👦 Nephew: (laughs) Noted, kaka.

If you found this useful, I write about backend engineering, system design, and these kinds of practical lessons from real projects. Less noise, more action.

— until the next nephew project goes sideways.

Long ChatGPT chats kept breaking my copy-paste. So I built an exporter.

surajrkhonde — Fri, 05 Jun 2026 06:01:08 +0000

A free Chrome extension I built to save full ChatGPT conversations as Markdown in one click — because the existing ways were genuinely painful.

tags: chrome, productivity, opensource, chatgpt

If you've ever had one of those long ChatGPT conversations — the kind where you spent half a day going deep into system design, or debugging something tricky, or learning a new concept end to end — you've probably run into this moment:

You finish. You realize the chat is gold. You want to save it.

You hit Ctrl + A.

Your browser freezes for ten seconds.

You finally manage to copy. You paste it into a doc. Half the formatting is gone. Code blocks are mangled. The structure that made the conversation useful is just… not there anymore.

That's where this story starts.

The problem (and why it's worse than it sounds)

I learn a lot from ChatGPT. Not in a passive "ask one question, move on" way — I have multi-day conversations going on system design, RabbitMQ internals, architecture decisions, debugging sessions. Some of my chats have 800+ messages.

These conversations are genuinely valuable. They're notes I keep coming back to. They're the work.

But getting them out of ChatGPT is shockingly bad:

Copy-paste: freezes the browser on long chats, mangles formatting
Screenshot: useless if you ever need to search or re-read it
ChatGPT's own export feature: emails you a giant ZIP… sometimes days later… of everything you've ever written. No per-chat option.
Manual cleanup: spending an hour formatting one conversation into Markdown is not what I want to be doing at 11 PM

I checked if a tool already existed. Some did — most were either broken on long chats, badly designed, full of trackers, or only saved the messages currently visible on screen (which is a fraction of a long conversation, because ChatGPT only keeps the visible messages loaded in memory).

So I built my own.

What it actually does

ChatGPT Exporter is a Chrome extension that does one thing:

Save your entire ChatGPT conversation as a clean Markdown file in one click.

That's it. No account, no signup, no upload, no cloud. You click a button, your file downloads.

It has three modes depending on the situation:

1. Export Full Chat — the main one. Works on chats of any length, including ones with thousands of messages. One click, full conversation, done.

2. Export Visible — when you only want to save the part you're looking at right now.

3. Scroll Capture — a fallback that captures messages as you scroll through the chat. Useful if the first mode ever hiccups.

The output is clean Markdown with proper headings, code blocks (with the right language), bold, italics, links, lists — everything that made the conversation structured stays structured.

Who it's for

It started as a tool for me, but it turns out a lot of people have this problem:

Code learners taking notes from long debugging or learning sessions
Writers and researchers preserving drafts and idea threads
Developers archiving architectural discussions they want to reference later
Anyone who realized too late that an important conversation got buried in their chat history

If you've ever thought "I should save that conversation somewhere I can actually find it" and then didn't, because the friction was too high — this is for you.

Try it

🔗 Chrome Web Store:
https://chromewebstore.google.com/detail/pmkhjmgnkohjdgninjfkdcegfkmnkfca

📦 Source code: https://github.com/SurajKhonde/chatgpt-exporter

It's open source under MIT. Free forever. No ads, no tracking, no account.

What I learned building it

A few things I picked up that might be useful to anyone building browser extensions:

Manifest V3 is much stricter than V2. Permissions get audited automatically — request only what you actually use, or your submission gets rejected. (Ask me how I know.)
DOM virtualization is a real problem. ChatGPT only mounts the messages currently near the viewport, so naively scraping the page misses 90% of a long conversation. Solving this properly took more iteration than I expected.
Stable IDs matter. Every message in ChatGPT has a unique ID — using that as the dedup key (instead of position) is the difference between a working exporter and a broken one.
Shipping a Chrome extension is a process. Icons, screenshots, promo tiles, privacy practices, justifications for every permission… expect to spend almost as much time on the store listing as the code.

What's next

A few things I'd like to add:

PDF export (Markdown → PDF in one click)
JSON export (raw conversation tree, for people who want to do their own processing)
Batch export — saving multiple conversations at once

If you have ideas, PRs welcome on GitHub.

One last thing

If you've also been losing important conversations in your ChatGPT history, give this a try. And if it saves you time, drop a ⭐ on the repo or share this with someone who needs it.

Built out of a real problem I face daily. Hope it helps you too.

— Suraj