DEV Community

Cover image for Building a RAG Pipeline: Chunking, Embeddings, Retrieval, Generation
Prakash Tank
Prakash Tank

Posted on Originally published at prakashtank.com

Building a RAG Pipeline: Chunking, Embeddings, Retrieval, Generation

Understanding Retrieval-Augmented Generation (RAG) conceptually is straightforward. Building a pipeline that returns fast, relevant answers over a large document collection is not. A naive implementation often breaks context at the wrong boundaries, retrieves the wrong passages, or spends more than expected on embedding and generation APIs.

This guide walks through eighteen stages of a production RAG pipeline: document parsing, structure-aware chunking, batch embeddings, vector storage and nearest-neighbor search, cross-encoder re-ranking, grounded prompt generation, document lifecycle invalidation, hybrid keyword search, automated chunk evaluators, multi-modal ingestion, database maintenance, disaster recovery, multi-tenant isolation, context compression, hash-based incremental re-indexing, audit logging, custom re-ranker training, and background ingestion workers. If you want the product-level picture first, see What is RAG? A Practical Guide for SaaS Product Teams.

Stage 1: Document ingestion and parser selection

Before you split documents into searchable fragments, the ingestion pipeline has to extract clean plain text from the files you actually receive. Those files are rarely uniform: PDF reports, Microsoft Word documents, Markdown exports, HTML documentation, and scanned images all show up in the same queue. A naive string extract often leaves garbage characters, broken tables, or missing headings, and that noise follows the text into every later embedding.

For PDFs, use a layout-aware parser such as PyMuPDF, Unstructured.io, or Apache Tika rather than a regex scrape. Layout-aware parsing keeps reading order across multiple columns, strips running headers and footers, and can turn tables into a clean Markdown representation. Getting the input text clean at this stage is usually the highest-leverage step for later retrieval quality.

Add pre-processing filters as well: strip non-printable control characters, standardize Unicode, and remove repetitive running header text. Cleaner text makes sentence-boundary detection and the resulting vectors more precise.

Stage 2: Structure-aware chunking strategies

Raw document text should not be embedded as one massive block. Split it into chunks: self-contained text segments that still represent a coherent thought.

The chunking strategy you choose has a direct effect on retrieval quality. Three common approaches:

  • Fixed-size chunking with overlap. A starting point is to split about every 512 tokens with a 10% (around 50-token) overlap. The overlap is there so a sentence that sits on a boundary is not cut in half and lost to both sides.
  • Structure-aware chunking (Markdown/HTML). Split along heading tags (# H1, ## H2, <h3>). That keeps a title with the paragraphs that belong to it.
  • Sentence-boundary chunking. Use a natural-language toolkit such as spaCy or NLTK to split at sentence boundaries rather than at an arbitrary character index.

Attach metadata to every chunk record: document_id, source_url, section_heading, and chunk_index. Those fields let a later vector query restrict results to a user account, a workspace, or a permission set, which matters as soon as more than one tenant shares the index.

When you pick a size, balance context against granularity. Very small chunks (for example, 50 words) often lack enough context for the model to use. Very large chunks (for example, 2,000 words) dilute the similarity signal. A range of roughly 250 to 500 tokens with a 10% sliding overlap is a reasonable place to start for technical documentation, then adjust from evaluation, not from a fixed rule.

Stage 3: Batch embedding generation

Once the text is chunked, each chunk has to become a numerical embedding. Do not call the embedding API once per chunk inside a tight loop. That adds a network round trip to every row and will run into provider rate limits quickly.

Process chunks in batches. A starting range of 100 to 500 inputs per request works with APIs such as OpenAI's text-embedding-3-small, or with a local model through Python's sentence-transformers. Check the provider's batch and rate limits and tune from there.

// Example: Batch Embedding Ingestion in PHP/Laravel
$chunks = Chunk::whereNull('embedding')->take(100)->get();

if ($chunks->isNotEmpty()) {
    $response = $openAI->embeddings()->create([
        'model' => 'text-embedding-3-small',
        'input' => $chunks->pluck('content')->toArray(),
    ]);

    foreach ($response->embeddings as $index => $item) {
        $chunks[$index]->update([
            'embedding' => '[' . implode(',', $item->embedding) . ']'
        ]);
    }
}
Enter fullscreen mode Exit fullscreen mode

Batching cuts the number of HTTP calls, so a background ingestion worker can move through a large queue without spending most of its time waiting on the network.

Stage 4: Vector storage and nearest-neighbor search in pgvector

You do not need a separate vector database on day one. Store the vectors in PostgreSQL with the pgvector extension, and you can join search results to the rest of your application tables in a single SQL query. For a comparison of pgvector with hosted options, see Vector Databases for Product Engineers: Pinecone, Chroma, and pgvector.

-- Migration: Enable pgvector extension and create index
CREATE EXTENSION IF NOT EXISTS vector;

CREATE TABLE document_chunks (
    id BIGSERIAL PRIMARY KEY,
    document_id BIGINT REFERENCES documents(id) ON DELETE CASCADE,
    content TEXT NOT NULL,
    source_url VARCHAR(255),
    section_heading VARCHAR(255),
    embedding vector(1536), -- 1536 dimensions for text-embedding-3-small
    created_at TIMESTAMP WITH TIME ZONE DEFAULT CURRENT_TIMESTAMP
);

-- Create HNSW index for Approximate Nearest Neighbor (ANN) search
CREATE INDEX idx_chunks_embedding ON document_chunks
USING hnsw (embedding vector_cosine_ops);
Enter fullscreen mode Exit fullscreen mode


sql

Executing a cosine similarity search in SQL

-- Finds top 5 most semantically similar chunks to a user query vector
SELECT id, content, source_url, section_heading,
       1 - (embedding <=> '[0.012, -0.045, ...]'::vector) AS similarity_score
FROM document_chunks
WHERE 1 - (embedding <=> '[0.012, -0.045, ...]'::vector) > 0.70 -- example threshold, not a rule
ORDER BY embedding <=> '[0.012, -0.045, ...]'::vector ASC
LIMIT 5;
Enter fullscreen mode Exit fullscreen mode

Add tenant or workspace filters to this query before you rely on it in a product; Stage 13 covers that isolation in more detail. The 0.70 similarity cutoff above is only an example. Pick a threshold from your own evaluation set, or skip the cutoff and take a top-k list into the re-ranker in Stage 5.

Stage 5: Re-ranking results with cross-encoders

Vector search (bi-encoder similarity) is fast, but it sometimes returns chunks that share similar wording without actually answering the question. A re-ranking step improves precision by scoring the query and each candidate together.

A cross-encoder re-ranker — for example Cohere's Rerank API, or an open-source model such as cross-encoder/ms-marco-MiniLM-L-6-v2 — can take the first vector results and score each one against the query. A common pattern is: retrieve a broader set with pgvector (for example, the top 20), re-rank with the cross-encoder, then send a shortlist (for example, the top 4) into the prompt. Those depths are starting points; Stage 17 covers adapting the re-ranker when your domain is more specialized.

Stage 6: Assembling grounded generation prompts

With the re-ranked chunks in hand, build the prompt the model will see. Give it explicit rules: answer only from the provided context, cite sources, and say when the context is not enough. Prompt Engineering for Production Apps (Not ChatGPT Tricks) goes further on this kind of production prompt.

$contextString = collect($topChunks)->map(function ($chunk, $index) {
    return "[Source " . ($index + 1) . " - " . $chunk->section_heading . "]:\n" . $chunk->content;
})->implode("\n\n---\n\n");

$systemPrompt = <<<EOT
You are an expert technical support AI for Acme Software.
Your task is to answer the user's question using ONLY the provided document context below.

STRICT RULES:
1. Grounding: Answer strictly using facts found in the context. If the answer cannot be determined from the context, reply: "I do not have sufficient information in our documentation to answer this question."
2. Citations: Cite your sources inline using [Source X] notation matching the context blocks.
3. No Hallucination: Do not speculate or use outside assumptions.

DOCUMENT CONTEXT:
{$contextString}
EOT;

$messages = [
    ['role' => 'system', 'content' => $systemPrompt],
    ['role' => 'user', 'content' => $userQuery],
];
Enter fullscreen mode Exit fullscreen mode

Stage 7: Document lifetime events and cache invalidation

Ingestion and indexing have to keep up with real document changes: edits, deletions, and permission updates. If a policy document, a help article, or a customer record is changed or removed in the primary tables, leftover embeddings can still be retrieved and shown as if they were current, or worse, shown to someone who should no longer see them.

Automate invalidation rather than cleaning the index by hand. Foreign keys with ON DELETE CASCADE, as in the Stage 4 table, remove chunk rows when the parent document is deleted. For edits, a content hash on each chunk lets a worker detect which sections actually changed and refresh only those vectors. Stage 15 describes that incremental re-index in more detail.

Stage 8: Hybrid keyword and vector search

Dense embeddings are good at broad meaning, but they often miss an exact keyword: a serial number, an error code, or a specialized acronym. Combining dense vector search with a sparse keyword search such as BM25 gives you both semantic matches and exact-term matches.

In PostgreSQL, full-text operators (to_tsvector and to_tsquery) can run alongside pgvector cosine similarity. Normalize the two scores and combine them so a user can find either an abstract concept or a precise technical token.

Stage 9: Automated chunk evaluators and quality metrics

Retrieval quality drifts as the corpus and the questions change, so you need a way to measure it. Frameworks such as Ragas or TruLens can score faithfulness and answer relevance over a held-out set of test queries. Chunks that consistently score poorly are a signal to revisit chunk size, overlap, or parser rules, rather than guessing.

Stage 10: Multi-modal RAG pipelines (text and image embeddings)

A lot of useful documentation is not only text. Architecture diagrams, charts, UI screenshots, and scanned receipts carry information that a text-only pipeline will never retrieve.

A multi-modal setup uses a vision-capable embedding model — for example OpenAI CLIP or Nomic Embed Vision — that can place text and images in a shared vector space. When someone asks to see the quarterly revenue chart, the system can retrieve both the related paragraphs and the matching image vectors, then pass the text and image URLs to a vision-capable model (such as GPT-4o) for a grounded answer.

Stage 11: Production vector database maintenance and re-indexing

As documents are created, updated, and deleted, a vector index accumulates stale fragments and query performance can fall off. With pgvector, schedule maintenance jobs that run REINDEX INDEX CONCURRENTLY idx_chunks_embedding; and VACUUM ANALYZE document_chunks;. Concurrent re-indexing builds a fresh HNSW graph in the background so production writes are not locked while the index is rebuilt. How often you run this depends on write volume; start from your own query latency and bloat, not from a fixed calendar.

Stage 12: Disaster recovery and multi-region vector replicas

If search has to stay available when a region fails, or has to run close to users in more than one place, run PostgreSQL read replicas with pgvector in each region you care about. Replica lag and failover still need a rehearsal: promote a replica, point the search path at it, and confirm that the index is present and queryable before you need that path in an outage.

Stage 13: Multi-tenant data isolation and row-level security

When more than one customer organization shares the same tables, every vector lookup has to stay inside that tenant's rows. A similarity search with no tenant filter can return another customer's documents.

Store a workspace_id or tenant_id on every chunk, and put a matching relational filter in the same SQL query that runs the nearest-neighbor search, so unauthorized rows are excluded before scores are used. Do not rely on the model, or on a later application check alone, to enforce that boundary. Securing AI Features: PII and Prompt Injection Basics covers related isolation and injection concerns.

Stage 14: Context compression and token reduction

Sending raw retrieved chunks straight into the prompt can use a large share of the context window. That raises API cost and can hurt quality: models often pay less attention to information in the middle of a long prompt (the "lost in the middle" effect).

A compression step, using a lightweight summarizer or sentence extractor such as LLMLingua, can shrink retrieved chunks before they are inserted into the system prompt. The aim is to drop conversational filler and keep facts, dates, and names. How much you can safely remove depends on the corpus; measure answer quality and cost together rather than aiming at a fixed percentage. Controlling AI Costs in SaaS: Tokens, Caching, and Budget Alerts discusses the cost side of the same problem.

Stage 15: Document hash versioning and incremental re-indexing

If documents change often, re-embedding an entire knowledge base on every small edit is wasted work. A content hash on each chunk makes the update incremental.

During ingestion, compute a SHA-256 hash of each paragraph (or chunk) before you embed it. On a later edit, compare the new hashes with the rows you already stored. Only new or modified chunks need a fresh embedding API call; unchanged text can keep its existing vector. That keeps the index current without re-paying for text that did not change.

Stage 16: Security audit logging and prompt telemetry

For a RAG feature in a SaaS product, you should be able to reconstruct what happened on a given request: the prompt, the retrieved chunk IDs, token usage, the model response, and how long the call took. In Laravel or FastAPI, dispatch that write through an asynchronous listener into a dedicated table, such as rag_execution_logs. Redact PII in anything a person can browse.

A simple internal view of those events lets support and security staff inspect real inputs, retrieved sources, and outputs without granting them raw database access, and it makes retrieval failures easier to diagnose. Observability for AI Features: Logs, Traces, Tokens, and Quality Signals goes further on traces and quality signals.

Stage 17: Fine-tuning re-ranker models for a custom domain

Off-the-shelf cross-encoders are a solid default on general English documentation. They are weaker on specialized language: medical terms, legal contracts, or dense engineering specifications. Fine-tuning a lightweight bi-encoder or cross-encoder on query/chunk pairs from your own domain can improve how those queries are ranked, so the passages that actually govern the answer are more likely to reach the prompt. Build that pair set from real user questions and reviewer judgments, and measure the gain on a held-out set rather than assuming a fixed improvement.

Stage 18: Asynchronous background ingestion and worker queues

Parsing a large document upload, generating embeddings, and writing rows to PostgreSQL is too slow to do inside a web request. You will hit HTTP timeouts and block application workers.

Move ingestion onto a background queue — Laravel Horizon, or Celery with Redis in Python — and let those workers parse, embed, and update tables asynchronously. When a job finishes, broadcast status to the UI, for example over WebSockets, so the user can see that the document is ready to search. Laravel Queue Architecture for Production Backends is a practical reference if you are running this on Laravel, and Laravel AI Developer Guide: OpenAI, RAG, and Safe Rollouts covers a Laravel-specific path around RAG more broadly.

Conclusion

A production RAG pipeline is a sequence of engineering choices, not a single API call: clean parsing, structure-aware chunking, batched embeddings, HNSW-indexed search in pgvector, hybrid keywords where exact terms matter, cross-encoder re-ranking, and a prompt that is required to stay inside the retrieved context. The later stages — invalidation, incremental re-indexing, tenant filters, compression, evaluation, maintenance, replicas, audit logs, and background workers — are what keep that path accurate and safe after the first index is built. None of these steps is a guarantee on its own.

If you are building or optimizing a RAG pipeline, get in touch.


Originally published at https://prakashtank.com/blog/building-rag-pipeline-chunking-embeddings-retrieval.

Top comments (0)