What Is a RAG Pipeline? A Plain‑English Guide

What Is a RAG Pipeline? A Plain‑English Guide

A RAG pipeline is a system that adds a retrieval step before the language model generates an answer — so the model works from a fresh, specific chunk of your data rather than guessing from its training memory. Think of it as giving the LLM an open‑book exam: the retrieval layer finds the right page, and the model then writes the answer using only that page.

Founders and product managers hear about RAG constantly, but most explanations stop at the concept and skip the part that actually decides whether the feature ships: retrieval quality. The whole pipeline lives or dies on getting the right chunk in front of the model. When a RAG‑powered feature feels dull or wrong, the fix is almost always in the chunking or retrieval — not in a fancier LLM. This article shows you a real RAG diagram from a production screener, explains why retrieval breaks, and walks through the practical steps to stop it.

The RAG Pipeline in Plain English

A RAG pipeline has two distinct stages: retrieve, then generate.

1. Query — a user asks a question or gives a task.
2. Retrieve — the pipeline searches your knowledge base (documents, database, API results) and pulls the most relevant chunks.
3. Augment — those chunks are stuffed into the prompt alongside the user’s question.
4. Generate — the LLM produces an answer that is anchored to the retrieved text.

That’s the basic loop, and NVIDIA’s RAG 101 describes it as an offline ingestion step followed by an online query phase. AWS frames it as “optimizing the output of a large language model so it references an authoritative knowledge base,” and Wikipedia calls it “a technique that enables LLMs to retrieve and incorporate new information from external data.”

In practice, the retrieval side is a search engine over your documents, and the generation side is a prompt that says: “Answer using only the context below. Quote the source passage.”

The Real Diagram: How Our Screener Uses a RAG Pipeline

At techpotions, we built ReadyShortlist — an end‑to‑end product for Pakistan’s vetted tech‑talent network that ships interview‑ready shortlists in 72 hours. The core is a rubric‑driven LLM screener that is essentially a RAG pipeline with one non‑negotiable rule: every score must quote the exact CV passage that justifies it.

The pipeline looks like this:

1. Job Description → Rubric A structured rubric with 4‑6 weighted criteria (e.g., “Flutter experience,” “team leadership”) is generated from the JD.
2. CV Ingestion & Chunking Each CV is split into semantic chunks — every project section, employment block, and skill list gets its own chunk with metadata (candidate ID, section name).
3. Querying by Criterion For each rubric criterion, a retrieval call fetches the top‑k chunks from that CV that are most likely to contain evidence.
4. Cite‑Then‑Score The prompt shows the candidate’s retrieved chunks to the LLM and forces a format: Score: X/5 Evidence: “... exact text from the CV chunk ...”
5. Ranked Shortlist Every candidate gets a per‑criterion score and a weighted total, all backed by highlighted evidence. The screener runs in under 30 seconds cold‑start on Vercel.

This enforce‑citations design is what makes the RAG pipeline trustworthy. A recruiter can click any score and see the exact sentence from the CV that drove it. Without that requirement, the system would hallucinate scores and lose credibility — exactly what happens when retrieval is an afterthought.

Results that matter:

• 84 % precision vs. a human‑assembled shortlist (measured on a purpose‑built eval set).
• 6× faster screening per role.
• A 380‑case golden eval set that gates every model or retrieval change before it reaches production.

Why Retrieval Fails in Production (And How to Fix It)

When a RAG feature feels “dumb,” founders usually assume the LLM is the bottleneck. In our experience across multiple retrieval‑heavy builds — a screener that extracts quoted evidence from resumes and an orientation engine that matches profiles against a 23 000‑item formation catalogue — the failure is almost always the retrieval/chunking step. The model can only be as smart as the text you give it.

Failure Mode	Symptom	Fix
Naive chunking (fixed character‑count, split mid‑sentence)	Scores backed by partial or irrelevant sentences	Section‑aware splitting; every chunk gets a label like “Work Experience — Company X”
Missing citation enforcement	Answers sound plausible but are unverifiable	Require the LLM to quote the passage it used; refuse to score without a citation
Weak retrieval (sparse keyword, no reranking)	Top‑k chunks miss the crucial information	Hybrid search (keywords + vector), and always keep enough chunks (k ≥ 10) so re‑ranking can surface the right one
No eval set	You can’t tell if a retrieval change is better or worse	Build an eval suite with ground‑truth passage IDs — we ship with 380 cases in ReadyShortlist
Blindly tuning the LLM	Scores become inconsistent but retrieval stays broken	Freeze the model and fix retrieval first; then use the eval set to validate

The ReadyShortlist screener faced exactly the first three failures in early prototypes. Once we moved to chunking by semantic CV sections, forced citation, and introduced a small re‑ranker, precision climbed from mid‑50s to 84 %. The model didn’t change.

How to Build a RAG Pipeline That Actually Works (Step‑by‑Step)

Every step below is shaped by the principle that retrieval quality gates everything.

1. Ingest and chunk documents semantically

Skip the naïve 500‑token split. For resumes, chunk by employment block, education, and projects. For long‑form docs, split at headings or meaningful paragraph boundaries. Add metadata — document_id, section_title, candidate_id — so you can trace any retrieved chunk back to its source.

def chunk_resume(resume_text, candidate_id, section_headers):
    chunks = []
    current_section = None
    for line in resume_text.splitlines():
        if any(header in line for header in section_headers):
            current_section = line.strip()
        elif current_section:
            chunks.append({
                "text": line.strip(),
                "section": current_section,
                "candidate_id": candidate_id
            })
    return chunks

2. Embed and store with hybrid search in mind

Use an embedding model to vectorise chunks, and store them in a vector database that supports both sparse (keyword) and dense (vector) retrieval. Even a simple BM25 + cosine hybrid often beats pure vector search on domain‑specific text.

3. Write a retrieval function that returns the sources

Never let the LLM generate an answer without seeing the actual passage ID and full text. The retrieval function must return the chunk text, the section name, and a unique identifier.

def retrieve_chunks(query, index, k=10):
    dense_hits = index.vector_search(query, top_k=k)
    sparse_hits = index.keyword_search(query, top_k=k)
    # fuse results, deduplicate, keep top-k
    fused = fuse_scores(dense_hits, sparse_hits)[:k]
    return [{"chunk_id": hit.id, "text": hit.text, "section": hit.section} for hit in fused]

4. Prompt with citation‑required formatting

Construct the prompt so it sees the retrieved chunks and is forced to cite them:

You are scoring a candidate on the criterion {criterion}. Only use the passages below. Output: Score: X/5 Evidence: quote the exact supporting text from a passage.Passages: [Chunk 1] … [Chunk 2] …

5. Build an eval set from day one

Collect real‑world examples: (query, correct chunk IDs). Run the pipeline offline, measure recall@k, and track how often the final answer cites the correct chunk. At ReadyShortlist, the golden eval set grew to 380 cases and became the single source of truth for every retrieval tweak.

When to Use a RAG Pipeline vs. Fine‑Tuning or Prompt Engineering

Approach	Best For	Weakness
Pure prompt engineering	Tiny, static tasks (classification, simple extraction)	Can’t handle large external knowledge; highly sensitive to prompt phrasing
RAG pipeline	Large, frequently‑updated knowledge bases; tasks that require verifiable citations	Requires careful chunking and eval; slightly higher latency
Fine‑tuning	Permanent style/domain adaptation (e.g., tone, format); small static dataset	Stale knowledge; expensive to update; can’t cite specific passages easily

Most production systems that answer questions over documents, screen candidates, or match profiles against catalogues start with RAG — and only reach for fine‑tuning when the model needs a consistent brand voice or domain vocabulary that retrieval can’t supply.

How We Build RAG Pipelines

At techpotions, we don’t build demo‑ware. Every RAG pipeline we ship includes:

• Chunking that respects document structure, not arbitrary token counts.
• Retrieval that returns citations, not just scores.
• An eval suite that gates every merge.

The ReadyShortlist case study shows what that looks like in practice: 84 % precision, 6× faster screening, and a screener that recruiters can trust because they can click any score and read the evidence.

If you’re evaluating a RAG project, see our AI services or start a conversation — we’ll help you design a retrieval step that doesn’t break when real users show up.

FAQ

What’s the difference between RAG and fine‑tuning?

RAG (retrieval‑augmented generation) gives an LLM access to a retrieval step that fetches relevant passages from a knowledge base before it generates an answer. Fine‑tuning changes the model’s weights on a specific dataset. Use RAG when you need the model to cite verifiable information from a large, frequently‑updated corpus; fine‑tuning is better when you want to bake a style or a narrow static domain directly into the model.

How do I measure retrieval quality in a RAG pipeline?

Build an eval set with ground‑truth passage‑level answers. For each query, store the chunk IDs that should be retrieved. Run the pipeline offline, measure recall@k and precision, and track how often the final answer cites the correct passage. Gate merges on these metrics — they’re the only reliable signal.

Can I use RAG without a vector database?

You can start with a simple keyword index or even an in‑memory search if your corpus is small. But vector search gives you semantic matching that drastically improves recall. For production, a vector database (or a hybrid keyword‑plus‑vector approach) is almost always worth the effort.