Building Retrieval + Generation for a RAG Pipeline

In Part 1 of this project, I built the document ingestion side of a minimal Retrieval-Augmented Generation (RAG) system.

The pipeline took us from:

Document
   ↓
Text Extraction
   ↓
Chunking
   ↓
Embedding Generation
   ↓
Qdrant

At the end of Part 1, we had documents converted into searchable vector data.

But storing the information is only half the problem.

How do we actually find the relevant information when someone asks a question?

And once we've found it, how do we give that information to an LLM so it can generate a useful answer?

That's what Part 2 covers.

The Complete Retrieval + Generation Pipeline

The pipeline now looks like this:

User Question
      ↓
Query Embedding
      ↓
Qdrant Vector Search
      ↓
Retrieved Chunks
      ↓
Cross-Encoder Reranking
      ↓
Relevant Context
      ↓
LLM Generation
      ↓
Answer

1. Query Embedding

The first step is converting the user's question into the same vector space used for the document embeddings.

This allows the system to compare the semantic meaning of the question against the document chunks stored in Qdrant.

2. Dense Vector Retrieval

The resulting query embedding is used to search Qdrant for semantically similar chunks.

Instead of relying purely on keyword matching, the vector search allows the system to retrieve information based on semantic similarity.

The result is a collection of candidate chunks that could be relevant to the question.

3. Reranking

Vector similarity provides useful candidates, but the highest similarity score isn't necessarily the best possible ranking.

For this project, I added a cross-encoder reranking stage.

The retrieved candidates are passed through the reranker and reordered based on their relevance to the actual query.

This gives the generation stage a smaller and more focused set of context.

4. Context Construction

The highest-ranked chunks are then assembled into the context that will be provided to the LLM.

The basic idea is:

Question
   +
Relevant Document Context
   ↓
LLM

The model isn't expected to search the entire document collection itself.

The retrieval system finds the relevant information first.

5. Generation

Finally, the question and retrieved context are passed to the LLM.

The model generates a response using the retrieved information as context.

For the local generation path, I'm using Ollama so the system can run locally without requiring a cloud model.

The response is also streamed back to the application rather than waiting for the entire generation to complete.

Why Split the Project Into Episodes?

Rather than making one long demo after completing the entire application, I'm building this project in stages.

Part 1 focused on understanding the ingestion side.

Part 2 focuses on retrieval and generation.

The next step is turning these components into an actual document intelligence application.

The goal isn't just to demonstrate that RAG works, but to understand the individual components well enough to build on top of them.

Tech Stack

• Python
• FastAPI
• Sentence Transformers
• BAAI/bge-m3
• Qdrant
• Cross-Encoder Reranking
• Ollama
• Docker

Watch the Build

🎥 Video walkthrough:
https://youtu.be/q2vCQFn-el8

Source Code

💻 GitHub:
https://github.com/azariah11dev/Minimal-RAG-Engine

The repository contains the backend and frontend components, along with their individual documentation.

What's Next?

With ingestion, retrieval, and generation working, the next step is to move beyond a minimal pipeline.

The goal is IntelliDoc: a complete document intelligence application built around the RAG pipeline.