INTRODUCTION TO RAG (RETRIEVAL AUGMENTED GENERATION)

Neville Kibwanga — Wed, 26 Aug 2026 15:25:31 +0000

Intro to RAG (Retrieval Augmented Generation)

I first heard of this technology about a year ago. I've been fascinated by RAG ever since I first encountered it, and I've decided to properly dive into the topic and document what I learn along the way. My goal is simple: explain RAG in a way that's easy to understand, especially for someone encountering it for the first time.

What problem is RAG even solving?

The first ideas about RAG came up around the year 2019/2020 (it was formally introduced by a Facebook AI research paper in 2020). The original paper is Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks by Patrick Lewis et al., published in 2020. RAG was introduced as a way of combining an LLM's internal knowledge with information retrieved from an external knowledge source. This helps address problems such as limited or outdated knowledge and can reduce the likelihood of hallucinations. It's important to note that RAG doesn't retrain or modify the LLM. Instead, it gives the model additional context at inference time. In our daily life, we tend to use LLM's from time to time for different tasks. Examples include OpenAI's GPT models and also Anthropic's Claude models.

Like we all know, LLM's rely heavily on the knowledge learned during training, which means their knowledge can be incomplete, outdated, or missing information about newer or highly specialized topics. Outdated or narrow training data will lead to outdated or narrow results/output. As a result, these models tend to "spew" nonsense output and wrong answers on many occasions, which might mislead the user. This happens since they are naturally designed to generate something, even when they don't actually know the answer.

A real-life analogy

Consider this real-life analogy and relate it to LLM's. I'm currently a fresher in University as I recently concluded my High School studies. There's a friend of mine who needs me to assist him with some Chemistry assignment. Mind you, these are concepts I covered about 3 years ago and have definitely forgotten most of them. It will be easy for me to look at the assignment and answer all the questions even though I'll be guessing most of the answers. The assignment will be done, but I'll have misled my friend into submitting a poorly done assignment. On the other hand, it would be better if I would acquire a Chemistry textbook to use as reference source. This gives me a reliable reference to work from and increases the likelihood that my answers will be accurate.

That is exactly how RAG works. We try to create a reference source (knowledge base) for our LLM in order to reduce hallucination and incorrect answers. We can use different data sources for our knowledge base e.g. PDFs, books, websites, spreadsheets e.t.c

How it actually works, in one sentence

Before generating a response, the RAG system retrieves relevant pieces of information from a knowledge base and provides them to the LLM alongside the user's question. This is the "retrieval" step that gives RAG its name - the model isn't just recalling from memory anymore, it's looking something up first.

                     User Query
                          │
                          ▼
                ┌─────────────────┐
                │    Retriever    │
                └────────┬────────┘
                         │
                   searches in
                         │
                         ▼
                ┌─────────────────┐
                │  Knowledge Base │
                │                 │
                │  Chunk A        │
                │  Chunk B  ✓     │
                │  Chunk C  ✓     │
                │  Chunk D        │
                └────────┬────────┘
                         │
               Relevant Chunks
                         │
                         ▼
                ┌─────────────────┐
                │       LLM       │
                │                 │
                │ Query + Context │
                └────────┬────────┘
                         │
                         ▼
                 Generated Answer

By creating a knowledge base, we give our LLM the chance to refer to an external knowledge source before providing a response. This can produce more accurate and better-grounded responses, especially when the underlying knowledge base is reliable and up to date.

Where it's used

This technology has been widely adopted across different domains, for example:

Customer-support assistants retrieving company documentation
Financial assistants retrieving financial reports and policies
Healthcare systems retrieving medical literature
Legal assistants retrieving legislation and case documents
Internal company chatbots retrieving organizational documentation

Medical knowledge systems can retrieve relevant information from medical literature or clinical guidelines to assist professionals with information retrieval.

A quick caveat

RAG isn't a silver bullet. It reduces hallucination, it doesn't eliminate it - if the retrieval step pulls the wrong chunk of information from the knowledge base, the model can still confidently generate an answer based on the wrong material. Getting retrieval right (what to store, how to search it, what counts as "relevant") is its own deep topic, and it's where most of the real engineering work in RAG happens.

Note that this technology is one of the most efficient/simple ways to help reduce LLM errors.There are other techniques such as fine-tuning models, which may prove to be more technical and also costly.

This is part 1 of a series where I'm writing my way through RAG to close my own knowledge gaps, and to also help those interested in the topic. Next up: the architecture behind RAG - embeddings, vector databases, and how retrieval actually finds the "relevant" chunk.