Retrieval-Augmented Generation: An Introduction Through Landmark Research Papers.

Modules

• Why RAG exists.
• How RAG works.
• RAG and it's siblings.
• Evolution of RAG
• When should you use RAG?
• My take on major RAG papers so far
• Conclusion
• References

Module 1 : Why RAG exists

I believe the best way to understand something is immediately refer to the question "why do we even need this?", that is exactly what we will do in this module.

"Large Language Models (LLMs) showcase impressive capabilities but encounter challenges like hallucination, outdated knowledge, and non-transparent, untraceable reasoning processes." - Gao et al., 2023

Meaning?
LLMs are great at what they do, but the entire basis of what they do comes from the data they train on, in a real life scenario you cannot always guarantee the quality of data being up to the mark.

For example - Suppose I ask an LLM to cite a research paper proving that coffee increases IQ. If no such paper exists, the model may confidently invent a paper title, author, and publication, this is called "Hallucination".
Similarly if the answer or the data in this context on which the LLM trains on is inaccurate we end up with a poor output. An LLM is simply not built to answer "I don't know this", if it can't find an accurate answer it will just throw the closest one at you.
To solve this issue, is exactly why RAG came into the picture

Module 2 : How RAG works

"The original RAG paper proposed combining a pretrained language model with a retrieval mechanism that fetches relevant documents from an external knowledge source before generating a response."
-Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks (2020)

Basically --> Instead of expecting the language model to remember everything, give it access to external knowledge before it answers.

Let's compare two simple workflows.

Scenario 1 (Without RAG)
Imagine you are an LLM and you are solving an exam question paper.

User Question
        │
        ▼
You (LLM)
(Search your trained knowledge)
        │
        ▼
Generated Answer

If your trained knowledge is outdated or missing -> Hallucination
If your trained knowledge is wrong -> inaccurate answer

Scenario 2 (With RAG)
Again, Imagine you are an LLM and you are solving an exam question paper.

BUT -> This time you, the LLM are given a book/document where all answers to all questions are present

+-------------------+
|   User Question   |
+-------------------+
          │
          ▼
+---------------------------+
| Retrieve Relevant         |
| Documents                 |
+---------------------------+
          │
          ▼
+---------------------------+
| Augment Prompt            |
| with Retrieved Context    |
+---------------------------+
          │
          ▼
+---------------------------+
| LLM Reads                 |
| Context + Memory          |
+---------------------------+
          │
          ▼
+---------------------------+
| Grounded Answer           |
| (Accurate & Reliable)     |
+---------------------------+

Whats the difference?
In scenario 2 :

The LLM has to scan just the given reference (the document) for the answers instead of vast amounts of data on the internet.

Instead of "finding out" the answers itself, you provide the LLM the needed answers reducing risks of outdated/inaccurate data.

This is what "Retrieval-Augmented Generation" or RAG is essentially.

Retrieval → Find relevant information from reference

Augmented → Enhance the model's input with that information

Generation → Produce the final answer

Module 3 : RAG and it's siblings

According to the Retrieval-Augmented Generation for Large Language Models Survey, modern RAG systems can be grouped into three types :

1. Naive RAG
2. Advanced RAG
3. Modular RAG

These aren't exactly three different technologies, rather just three different levels or stages of RAG.

Naive RAG

This is where everything began. This is the simplest implementation of a RAG system. The worlflow we explored in module 2 is what a Naive Rag architecture is, the simplest and the most straightforward.

"This simple design significantly improves factual accuracy compared to relying solely on the language model's internal knowledge."

• Retrieval-Augmented Generation for Large Language Models: A Survey (2024)

Advanced RAG

Since Naive RAG depends so much on the context or its reference, Naive RAG failed if the reference or information you gave it was poor.

"Garbage in, Garbage Out"

Hence to improve the retrieval process itself researchers started looking at Advanced RAG.
According to Gao et al., advanced RAG introduces additional techniques before and after retrieval to improve the quality of the information provided to the language model.

"Improve the retrieved context before asking the LLM to generate an answer."

So how exactly did advanced RAG improve the retrieval pipeline? Some ways include :

Query rewriting – Rephrasing the user's question to improve search results.

Re-ranking – Retrieving several documents and then ranking the most relevant ones.

Hybrid retrieval – Combining traditional keyword search with semantic search.

Context compression – Removing unnecessary information before passing it to the LLM.

The important takeaway is that the retrieval process became smarter.

Modular RAG

As RAG systems became more and more advanced a very critical observation was made. We don't really need the same kind of RAG system for every app do we? A chatbot answering questions about company policies doesn't need the same retrieval strategy as a medical assistant or a legal research tool.

Instead of building one fixed pipeline, developers began treating each stage as an independent component. This idea is called "Modular RAG"

"Instead of thinking of RAG as a fixed pipeline (Retrieve → Generate), think of it as a collection of interchangeable modules."
-Modular RAG: Transforming RAG Systems into LEGO-like Reconfigurable Frameworks (2024)

These interchangeable modules can be the different stages of the whole architecture like,

1.indexing,
2.retrieval,
3.re-ranking,
4.memory,
5.routing,
6.generation,
7.post-processing and etc.

Hence, treat RAG like lego. Each stage of the process is each brick. With such an architecture the flexibility of the whole system exponentiates for different kinds of apps.

I am inserting the workflow diagram for all three stages of RAG from the survey paper of Retrieval-Augmented Generation for Large Language Models.

Module 4 : Evolution of RAG

• 2020 : The Birth of RAG

In 2020, Patrick Lewis and his colleagues introduced the original RAG architecture. Instead of expecting the model to memorize everything, they proposed a simple idea that what if the LLM could retrieve relevant documents before generating an answer.

• 2020–2022 : Researchers Realized Retrieval Was the Bottleneck

Around this period researchers realized simply making the language model larger wont help, the real bottleneck was the quality of the information retreived.

`"Many improvements after the original RAG architecture focused on enhancing retrieval quality through better indexing, retrieval strategies, query processing, and post-retrieval optimization."

• Gao et al. (2023)`

• 2024 : Smarter RAG Systems

Once retrieval quality improved, it was time to ask some more questions. Questions like "do we really need to scan through an entire database or reference for every little question?"
Imagine a question like "What is 2+2", why search a database? What if the model could already know? This led to "Self-Rag"

"Self-RAG trains the language model to determine whether retrieval is necessary, rather than assuming every question requires external knowledge."

• SELF-RAG: LEARNING TO RETRIEVE, GENERATE, AND CRITIQUE THROUGH SELF-REFLECTION (2024)

Around this period another issue was also noticed. What if the retreived information was simply bad? they could be irrelevant, wrong etc.
This led to another observation, "What if the system could detect poor retrieval before generating an answer?".
This led to Corrective RAG (CRAG).

"CRAG introduces a mechanism for evaluating the quality of retrieved documents and correcting the retrieval process when necessary."

• Corrective Retrieval-Augmented Generation (CRAG) (2024)

Module 5 : When should you use RAG?

I have come across some very technical jargonish definitions that try to answer this question, but what I have realized is that in real world applications there are only a few scenarios in which using RAG makes the utmost sense. Shoving RAG into a system which doesnt even need RAG is the last thing you should do.
So what are these few scenarios in which RAG makes the most sense?

1. Scenario 1 : Knowledge changes frequently
   Basically any scenario where you might need to retreive the latest information, e.g - stock market, news etc.

2. Scenario 2 : Private knowledge
   If you're dealing with data which is not available on the internet, RAG in that case lets the LLM access it.

3. Scenario 3 : Large document collection
   Any circumstance where you might have to scan through too many documents, pdf's etc (Scenarios where semantic searching makes sense).

Module 6 : My take on major RAG papers so far.

Based on what I read, heres my key observations,

Rag changed the philosophy based on which AI works :
Before I started reading about RAG, i had a very general idea that RAG is just another way to improve an LLM and while that is true, it wasn't untill i went deep that i understood how exactly the models are being improved.
Before RAG, the common approach to improving language models was straightforward: make them larger, train them on more data, and hope they memorize more knowledge. The key assumption was that if we somehow make the model powerful enough to remember everything, it would naturally become better at answering questions, and that was wrong. Papers like Lewis et al. (2020), Gao et al. (2023) clearly reflect how RAG challenged that primary assumption. Especially after breakthrough Lewis et al. (2020) paper, you can almost see how the way of thinking regarding LLM's sort of shifted, the shift from "How can we make the model remember more?" to "How can we make the model remember the necessary things?". Its almost like, instead of having one switch at your house which turns on lights of every room, you introduce a switch for each room so you can identify when you want each room and turn on that light using that switch.

Retrieval Has Become the Real Research Focus :
One trend has become very clear with the more recent papers, i.e very few of them try to reinvent the language model itself. Instead most of them are trying to improve retrieval. One might assume that the LLM is the most important component but turns out in a lot of cases it is theretrieval process. The focus has shifted from "How much the LLM can retreive" to "What the LLM retreives", classic "Quality over Quantity".

Even though papers like Self-Rag, CRAG etc kind of differ in what kind of approaches they take in terms of the retrieval process, for example the Self-Rag paper primarliy questions whether retrieval is even necessary before searching, the CRAG paper explores whether the retrieved documents are trustworthy enough to use, GraphRAG explores what kind of knowledge should be retrieved in the first place, many Advanced RAG techniques focus on rewriting queries, re-ranking documents et, but all these papers try to solve the same underlying problem : The challenge is no longer generating fluent language. The challenge is finding the right information to generate from. This is the biggest trend in RAG right now.

Conclusion

When Lewis et al. introduced RAG in 2020, the idea was remarkably simple: let language models retrieve information instead of relying entirely on memory. Four years later, researchers are still building on that same idea not by making models memorize more, but by helping them retrieve better. That, to me, is what makes RAG one of the most influential ideas in modern AI.

References

Lewis, P., Perez, E., Piktus, A., Petroni, F., Karpukhin, V., Goyal, N., Küttler, H., Lewis, M., Yih, W., Rocktäschel, T., Riedel, S., & Kiela, D. (2020). Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks. Advances in Neural Information Processing Systems (NeurIPS 2020).

Gao, Y., Xiong, Y., Gao, X., Jia, K., Pan, J., Bi, Y., Dai, Y., Sun, J., Wang, M., & Wang, H. (2023). Retrieval-Augmented Generation for Large Language Models: A Survey. arXiv preprint arXiv:2312.10997.

Karpukhin, V., Oguz, B., Min, S., Lewis, P., Wu, L., Edunov, S., Chen, D., & Yih, W. (2020). Dense Passage Retrieval for Open-Domain Question Answering. Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP).

Izacard, G., Caron, M., Hosseini, L., Riedel, S., et al. (2022). Atlas: Few-shot Learning with Retrieval Augmented Language Models.

Asai, A., Wu, Z., Wang, Y., Sil, A., & Hajishirzi, H. (2024). Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflection.

Yan, S., et al. (2024). Corrective Retrieval-Augmented Generation (CRAG).