Introduction -> RAG

What is RAG?

Formal Defination: Retrieval Augmented Generator it is an AI framework that improves the accuracy of LLM's by fetching facts from an external authoritative knowledge base before generating a response. It prevents the AI from "hallucinating"(making the facts by it's own doesn't matter the fact is false) and provides up to date, domain specific answers.

Why RAG shows up in AI eco system?

Okay, let's talk about without RAG. So we use AI for our day to day life, in different domains. LLM's are trained on the real world data. But as you know in our day to day life every second new data produce and the size of those data is humongous. Now based on the that humongous data if the companies try to update their LLM models than their will be chaos because to trained LLM models we need a lot of computing power and other things. So it is not possible now to trained LLM's based on real time data. Their is a term for this situation which is "Knowledge cutoff".

Now in your mind a question might arise, that so if they can not trained models with real time data than how can it gives response to question related to present time?

Here enters the RAG (Retrieval Augmented Generator). So let's talk about what RAG actually do. So at first when you give AI a prompt related to present time, where the LLM's are not trained yet with present time data. So instead of giving the response to the user, LLM calls tools to get information from the internet. For example you prompt this "What is the weather today in Dhaka? ". So here present time comes in so LLM's are smart enough and it detects that user wants to know about present day weather information, so it has some tools, it uses them to get the data from the internet in current time and in this way it is able to get the real time data. This all happens by using RAG framework.

Now you have the idea about RAG, we can go to the simple architecture of RAG:

So basically in RAG there are two pipelines present:

1. Indexing phrase
2. Query phrase

Indexing phrase:

In indexing phrase these steps are followed:

• Text to Chunk
• Chunk to vector embedding
• Store chunk and vector embedding into a vector db.

Now let's explore each steps:

• Text to chunk
  First text to chunk it means that text is basically the prompt, docs, files etc that user provides. Suppose user give some pdf's and later based on these pdf's user will asked questions. So now on the pdf all the texts will divided into small small pieces and each piece is basically called a chunk.

• Chunk to vector embedding
  We have chunks but these chunks does not mean anything, so to give meaning to each chunk we use vector embedding. After doing the vector embedding we have the real world meaning about each chunk.

• Store chunk and vector embedding in vector db
  After getting the meaning of each chunk now we need to store those chunks with vector embedding so that in the query phrase where user pass any query it can go through all the meaning and select the closest one and based on that chunk it can perform next things. So there are so many vector databases present to store the vector embeddings with chunk. So we store the chunks and meaning to a db.

Query phrase:

In query phrase these steps are followed:

• User text to Chunk
• Chunk to vector embedding
• Similarity query search in the vectorDB.
• Chunk retrieved from vectorDB.
• Send user query and retrieved chunks to the LLM as system prompt.

Now let's explore each steps:

• User text to Chunk:
  
  Just like indexing phrase here also user query or say prompt or text convert into chunks.

• Chunk to vector embedding:
  Same as indexing phrase. To give mathematical meaning to the chunks we go through vector embedding.

• Similarity query search in the vectorDB:
  We have the user query chunk and it's mathematical meaning we can search the vectorDB to find that, is there any close to user query vector meaning find in the db. Only this thing is done here.

• Chunk retrieved from VectorDB:
  Here the similar chunks retrieved from the db.

• Send user query and retrieved chunks to the LLM as system prompt:
  Here the user query and the chunks that are retrieved from the DB send to the LLM models as a system prompt. So the LLM have the context and based on the user query give response to it.

Image credit: Piyush Garg

---

Common scenarios where RAG works well

• For students or any learning professionals if they wants to ask questions or find topics from big pdf's than RAG can be used, where user can upload files and based on those files student can asked the agent related questions. Now the agent will have a good reference and it will also look to it's trained data to give a solid answer, which will help the student to understand the topic more better.

• Employees waste their most of the time searching something in docs, pdf's, files, chats etc.So a RAG system can be introduced which will decrease the time needed for searching stuffs and employee can focus on more to solve the real problems.

• Privacy policies search: Most of the time users are not interested to read the privacy policies of a website or company because of the long listed paragraphs and the number of pages there in the policy section. So there can be RAG introduced so that it can read the policies and than user can asked questions about the policies.

• Financial Analysis: In financial sectors there are lots of documents and reports. Until now all the documents need to check at a time which takes a lot of time and also man power also money. So if we use RAG here user can uploads documents and then the agents will go through the documents and answer the user questions. Also it can remember that where which data is place like page number etc which makes easier for user to find out correct things in correct place.

• Education area: In all online courses each course contains many videos. So if someone wants to learn something or look at some specific topic he have to go through so many videos. Sometimes he get will found the topics he is looking for but most of the time didn't get the desired topic which makes learning more difficult,if student wants to go through some topics he can not. So if we use RAG here we can help students to find their desired topic and also able to give them exact time stamp and video number which will make the study more reliable and more enjoyable.

---

RAG issues

If the agent have incorrect data than it will give incorrect answers. Because it takes context or idea based on incorrect data. We human have the capability to understand corrupt or say unstructured data which is not possible for the LLM models. Also there is a limitation of memory it can have. If you give a huge data set into the context than it started hallucinating as a result it gives wrong answers with full confident.

Why issues arise?

• Poor retrieval: Poor retrieval simply means that retriever found documents, but they are the wrong ones. For example: You provide documents about "how to reset password in facebook" but you give this prompt "how can i reset my linkedin password?" now the LLM will give you response based on that provided documents which will be related to facebook password reset but you want for linkedin.

Why poor retrieval happens:
->Bad embedding : Embedding is basically converts text into vectors. If two related sentences end up far a part in that 3D vector space than similarity search fails. For example: You give a document which contains "How to reset your fb password." but the user query is "I forgot my fb password." Now if bad embedding happens than the "reset password" and "forgot password" the distance between them will be so far that the model can not make a proper similarity match between them, as a result the model will not understand the user query.

• Poor Chunking: In the RAG their is a phrase where the text converted into small chunks. Now if these chunks divided in a way were in each chunk there is no connection between other chunks than it will be a reason for bad response. For example: Suppose your document is this:

Step 1:
Go to settings
Click password
Enter new password

now when after you provide this document and the chunking is like this:

Chunk A: Step 1:
Go to settings

Chunk B: Click password

Chunk C: Enter new password

Now if the chunks are in this way than if the retriever retrieve chunk C than the LLM will get "Enter new password" but it has no idea where to go as a result it can not provide a good response to the user because the LLM itself don't know where to go.

• Missing context: It means the retriever actually found the correct document but the information provided is incomplete. For example: User provide a document which contains all the ingredients name and the portion numbers, but it doesn't contain anything related to "how to make biryani?" and if the user query is like "tell me how to cook biryani?" that time for missing the context the agent response will be incomplete or wrong.