In this new AI world data is one of the most valuable things. But having data is not enough. It needs to be useful and up to date.
What do I mean by up to date?
Ever thought of uploading your 3000 page PDF into ChatGPT? The quality of the answers you get is horrible. Now imagine the cost of sending that PDF every time you ask a question related to it.
There is also one more issue. If you ask an LLM about some guidelines that changed a week ago it is unable to answer correctly because LLMs do not search the web every time you ask a question.
If you want to know how ChatGPT actually answers your questions you can read this.
This is where RAG comes in , Now lets see how it does all these things.
Topics to Cover
- What RAG is and why it was introduced
- How a basic RAG pipeline works
- Common scenarios where RAG works well
- Why RAG sometimes gives incorrect answers
- Poor retrieval and missing context
- Poor chunking and its impact on responses
- Context window limitations
- Hallucinations even with RAG
- Keeping knowledge bases up to date
- When RAG is not the right solution
What RAG is and why it was introduced
RAG stands for Retrieval Augmented Generation. Let's understand what it means.
Retrieval :- Find useful information.
Augmented :- Add that information to the AI's knowledge.
Generation :- Use it to generate a better response.
why it was introduced
As i discus earlier that LLMs do not actually know things beyond their training data. Ask them about your a week change ago company guideliance , and they’re clueless.
That is where RAG comes in. Instead of relying on a static model RAG adds some functionality around the AI that helps it give better and more efficient answers. It does not matter if you upload a 3000 page PDF. It can still answer questions from it.
now we know the basic overview of a RAG and what it does lets understand how it does ?
How a basic RAG pipeline works
There are basically 2 pipelines we will discuss
- Indexing
- Query
Indexing
Indexing basically means converting your files into embeddings and storing them inside a vector database. Don't get confused by the terminology. It is much simpler than it sounds.
Let's take a PDF as an example.
First your PDF is converted into plain text.
Then the text is split into smaller chunks. The chunk size depends on what you choose. It can be a paragraph a page a sentence or even a single word.
These text chunks are then sent to an embedding model such as text-embedding-3-small or text-embedding-3-large. Almost every LLM provider offers embedding models.
The embedding model converts each text chunk into an array of numbers.
"This is a text chunk"
↓
[1246 325 3 45 2 ...]
Finally these embeddings are stored inside a vector database which is specially optimized for storing and searching embeddings.
Query
Query simply means asking a question to your RAG based on the files you have already uploaded.
Let's say you uploaded a PDF about DSA and you ask:
What is a sliding window
First your question is sent to the same embedding model which converts it into an embedding.
"What is a sliding window"
↓
[845 91 201 56 ...]
This embedding is then compared with all the embeddings stored inside the vector database using vector similarity search.
Instead of finding the exact words it finds the most relevant chunks that are semantically similar to your question.
Finally your original question and those relevant chunks are sent to a normal LLM which then so some generation and generate a response and you will get your answer.
Common scenarios where RAG works well
We can say that the limitation of LLMs is the advantage of RAG like the RAG works best when the information is too large for an LLMs to remember an constantly changing,
Some common examples
- Chat with PDFs
- Company knowledge base
- Customer support bots
- Internal documentation search
- Legal and policy documents
Why RAG sometimes gives incorrect answers
Now we know the advantages of RAG and understand its pipeline let look at some failures where RAG fails
Context window limitations
First i would tell you about context window so a context window as the name says a window that decides the maximum amount of text that a LLM can process and remember at one time
Retrieved Chunks
[Chunk 1]
[Chunk 2]
[Chunk 3]
[Chunk 4]
[Chunk 5]
-------------------------- Context Window --------------------------
[Chunk 6] ❌ Ignored
[Chunk 7] ❌ Ignored
[Chunk 8] ❌ Ignored
If too many chunks are retrieved or the chunks are too large they may exceed the context window. In that case some information has to be removed or the model may miss important details.
This is why choosing the right chunk size and the right top_k value is important
Poor retrieval and missing context
RAG often fails because of poor data quality and badly prepared documentation.
Imagine you upload your friend's college project documentation. Do you really expect RAG to give great answers? Probably not.
If the input is poor the output will also be poor. RAG can only retrieve information that already exists in your documents.
Poor chunking also reduces retrieval accuracy. If a sentence or idea is broken into the wrong chunks the meaning gets lost. Because of that RAG may retrieve the wrong chunks instead of the ones that actually answer your question.
Better documents and better chunking almost always lead to better responses.
Poor chunking and its impact on responses
Poor chunking breaks the meaning of your documents.
When important information is split into the wrong chunks RAG may retrieve incomplete or irrelevant context. As a result the LLM receives weak context and generates less accurate answers.
Hallucinations even with RAG
It is very common failure on RAG where RAG starts hallucinating because when retrieved chunks does not have any context to answer the user query the LLM start guessing the output to fill the gaps
Let's say you upload a legal document into a RAG system. The document contains law references like Section 5A2DC or other legal codes. As it does not have this code context so its start guessing the output that leads to hallucinate the system
Keeping knowledge bases up to date
A knowledge base is simply the collection of documents your RAG uses to answer questions. It is like our 3000 files of PDF documentation or manuals FAQs or any other files.
If the knowledge base is outdated RAG will also return outdated answers. Keeping it updated helps the system retrieve the latest and most accurate information.
When RAG is not the right solution
RAG is not the right choice for every problem.
If the answer does not depend on external documents a normal LLM is often enough.
For example:
- General conversations
- Creative writing
- Brainstorming ideas
- Summarization
- Simple coding questions
- Tasks that do not require private or up to date knowledge
Use RAG only when the model needs to retrieve information from your own documents or an external knowledge base.
Thanks for reading ! if enjoyed this blog , you can read more on this 👇
Top comments (0)