Artificial Intelligence has made it possible for Large Language Models (LLMs) to answer a wide range of questions. However, one major limitation of LLMs is that they only know the information they were trained on. They cannot automatically access the latest documents or organization-specific knowledge. This is where Retrieval-Augmented Generation (RAG) becomes valuable.
What is RAG?
Retrieval-Augmented Generation (RAG) is a technique that allows an LLM to retrieve relevant information from external knowledge sources before generating a response. Instead of relying only on its pre-trained knowledge, the model first searches for relevant information and then uses it to produce a more accurate and context-aware answer.
In simple terms:
RAG = Retrieval + Generation
- Retrieval: Find the most relevant information.
- Generation: Generate an answer using the retrieved information.
The RAG Pipeline
1. Document Loading
The first step in the RAG pipeline is loading the documents. These documents can come from different sources such as:
- PDF files
- Word documents
- Text files
- Websites
- Images (after OCR)
- Audio transcripts
- Databases
In frameworks like LangChain, a document loader converts these sources into Document objects.
Each document contains two important attributes:
page_content
page_content contains the actual textual content of the document.
Example:
Python is a high-level programming language.
metadata
metadata stores additional information about the document.
For example:
{
"source": "python_notes.pdf",
"page": 3,
"author": "John"
}
Metadata helps identify where the retrieved information came from, making the responses more reliable and traceable.
2. Chunking
Large documents are difficult for embedding models and LLMs to process directly. Therefore, the documents are divided into smaller pieces called chunks.
This process is known as chunking or text splitting.
Different chunking strategies include:
- Fixed-size chunking
- Recursive chunking
- Sentence-based chunking
- Paragraph-based chunking
- Semantic chunking
Two important parameters during chunking are:
- Chunk Size: The amount of text contained in each chunk.
- Chunk Overlap: The shared text between consecutive chunks, which helps preserve context.
Choosing the right chunk size and overlap significantly improves retrieval quality.
3. Embedding
After chunking, each chunk is converted into an embedding.
An embedding is a numerical representation of text in a high-dimensional vector space. Instead of storing text directly, the embedding model converts each chunk into a vector that captures its semantic meaning.
For example,
Chunk:
"Python is widely used for Artificial Intelligence."
may become
[0.12, -0.45, 0.78, 0.91, -0.23, ...]
It is important to understand that:
- One chunk produces one embedding vector.
- One embedding vector contains hundreds or thousands of numerical values (dimensions).
Depending on the embedding model, the vector may contain:
- 384 dimensions
- 768 dimensions
- 1536 dimensions
- 3072 dimensions
These vectors allow the system to compare the meanings of different pieces of text rather than simply matching keywords.
Different embedding models have different token limits and performance characteristics, so selecting the appropriate model depends on the application's requirements.
4. Vector Database
The generated embeddings are stored in a vector database.
Unlike traditional databases, a vector database stores vector embeddings and performs semantic similarity searches.
Popular vector databases include:
- ChromaDB
- FAISS
- Pinecone
- Milvus
- Weaviate
The vector database stores the embeddings of the document chunks so they can be retrieved efficiently when a user asks a question.
5. User Query
When a user asks a question, the same embedding model converts the user's query into an embedding vector.
For example,
User Question
How is Python used in Artificial Intelligence?
The question is converted into a vector representation similar to:
[0.91, -0.20, 0.28, 0.62, 0.80, ...]
This query vector is not permanently stored in the vector database. It is generated temporarily to search for similar document embeddings.
6. Similarity Search
The vector database compares the query embedding with every stored document embedding.
It calculates how similar the vectors are using similarity measures such as:
- Cosine Similarity (most commonly used)
- Dot Product
- Euclidean Distance
For example:
| Stored Chunk | Similarity Score |
|---|---|
| Chunk 1 | 0.45 |
| Chunk 2 | 0.30 |
| Chunk 3 | 0.98 ✅ |
| Chunk 4 | 0.40 |
A higher similarity score indicates that the chunk's meaning is closer to the user's question.
In this example, Chunk 3 has the highest similarity score because it is semantically most similar to the user's query.
The vector database retrieves this chunk and sends it to the application.
7. Context Augmentation
The retrieved chunks are combined with the user's original question to form an augmented prompt.
Instead of sending only the user's question to the LLM, the application sends both the retrieved context and the question.
For example,
Context:
Python is widely used for Artificial Intelligence.
Question:
How is Python used in AI?
This additional context enables the LLM to generate more accurate and grounded responses.
8. Response Generation
Finally, the augmented prompt is sent to the LLM.
The LLM reads both:
- The user's question
- The retrieved context
It then generates the final response based on both pieces of information.
RAG with Different Types of Data
One interesting aspect of RAG is that it can work with both structured and unstructured data.
Unstructured Data
Examples include:
- PDFs
- Word documents
- Text files
- Images
- Audio transcripts
The typical pipeline is:
Load → Chunk → Embed → Store in Vector Database
Structured Data
Structured data is stored in relational databases such as MySQL, PostgreSQL, or SQLite, where the data is organized into rows and columns.
Unlike unstructured data, structured data does not require chunking, embeddings, or a vector database. Instead, AI frameworks such as LangChain use a SQL Toolkit to retrieve information from the database.
The process works as follows:
- The user asks a question in natural language.
- LangChain sends the user's question to the LLM.
- The LLM analyzes the question and decides whether a tool is required.
- If database access is needed, the LLM requests the SQL Toolkit.
- LangChain executes the SQL Toolkit.
- The SQL Toolkit generates and executes the appropriate SQL query on the relational database.
- The database returns the query results to the SQL Toolkit.
- LangChain passes these results back to the LLM.
- Finally, the LLM generates a natural-language response, which is returned to the user.
The overall flow is:
User → LangChain → LLM → SQL Toolkit → Database → SQL Toolkit → LangChain → LLM → User
Why Do We Need RAG?
Without RAG, an LLM answers questions using only the knowledge it learned during training.
With RAG:
- It can access up-to-date information.
- It can answer questions about private documents.
- It reduces hallucinations.
- It improves the accuracy and reliability of AI applications.
- It provides answers grounded in external knowledge.
My Key Takeaways
While learning RAG, I realized that building a RAG application involves much more than connecting an LLM to a database. Every stage of the pipeline contributes to the quality of the final answer.
- Document loaders prepare documents with both content and metadata.
- Chunking divides large documents into meaningful sections.
- Embeddings convert each chunk into a high-dimensional vector representing its semantic meaning.
- Vector databases store these vectors and perform efficient similarity searches.
- The user's query is also converted into an embedding vector.
- Similarity search retrieves the chunks whose meanings are closest to the user's question.
- The retrieved context is combined with the user's query before being sent to the LLM.
Overall, RAG extends the capabilities of Large Language Models by allowing them to retrieve and use external knowledge, making AI applications more accurate, reliable, and useful for real-world scenarios.
Top comments (0)