Imagine joining a company with hundreds or even thousands of internal documents.
There are HR policies, employee handbooks, product documentation, technical guides, compliance documents, SOPs, and internal processes.
Now an employee asks:
“How many unused paid leaves can I carry forward to next year?”
You could search through folders, open PDFs, use Ctrl+F, or ask the HR team.
But what if an AI assistant could answer the question in seconds — using the company's actual documents instead of guessing?
That's where Retrieval-Augmented Generation (RAG) becomes useful.
In this article, we'll build a simplified version of a real-world RAG system and understand what's happening behind the scenes.
What Problem Are We Solving?
Let's imagine a company has this information:
/company-documents
├── employee-handbook.pdf
├── leave-policy.pdf
├── health-insurance.pdf
├── remote-work-policy.pdf
├── travel-policy.pdf
└── expense-policy.pdf
An employee asks:
“Can I carry forward unused paid leaves to next year?”
A general-purpose LLM may know something about leave policies in general.
But it doesn't automatically know this company's specific policy.
We need to give the model access to the company's private knowledge.
That's the problem RAG solves.
What Is RAG?
RAG stands for Retrieval-Augmented Generation.
The basic idea is simple:
User Question
↓
Retrieve Relevant Information
↓
Give Information to LLM
↓
Generate Answer
Instead of asking the LLM to answer entirely from what it already knows, we first retrieve relevant information from our own knowledge base.
The complete flow looks like this:
User Question
↓
Generate Embedding
↓
Search Vector DB
↓
Retrieve Documents
↓
Relevant Context Found
↓
Context + Question
↓
LLM
↓
Final Answer
This makes the LLM much more useful for private or frequently changing information.
Step 1: Collect the Documents
Our first step is to collect the company's documents.
For example:
HR Policy
Product Documentation
SOPs
Legal Documents
Technical Documentation
Employee Handbook
These documents might come from:
- PDFs
- Word documents
- Websites
- Databases
- Cloud storage
- Internal knowledge bases
For our example, we'll assume the documents are PDFs.
Step 2: Extract the Text
The PDF isn't immediately useful to the LLM.
We first need to extract the text.
Conceptually:
documents = load_documents("./company-documents")
The result might look like:
Document 1:
Employee Leave Policy
Employees are entitled to 24 paid leaves per year...
Unused paid leaves may be carried forward up to a
maximum of 10 days...
Now we have machine-readable content.
Step 3: Split the Documents into Chunks
Here's an important part of building a RAG system.
We usually don't want to put an entire 100-page document into every LLM request.
Instead, we split it into smaller pieces called chunks.
For example:
100-page PDF
↓
Chunks
↓
┌───────────────┐
│ Chunk 1 │
├───────────────┤
│ Chunk 2 │
├───────────────┤
│ Chunk 3 │
├───────────────┤
│ Chunk 4 │
├───────────────┤
│ ... │
└───────────────┘
A simple implementation could look like:
from langchain_text_splitters import RecursiveCharacterTextSplitter
splitter = RecursiveCharacterTextSplitter(
chunk_size=800,
chunk_overlap=100
)
chunks = splitter.split_documents(documents)
The overlap helps preserve context between neighboring chunks.
However, chunking isn't simply about choosing a number like 500 or 1,000 characters.
Good chunking should consider the structure of the information.
For example, keeping a policy heading together with its explanation is generally more useful than splitting them apart.
Step 4: Convert Text into Embeddings
Now we have chunks of text.
But how do we search them intelligently?
This is where embeddings come in.
An embedding converts text into a numerical representation — a vector.
For example:
"Employees can carry forward 10 unused leaves"
↓
Embedding Model
↓
[0.12, -0.43, 0.81, 0.17, ...]
The important idea is that text with similar meaning tends to have vectors that are close together in vector space.
For example:
"How many leaves can I carry forward?"
should be semantically close to:
"Employees may carry forward up to 10 unused paid leaves."
even though the wording is different.
That's why vector search is so useful for RAG.
Step 5: Store the Embeddings in a Vector Database
We need somewhere to store these vectors.
Popular vector databases include:
- Qdrant
- Pinecone
- Weaviate
- Milvus
- Chroma
Our architecture now looks like:
Documents
↓
Text Extraction
↓
Chunking
↓
Embeddings
↓
Vector Database
Each stored record can contain more than just the vector.
For example:
{
"text": "Employees may carry forward up to 10 unused paid leaves.",
"document": "leave-policy.pdf",
"page": 7,
"department": "HR"
}
That metadata becomes extremely useful later.
Step 6: A User Asks a Question
Now an employee asks:
“How many paid leaves can I carry forward?”
We convert that question into an embedding as well.
User Question
↓
Embedding Model
↓
Question Vector
Then we search the vector database for the most relevant chunks.
Conceptually:
results = vector_store.similarity_search(
"How many paid leaves can I carry forward?",
k=5
)
The database might return:
Result 1
Leave Policy — Page 7
"Employees may carry forward up to 10 unused paid leaves."
Result 2
Employee Handbook — Page 22
"Unused leave is subject to company leave policy."
Result 3
Leave Policy — Page 8
"Carry-forward balances are reset according to..."
Now we have relevant context.
Step 7: Give the Context to the LLM
This is where generation happens.
Instead of asking:
LLM:
"How many paid leaves can employees carry forward?"
we provide the retrieved information:
Context:
Employees may carry forward up to 10 unused paid leaves.
Question:
How many paid leaves can I carry forward?
The LLM can then generate:
Employees can carry forward up to 10 unused paid leaves to the next year, according to the company's leave policy.
This is the fundamental idea behind RAG.
The Complete Pipeline
Our system now looks like this:
USER
│
↓
"How many leaves
can I carry?"
│
↓
Query Embedding
│
↓
┌─────────────────┐
│ Vector Search │
└────────┬────────┘
↓
Relevant Documents
│
↓
┌─────────────────┐
│ LLM │
└────────┬────────┘
↓
Final Answer
│
↓
Source: Leave Policy
This is a basic RAG application.
But a production system needs more.
What Happens If the Answer Doesn't Exist?
This is one of the most important parts of a real-world RAG system.
Suppose an employee asks:
“Does the company provide free international travel insurance for personal vacations?”
But the company documents don't contain this information.
A poorly designed system might generate a confident answer.
That's dangerous.
A better system should recognize that there isn't enough evidence.
For example:
“I couldn't find information about personal international travel insurance in the available company documents.”
This is much better than making something up.
Preventing Hallucinations
RAG can reduce hallucinations, but it doesn't magically eliminate them.
A strong implementation should instruct the model to use the retrieved context and avoid unsupported claims.
For example:
You are an internal company assistant.
Answer the user's question using only the provided context.
If the answer cannot be found in the context,
clearly state that the information is unavailable.
Do not invent policies, numbers, dates, or procedures.
You can also add additional validation and evaluation layers.
Adding Sources to the Answer
There's another useful feature we can add.
Instead of simply returning:
“You can carry forward 10 leaves.”
we can return:
“You can carry forward up to 10 unused paid leaves.”
Source: Leave Policy, Page 7
This makes the system easier to trust.
The user can verify where the answer came from.
For enterprise applications, source attribution can be extremely valuable.
Access Control Is Critical
Here's a problem that's easy to overlook.
Imagine the company has:
Public Documents
Employee Documents
Management Documents
Finance Documents
Legal Documents
Not every employee should be able to search everything.
A RAG system therefore needs access control.
For example:
Employee
↓
Authentication
↓
Permission Check
↓
Allowed Documents
↓
Vector Search
↓
LLM
Metadata filtering can help.
For example:
{
"department": "HR",
"access_level": "employee"
}
The retrieval system can then restrict searches based on the user's permissions.
Never assume that because an LLM can retrieve something, the user should be allowed to see it.
What About Updated Documents?
Company policies change.
Suppose the old leave policy says:
Carry forward up to 10 days.
Six months later, the company changes it to:
Carry forward up to 15 days.
If we don't update the knowledge base, the AI may return outdated information.
A production RAG system therefore needs a document ingestion pipeline:
New Document
↓
Extract Text
↓
Chunk
↓
Generate Embeddings
↓
Update Vector DB
↓
Remove / Version Old Data
This is one reason RAG is often preferable to relying on a model's static knowledge for frequently changing business information.
RAG vs Fine-Tuning
A common question is:
“Why not just fine-tune the model with company documents?”
Fine-tuning and RAG solve different problems.
RAG is useful when:
- Information changes frequently.
- You need access to private documents.
- You need source citations.
- You need to update knowledge without retraining.
- You need document-level access control.
Fine-tuning is useful when:
- You want to change model behavior.
- You need a particular output style.
- You need specialized task performance.
- You have a suitable training dataset.
In many real applications, you don't have to choose only one.
You can combine techniques depending on the problem.
Improving Retrieval Quality
The quality of your final answer is heavily influenced by the quality of retrieval.
If the wrong documents are retrieved:
Bad Retrieval
↓
Wrong Context
↓
Potentially Wrong Answer
Some ways to improve retrieval include:
Better chunking
Preserve meaningful sections.
Metadata filtering
Restrict results by department, document type, date, permissions, etc.
Hybrid search
Combine semantic search with keyword-based search.
Reranking
Retrieve a larger candidate set and then rank the most relevant results.
Query transformation
Rewrite or expand the user's question before searching.
For example:
User:
"What happens to unused leaves?"
↓
Search Query:
"paid leave carry-forward policy unused leave balance"
Production Architecture
A production-ready version might look like this:
USER
│
↓
┌────────────────┐
│ Authentication │
└───────┬────────┘
↓
┌────────────────┐
│ API Gateway │
└───────┬────────┘
↓
┌────────────────┐
│ RAG Application│
└───────┬────────┘
│
┌────────────┼─────────────┐
↓ ↓ ↓
Query Engine Permissions Conversation
│ State
↓
┌─────────────┐
│ Vector DB │
└──────┬──────┘
↓
Relevant Context
│
↓
┌─────────────┐
│ LLM │
└──────┬──────┘
↓
Validation
↓
Answer + Sources
Behind the scenes, you would also want:
- Logging
- Monitoring
- Error handling
- Rate limiting
- Cost tracking
- Evaluation
- Data encryption
- Access control
Common RAG Mistakes
1. Making chunks too large
Large chunks can introduce irrelevant information.
2. Making chunks too small
Tiny chunks may lose important context.
3. Ignoring metadata
Metadata can dramatically improve retrieval and access control.
4. Sending too much context to the LLM
More context doesn't always mean better answers.
Irrelevant information can actually make responses worse.
5. Assuming RAG eliminates hallucinations
It doesn't.
You still need good prompts, validation, retrieval evaluation, and application-level controls.
6. Ignoring permissions
Private company data requires strict access control.
7. Never evaluating retrieval quality
You should test whether the system is actually retrieving the correct information.
How Would This Work in a Real Company?
Let's return to our original example.
An employee asks:
“How many paid leaves can I carry forward?”
The system processes the request:
1. Authenticate employee
↓
2. Receive question
↓
3. Generate query embedding
↓
4. Search permitted documents
↓
5. Retrieve leave-policy sections
↓
6. Send context to LLM
↓
7. Generate answer
↓
8. Attach source
↓
9. Return response
The employee gets:
You can carry forward up to 10 unused paid leaves to the next year, according to the company's leave policy.
Source: Employee Leave Policy — Page 7
What previously required searching through documents or contacting HR can now happen in seconds.
That's the practical value of RAG.
Final Takeaway
RAG isn't simply:
“Put documents into a vector database and connect an LLM.”
A reliable RAG application is a complete software system.
You need to think about:
- Document processing
- Chunking
- Embeddings
- Vector databases
- Retrieval
- Prompt design
- LLM selection
- Source attribution
- Access control
- Data freshness
- Evaluation
- Monitoring
- Security
The LLM is only one part of the architecture.
The real engineering challenge is building a system that retrieves the right information, gives the model the right context, and produces an answer that users can trust and verify.
What's Next?
In a follow-up article, we can take this concept further and build a complete RAG application using Python, FastAPI, a vector database, and an LLM, including document ingestion and a simple API.
If you're building a RAG application yourself, the most important question isn't:
“Which LLM should I use?”
Start with:
“What information does my application need, who is allowed to access it, and how can I reliably retrieve it?”
Once you answer those questions, choosing the rest of the architecture becomes much easier.
About NeuroML.ai
NeuroML.ai is an AI and software product engineering company helping startups, enterprises, agencies, and growing businesses build, modernize, and scale digital products.
We work across AI/ML, Generative AI, RAG systems, AI agents, SaaS, custom software, web and mobile applications, automation, cloud, DevOps, and dedicated development teams.
Our focus is simple: turn ideas and business problems into production-ready software.
If you're exploring RAG, AI agents, or AI-powered software, we'd love to hear what you're building.
Top comments (0)