How RAG Works: Building an AI Assistant That Can Answer Questions from Company Documents | NeuroML.ai

neuroMl ai — Wed, 19 Aug 2026 07:25:16 +0000

Imagine joining a company with hundreds or even thousands of internal documents.

There are HR policies, employee handbooks, product documentation, technical guides, compliance documents, SOPs, and internal processes.

Now an employee asks:

“How many unused paid leaves can I carry forward to next year?”

You could search through folders, open PDFs, use Ctrl+F, or ask the HR team.

But what if an AI assistant could answer the question in seconds — using the company's actual documents instead of guessing?

That's where Retrieval-Augmented Generation (RAG) becomes useful.

In this article, we'll build a simplified version of a real-world RAG system and understand what's happening behind the scenes.

What Problem Are We Solving?

Let's imagine a company has this information:

/company-documents
    ├── employee-handbook.pdf
    ├── leave-policy.pdf
    ├── health-insurance.pdf
    ├── remote-work-policy.pdf
    ├── travel-policy.pdf
    └── expense-policy.pdf

An employee asks:

“Can I carry forward unused paid leaves to next year?”

A general-purpose LLM may know something about leave policies in general.

But it doesn't automatically know this company's specific policy.

We need to give the model access to the company's private knowledge.

That's the problem RAG solves.

What Is RAG?

RAG stands for Retrieval-Augmented Generation.

The basic idea is simple:

User Question
      ↓
Retrieve Relevant Information
      ↓
Give Information to LLM
      ↓
Generate Answer

Instead of asking the LLM to answer entirely from what it already knows, we first retrieve relevant information from our own knowledge base.

The complete flow looks like this:

                User Question
                      ↓
              Generate Embedding
                      ↓
              Search Vector DB
                      ↓
             Retrieve Documents
                      ↓
          Relevant Context Found
                      ↓
             Context + Question
                      ↓
                    LLM
                      ↓
               Final Answer

This makes the LLM much more useful for private or frequently changing information.

Step 1: Collect the Documents

Our first step is to collect the company's documents.

For example:

HR Policy
Product Documentation
SOPs
Legal Documents
Technical Documentation
Employee Handbook

These documents might come from:

PDFs
Word documents
Websites
Databases
Cloud storage
Internal knowledge bases

For our example, we'll assume the documents are PDFs.

Step 2: Extract the Text

The PDF isn't immediately useful to the LLM.

We first need to extract the text.

Conceptually:

documents = load_documents("./company-documents")

The result might look like:

Document 1:
Employee Leave Policy

Employees are entitled to 24 paid leaves per year...

Unused paid leaves may be carried forward up to a
maximum of 10 days...

Now we have machine-readable content.

Step 3: Split the Documents into Chunks

Here's an important part of building a RAG system.

We usually don't want to put an entire 100-page document into every LLM request.

Instead, we split it into smaller pieces called chunks.

For example:

100-page PDF
      ↓
    Chunks
      ↓
┌───────────────┐
│ Chunk 1       │
├───────────────┤
│ Chunk 2       │
├───────────────┤
│ Chunk 3       │
├───────────────┤
│ Chunk 4       │
├───────────────┤
│ ...           │
└───────────────┘

A simple implementation could look like:

from langchain_text_splitters import RecursiveCharacterTextSplitter

splitter = RecursiveCharacterTextSplitter(
    chunk_size=800,
    chunk_overlap=100
)

chunks = splitter.split_documents(documents)

The overlap helps preserve context between neighboring chunks.

However, chunking isn't simply about choosing a number like 500 or 1,000 characters.

Good chunking should consider the structure of the information.

For example, keeping a policy heading together with its explanation is generally more useful than splitting them apart.

Step 4: Convert Text into Embeddings

Now we have chunks of text.

But how do we search them intelligently?

This is where embeddings come in.

An embedding converts text into a numerical representation — a vector.

For example:

"Employees can carry forward 10 unused leaves"
                 ↓
        Embedding Model
                 ↓
[0.12, -0.43, 0.81, 0.17, ...]

The important idea is that text with similar meaning tends to have vectors that are close together in vector space.

For example:

"How many leaves can I carry forward?"

should be semantically close to:

"Employees may carry forward up to 10 unused paid leaves."

even though the wording is different.

That's why vector search is so useful for RAG.

Step 5: Store the Embeddings in a Vector Database

We need somewhere to store these vectors.

Popular vector databases include:

Qdrant
Pinecone
Weaviate
Milvus
Chroma

Our architecture now looks like:

Documents
    ↓
Text Extraction
    ↓
Chunking
    ↓
Embeddings
    ↓
Vector Database

Each stored record can contain more than just the vector.

For example:

{
  "text": "Employees may carry forward up to 10 unused paid leaves.",
  "document": "leave-policy.pdf",
  "page": 7,
  "department": "HR"
}

That metadata becomes extremely useful later.

Step 6: A User Asks a Question

Now an employee asks:

“How many paid leaves can I carry forward?”

We convert that question into an embedding as well.

User Question
      ↓
Embedding Model
      ↓
Question Vector

Then we search the vector database for the most relevant chunks.

Conceptually:

results = vector_store.similarity_search(
    "How many paid leaves can I carry forward?",
    k=5
)

The database might return:

Result 1
Leave Policy — Page 7
"Employees may carry forward up to 10 unused paid leaves."

Result 2
Employee Handbook — Page 22
"Unused leave is subject to company leave policy."

Result 3
Leave Policy — Page 8
"Carry-forward balances are reset according to..."

Now we have relevant context.

Step 7: Give the Context to the LLM

This is where generation happens.

Instead of asking:

LLM:
"How many paid leaves can employees carry forward?"

we provide the retrieved information:

Context:
Employees may carry forward up to 10 unused paid leaves.

Question:
How many paid leaves can I carry forward?

The LLM can then generate:

Employees can carry forward up to 10 unused paid leaves to the next year, according to the company's leave policy.

This is the fundamental idea behind RAG.

The Complete Pipeline

Our system now looks like this:

                     USER
                       │
                       ↓
              "How many leaves
               can I carry?"
                       │
                       ↓
                Query Embedding
                       │
                       ↓
              ┌─────────────────┐
              │  Vector Search  │
              └────────┬────────┘
                       ↓
             Relevant Documents
                       │
                       ↓
              ┌─────────────────┐
              │      LLM        │
              └────────┬────────┘
                       ↓
                  Final Answer
                       │
                       ↓
             Source: Leave Policy

This is a basic RAG application.

But a production system needs more.

What Happens If the Answer Doesn't Exist?

This is one of the most important parts of a real-world RAG system.

Suppose an employee asks:

“Does the company provide free international travel insurance for personal vacations?”

But the company documents don't contain this information.

A poorly designed system might generate a confident answer.

That's dangerous.

A better system should recognize that there isn't enough evidence.

For example:

“I couldn't find information about personal international travel insurance in the available company documents.”

This is much better than making something up.

Preventing Hallucinations

RAG can reduce hallucinations, but it doesn't magically eliminate them.

A strong implementation should instruct the model to use the retrieved context and avoid unsupported claims.

For example:

You are an internal company assistant.

Answer the user's question using only the provided context.

If the answer cannot be found in the context,
clearly state that the information is unavailable.

Do not invent policies, numbers, dates, or procedures.

You can also add additional validation and evaluation layers.

Adding Sources to the Answer

There's another useful feature we can add.

Instead of simply returning:

“You can carry forward 10 leaves.”

we can return:

“You can carry forward up to 10 unused paid leaves.”

Source: Leave Policy, Page 7

This makes the system easier to trust.

The user can verify where the answer came from.

For enterprise applications, source attribution can be extremely valuable.

Access Control Is Critical

Here's a problem that's easy to overlook.

Imagine the company has:

Public Documents
Employee Documents
Management Documents
Finance Documents
Legal Documents

Not every employee should be able to search everything.

A RAG system therefore needs access control.

For example:

Employee
   ↓
Authentication
   ↓
Permission Check
   ↓
Allowed Documents
   ↓
Vector Search
   ↓
LLM

Metadata filtering can help.

For example:

{
  "department": "HR",
  "access_level": "employee"
}

The retrieval system can then restrict searches based on the user's permissions.

Never assume that because an LLM can retrieve something, the user should be allowed to see it.

What About Updated Documents?

Company policies change.

Suppose the old leave policy says:

Carry forward up to 10 days.

Six months later, the company changes it to:

Carry forward up to 15 days.

If we don't update the knowledge base, the AI may return outdated information.

A production RAG system therefore needs a document ingestion pipeline:

New Document
     ↓
Extract Text
     ↓
Chunk
     ↓
Generate Embeddings
     ↓
Update Vector DB
     ↓
Remove / Version Old Data

This is one reason RAG is often preferable to relying on a model's static knowledge for frequently changing business information.

RAG vs Fine-Tuning

A common question is:

“Why not just fine-tune the model with company documents?”

Fine-tuning and RAG solve different problems.

RAG is useful when:

Information changes frequently.
You need access to private documents.
You need source citations.
You need to update knowledge without retraining.
You need document-level access control.

Fine-tuning is useful when:

You want to change model behavior.
You need a particular output style.
You need specialized task performance.
You have a suitable training dataset.

In many real applications, you don't have to choose only one.

You can combine techniques depending on the problem.

Improving Retrieval Quality

The quality of your final answer is heavily influenced by the quality of retrieval.

If the wrong documents are retrieved:

Bad Retrieval
      ↓
Wrong Context
      ↓
Potentially Wrong Answer

Some ways to improve retrieval include:

Better chunking

Preserve meaningful sections.

Metadata filtering

Restrict results by department, document type, date, permissions, etc.

Hybrid search

Combine semantic search with keyword-based search.

Reranking

Retrieve a larger candidate set and then rank the most relevant results.

Query transformation

Rewrite or expand the user's question before searching.

For example:

User:
"What happens to unused leaves?"

        ↓

Search Query:
"paid leave carry-forward policy unused leave balance"

Production Architecture

A production-ready version might look like this:

                         USER
                           │
                           ↓
                  ┌────────────────┐
                  │ Authentication │
                  └───────┬────────┘
                          ↓
                  ┌────────────────┐
                  │   API Gateway  │
                  └───────┬────────┘
                          ↓
                  ┌────────────────┐
                  │ RAG Application│
                  └───────┬────────┘
                          │
             ┌────────────┼─────────────┐
             ↓            ↓             ↓
        Query Engine   Permissions   Conversation
             │                         State
             ↓
       ┌─────────────┐
       │ Vector DB   │
       └──────┬──────┘
              ↓
        Relevant Context
              │
              ↓
       ┌─────────────┐
       │     LLM     │
       └──────┬──────┘
              ↓
        Validation
              ↓
       Answer + Sources

Behind the scenes, you would also want:

Logging
Monitoring
Error handling
Rate limiting
Cost tracking
Evaluation
Data encryption
Access control

Common RAG Mistakes

1. Making chunks too large

Large chunks can introduce irrelevant information.

2. Making chunks too small

Tiny chunks may lose important context.

3. Ignoring metadata

Metadata can dramatically improve retrieval and access control.

4. Sending too much context to the LLM

More context doesn't always mean better answers.

Irrelevant information can actually make responses worse.

5. Assuming RAG eliminates hallucinations

It doesn't.

You still need good prompts, validation, retrieval evaluation, and application-level controls.

6. Ignoring permissions

Private company data requires strict access control.

7. Never evaluating retrieval quality

You should test whether the system is actually retrieving the correct information.

How Would This Work in a Real Company?

Let's return to our original example.

An employee asks:

“How many paid leaves can I carry forward?”

The system processes the request:

1. Authenticate employee
             ↓
2. Receive question
             ↓
3. Generate query embedding
             ↓
4. Search permitted documents
             ↓
5. Retrieve leave-policy sections
             ↓
6. Send context to LLM
             ↓
7. Generate answer
             ↓
8. Attach source
             ↓
9. Return response

The employee gets:

You can carry forward up to 10 unused paid leaves to the next year, according to the company's leave policy.

Source: Employee Leave Policy — Page 7

What previously required searching through documents or contacting HR can now happen in seconds.

That's the practical value of RAG.

Final Takeaway

RAG isn't simply:

“Put documents into a vector database and connect an LLM.”

A reliable RAG application is a complete software system.

You need to think about:

Document processing
Chunking
Embeddings
Vector databases
Retrieval
Prompt design
LLM selection
Source attribution
Access control
Data freshness
Evaluation
Monitoring
Security

The LLM is only one part of the architecture.

The real engineering challenge is building a system that retrieves the right information, gives the model the right context, and produces an answer that users can trust and verify.

What's Next?

In a follow-up article, we can take this concept further and build a complete RAG application using Python, FastAPI, a vector database, and an LLM, including document ingestion and a simple API.

If you're building a RAG application yourself, the most important question isn't:

“Which LLM should I use?”

Start with:

“What information does my application need, who is allowed to access it, and how can I reliably retrieve it?”

Once you answer those questions, choosing the rest of the architecture becomes much easier.

About NeuroML.ai

NeuroML.ai is an AI and software product engineering company helping startups, enterprises, agencies, and growing businesses build, modernize, and scale digital products.

We work across AI/ML, Generative AI, RAG systems, AI agents, SaaS, custom software, web and mobile applications, automation, cloud, DevOps, and dedicated development teams.

Our focus is simple: turn ideas and business problems into production-ready software.

If you're exploring RAG, AI agents, or AI-powered software, we'd love to hear what you're building.

DEV Community: neuroMl ai

How RAG Works: Building an AI Assistant That Can Answer Questions from Company Documents | NeuroML.ai

What Problem Are We Solving?

What Is RAG?

Step 1: Collect the Documents

Step 2: Extract the Text

Step 3: Split the Documents into Chunks

Step 4: Convert Text into Embeddings

Step 5: Store the Embeddings in a Vector Database

Step 6: A User Asks a Question

Step 7: Give the Context to the LLM

The Complete Pipeline

What Happens If the Answer Doesn't Exist?

Preventing Hallucinations

Adding Sources to the Answer

Access Control Is Critical

What About Updated Documents?

RAG vs Fine-Tuning

RAG is useful when:

Fine-tuning is useful when:

Improving Retrieval Quality

Better chunking

Metadata filtering

Hybrid search

Reranking

Query transformation

Production Architecture

Common RAG Mistakes

1. Making chunks too large

2. Making chunks too small

3. Ignoring metadata

4. Sending too much context to the LLM

5. Assuming RAG eliminates hallucinations

6. Ignoring permissions

7. Never evaluating retrieval quality

How Would This Work in a Real Company?

Final Takeaway

What's Next?

About NeuroML.ai