Why Can't I Just Use ChatGPT to Understand My Project? An Introduction to RAG

Karthik DASARI — Wed, 04 Nov 2026 13:24:17 +0000

I recently joined a software project that had been running for almost 12 years.

As part of the onboarding process, the team shared a lot of documentation with me: architecture documents, technical specifications, workflows, and other project-related information.

I started going through them. Honestly, it was overwhelming. There was so much information to read, and manually going through document after document wasn't exactly an exciting experience.

One day, I was having a conversation with one of the senior developers on the team.

Senior Developer: "Have you gone through the documentation and got a good understanding of the project?"

Me: "I'm still going through it."

Senior Developer: "How are you reading all these documents?"

Me: "Manually, one by one."

He then mentioned that the team was building an AI assistant to make the onboarding process smoother for new joiners.

That immediately caught my attention.

Me: "So, can't I just use ChatGPT or Gemini for this?"

Senior Developer: "Not exactly. ChatGPT is a general-purpose LLM. It doesn't automatically know the internal details of our project. This is where RAG comes into play."

That conversation introduced me to a concept I wanted to understand better: Retrieval-Augmented Generation (RAG).

But why isn't a general-purpose LLM enough?

Why can't I just use ChatGPT?

Let's say I ask ChatGPT:

"How does authentication work in a Java application?"

It can explain authentication concepts, common approaches, and implementation patterns.

Now consider a different question:

"Where is authentication implemented in our project?"

The answer depends on our actual source code and internal documentation.

An LLM may understand authentication very well, but that doesn't mean it knows how authentication is implemented in our project. It doesn't automatically have access to our private project information.

Of course, we can upload documents or paste information into ChatGPT. But imagine doing that for a project with thousands of files and hundreds of documents.

We need a better approach: retrieve only the information relevant to the question and provide it to the LLM.

That's the fundamental idea behind RAG.

What exactly is RAG?

RAG stands for Retrieval-Augmented Generation.

Let's break it down:

Retrieval: Finding relevant information from an external knowledge source.
Augmented: Providing that information along with the user's question.
Generation: Using an LLM to generate an answer based on the provided information.

In simple terms, RAG connects an LLM with external knowledge so it can answer questions using information that isn't necessarily part of its training data.

But how does it actually work?

How does RAG work?

A basic RAG application has two major workflows:

Preparing the knowledge.
Retrieving information and generating an answer.

1. Preparing the knowledge

Before our AI assistant can answer questions about the project, it needs access to the project's information.

This could include architecture documents, technical specifications, API documentation, and other internal resources.

We need to prepare this information so that it can be searched efficiently. This process is called data ingestion.

There are four important concepts involved:

Document loading

First, we collect and load information from different sources into the system.

Chunking

Imagine an architecture document containing 100 pages. We don't need to retrieve the entire document whenever someone asks a question.

Instead, we divide it into smaller, manageable pieces called chunks. This makes it easier to find specific information.

Embeddings

How can a system find information based on meaning rather than just matching exact words?

This is where embeddings come in.

An embedding is a numerical representation of text that captures aspects of its meaning. It helps a system identify semantically similar pieces of information, even when they use different words.

Vector database

Once we convert the chunks into embeddings, we need to store them somewhere searchable.

A vector database stores these embeddings along with their associated information, such as the original text and metadata.

Now our project knowledge is prepared and ready to be searched.

2. Retrieving information and generating an answer

Let's say I ask our AI assistant:

"Where is authentication handled in our project?"

What happens next?

Step 1: Convert the question into an embedding

The user's question is converted into a numerical representation, just like the document chunks.

Step 2: Retrieve relevant information

The system searches the vector database to find chunks that are semantically similar to the question.

For example, it might retrieve relevant information from authentication documentation, login API documentation, or security configuration documents.

This is called retrieval.

Step 3: Provide context to the LLM

The RAG application combines two things:

The original question.
The relevant information retrieved from the knowledge base.

It sends both to the LLM.

Step 4: Generate the answer

The LLM uses the question and retrieved information as context to generate a response.

Instead of relying only on its general training knowledge, it now has access to information about our specific project.

And that's the basic RAG workflow!

Putting it all together

Let's look at the complete process:

Knowledge preparation (before users ask questions):

Project documents → Document loading → Chunking → Embeddings → Vector database

Question answering (when a user asks a question):

User question → Question embedding → Retrieve relevant chunks → Combine question and retrieved context → LLM → Generated answer

The important distinction is that knowledge preparation happens beforehand. We don't need to process all the project documents from scratch every time someone asks a question.

When new documentation is added or existing information changes, the knowledge base can be updated through the ingestion process.

What's next?

We now understand the basic idea behind RAG: retrieving relevant external information and providing it to an LLM to generate an answer.

But does this mean every answer generated by a RAG application will be correct?

Not necessarily.

What happens when the system retrieves irrelevant information? Why do LLMs sometimes generate answers that aren't supported by the provided context? How can we make the system more reliable?

In the next article, we'll explore these challenges through concepts such as hallucinations, prompt engineering, context management, hybrid search, and reranking.

That's where we move beyond understanding RAG to understanding how to build more reliable RAG applications.

DEV Community: Karthik DASARI