LangChain for Absolute Beginners - Part 4: RAG - Teaching Your Agent to Read Documents=

Every model has a knowledge cutoff, and none of them have ever seen your company's internal wiki, your PDFs, or last week's meeting notes. Retrieval-Augmented Generation (RAG) solves this by fetching relevant chunks of your documents at question time and feeding them into the model's context before it answers.

By the end of this article, your agent from Part 3 will be able to answer questions about a document it was never trained on.

Recap: The Series So Far

1. Part 1 - What LangChain is & your first script
2. Part 2 - Prompts, models, and LCEL chains
3. Part 3 - Tools and your first agent
4. Part 4 (this article) - RAG: reading your own documents
5. Part 5 - Memory and middleware
6. Part 6 - Debugging and observing agents with LangSmith

The RAG Pipeline, Conceptually

RAG has two phases:

1. Indexing (done once, offline): Load your documents → split them into small chunks → convert each chunk into a numeric vector ("embedding") → store those vectors in a vector store.
2. Retrieval (done on every question): Convert the user's question into a vector the same way → find the stored chunks whose vectors are most similar → hand those chunks to the model as extra context.

The model never "reads your whole document." It only ever sees the handful of chunks that were judged most relevant to the specific question - which is exactly why chunking and retrieval quality matter so much.

Step 1: Install What You Need

pip install langchain langchain-openai langchain-text-splitters langchain-community

Step 2: Load and Split a Document

Let's index a plain text file. (The same pattern works for PDFs, web pages, and more - only the loader class changes.)

from langchain_community.document_loaders import TextLoader
from langchain_text_splitters import RecursiveCharacterTextSplitter

loader = TextLoader("company_faq.txt")
docs = loader.load()

splitter = RecursiveCharacterTextSplitter(
    chunk_size=500,
    chunk_overlap=50,
)
doc_splits = splitter.split_documents(docs)

print(f"Split into {len(doc_splits)} chunks")

Why split at all? Two reasons: embedding models work better on focused, shorter passages than on entire documents, and it keeps the context you eventually feed the model tight and relevant instead of bloated. chunk_overlap=50 keeps a little context from the end of one chunk carried into the next, so you don't awkwardly cut a sentence in half at a chunk boundary.

Step 3: Embed and Store

from langchain_core.vectorstores import InMemoryVectorStore
from langchain_openai import OpenAIEmbeddings

vectorstore = InMemoryVectorStore.from_documents(
    documents=doc_splits,
    embedding=OpenAIEmbeddings(),
)

retriever = vectorstore.as_retriever(search_kwargs={"k": 3})

InMemoryVectorStore is perfect for learning and small projects - no separate database to run. For production apps with thousands of documents, you'd swap this one line for something like Chroma, Pinecone, or FAISS; the rest of your pipeline doesn't change, which is the same "swap the component, keep the interface" pattern we saw with chat models in Part 1.

search_kwargs={"k": 3} tells the retriever to return the 3 most relevant chunks per question - a starting point worth tuning for your own data.

Step 4: Turn Retrieval Into a Tool

This is the key idea that connects RAG back to what you learned in Part 3: retrieval is just another tool. Instead of hardcoding "always search the docs," we let the agent decide when a question actually needs your documents versus when it can answer directly.

from langchain.tools import tool

@tool
def search_company_docs(query: str) -> str:
    """Search the company FAQ document for information relevant to the query."""
    results = retriever.invoke(query)
    return "\n\n".join(doc.page_content for doc in results)

Step 5: Build the RAG Agent

from dotenv import load_dotenv
from langchain.agents import create_agent

load_dotenv()

agent = create_agent(
    "gpt-5",
    tools=[search_company_docs],
    system_prompt=(
        "You are a helpful support assistant. "
        "Use the search_company_docs tool to answer questions about company policy. "
        "If the docs don't contain the answer, say so honestly instead of guessing."
    ),
)

response = agent.invoke(
    {"messages": [{"role": "user", "content": "What is our refund policy?"}]}
)

print(response["messages"][-1].content)

Notice the instruction "if the docs don't contain the answer, say so honestly instead of guessing" in the system prompt. This is one of the most important lines you'll write in any RAG app - without it, models will often confidently invent a plausible-sounding answer instead of admitting the retrieved chunks didn't cover the question.

A Quick Sanity Check

Before trusting a RAG pipeline, always look at what got retrieved, not just the final answer:

results = retriever.invoke("What is our refund policy?")
for i, doc in enumerate(results, 1):
    print(f"--- Chunk {i} ---")
    print(doc.page_content[:200], "...\n")

If the printed chunks don't actually contain refund information, the problem is in your indexing (chunk size, overlap, or embedding choice) - not in the model. This is the single most common debugging step for RAG apps, and it's worth doing before you touch the prompt.

What's Next

Your agent can now reason (Part 3) and ground its answers in your own data (Part 4). In Part 5, we added memory, so it remembers earlier turns in a conversation, and middleware, so you can enforce rules like "always ask for confirmation before calling a sensitive tool."

---

This is Part 4 of a 6-part LangChain beginner series.