DEV Community: Piyush Kumar Soni

RAG for Beginners: Build a Document Q&A System with LangChain (Full Code)

Piyush Kumar Soni — Sun, 19 Jul 2026 08:11:29 +0000

RAG for Beginners: Build a Document Q&A System with LangChain (Full Code)

Tags: #ai #langchain #python #tutorial

LLMs don't know your private documents. ChatGPT can write you an essay about quantum computing, but it has zero idea what's in your company's internal wiki, your client reports, or that PDF you saved last Tuesday — unless you paste it in yourself, every single time.

RAG (Retrieval-Augmented Generation) fixes this. This post walks through the concept and a full working code example you can run today.

The Architecture

Your Documents  →  Split into chunks  →  Convert to embeddings  →  Stored in a vector database
                                                                              ↓
User's Question  →  Convert to embedding  →  Find closest matching chunks  →  Send question + chunks to LLM  →  Answer

Two steps, really: retrieval (find the relevant pieces of your data) and generation (let the LLM answer using those pieces instead of guessing from training data alone).

Setup

pip install langchain langchain-community langchain-openai faiss-cpu

You'll also need an OPENAI_API_KEY environment variable set:

export OPENAI_API_KEY="your-key-here"

Full Working Example — Text Document

from langchain_community.document_loaders import TextLoader
from langchain.text_splitter import RecursiveCharacterTextSplitter
from langchain_community.vectorstores import FAISS
from langchain_openai import OpenAIEmbeddings, ChatOpenAI
from langchain.chains import RetrievalQA

# 1. Load your document
loader = TextLoader("client_seo_report.txt")
documents = loader.load()

# 2. Split into chunks — LLMs work better with focused chunks than walls of text
splitter = RecursiveCharacterTextSplitter(
    chunk_size=500,
    chunk_overlap=50  # overlap prevents cutting a key sentence in half
)
chunks = splitter.split_documents(documents)

# 3. Convert chunks into embeddings and store them in a local vector index
embeddings = OpenAIEmbeddings()
vector_store = FAISS.from_documents(chunks, embeddings)

# 4. Wire up retrieval + generation
llm = ChatOpenAI(model="gpt-4o-mini", temperature=0)
qa_chain = RetrievalQA.from_chain_type(
    llm=llm,
    retriever=vector_store.as_retriever(search_kwargs={"k": 4})  # top 4 matching chunks
)

# 5. Ask a question about YOUR document
answer = qa_chain.invoke("What was the biggest traffic change this month?")
print(answer["result"])

That's a complete, runnable RAG pipeline in about 20 lines.

Swapping in a PDF Instead of Plain Text

Most real documents aren't .txt files. Swapping the loader is the only change needed:

from langchain_community.document_loaders import PyPDFLoader

loader = PyPDFLoader("client_seo_report.pdf")
documents = loader.load()

# everything else in the pipeline stays exactly the same

LangChain has loaders for most formats you'll actually run into — Docx2txtLoader for Word docs, CSVLoader for spreadsheets, WebBaseLoader for scraping a URL directly. Same downstream pipeline every time.

Why `chunk_size` and `chunk_overlap` Matter More Than They Look Like They Should

This trips up almost everyone the first time:

Chunk size too large → retrieval pulls back a lot of irrelevant surrounding text along with the useful part, and the LLM's answer gets muddier.
Chunk size too small → you lose context; a chunk might not contain enough surrounding information to be useful on its own.
No overlap → you can accidentally split a sentence's meaning exactly at the point where the important information was.

There's no universal "correct" number — 500 characters with 50 overlap is a reasonable starting point for report-style text, but experiment for your actual documents.

Common Mistakes (Beyond Chunking)

Expecting perfect retrieval on the first try. Sometimes the "closest match" by embedding similarity isn't actually the most relevant chunk for the question. This is normal — tuning retrieval quality is where most of the real engineering effort goes, well past this example.
Not filtering by metadata. If you're indexing documents from multiple clients or time periods, tag your chunks with metadata (client_name, date) so you can filter the search instead of searching everything every time:

vector_store.as_retriever(
    search_kwargs={"k": 4, "filter": {"client_name": "acme_corp"}}
)

Using the wrong k value. Too few retrieved chunks (k=1 or 2) and the LLM might not have enough context. Too many (k=10+) and you're paying for tokens the model doesn't need and diluting the relevant signal. k=3 to k=5 is a reasonable starting range.

Where This Goes Next

Once this basic pattern works, natural next steps:

Try a production-grade vector store (Pinecone, Chroma, or Weaviate) instead of local FAISS once you need this to scale or persist across sessions
Add metadata filtering as shown above once you're indexing more than one source
Experiment with hybrid search (combining keyword search with embedding similarity) — pure embedding search sometimes misses exact-match terms like product names or codes

I'm covering agents next — giving an LLM the ability to actually take actions instead of just answering questions. RAG is the foundation a lot of that builds on, so if this made sense, you're already most of the way there.

If you build this and hit a snag, drop a comment — happy to help debug.

I'm Piyush Kumar Soni, an AI Developer and LLM Integration Expert with 15+ years running digital agency operations. I write about practical AI tools for SEO and digital marketing.

🔗 LinkedIn · 💻 GitHub · 📰 Substack

Build Your First LLM Tool: A Beginner's Guide to LangChain

Piyush Kumar Soni — Sat, 18 Jul 2026 07:14:17 +0000

Build Your First LLM Tool: A Beginner’s Guide to LangChain

Building an LLM application can sound complicated, but your first working tool may require fewer than 15 lines of Python.

In this tutorial, we’ll build a simple AI explainer using LangChain and an OpenAI model.

Why Use LangChain Instead of Calling the OpenAI API Directly?

Calling an AI model directly is perfectly reasonable for a small script. You send a prompt, receive a response, and display it.

The complexity begins when your application needs:

Reusable prompt templates
Structured output
Multiple model providers
Documents or external data
Conversation history
Tools and agents
Logging and debugging

LangChain provides a standard interface for composing these pieces. It also makes switching between supported providers easier because your application does not need to be completely rewritten around each provider’s API. (Docs by LangChain)

Think of the raw API as an engine. LangChain gives you reusable parts for building the rest of the vehicle.

Install the Required Packages

pip install langchain langchain-openai python-dotenv

Create a .env file:

OPENAI_API_KEY=your_api_key_here

Never commit this file or expose your API key in a public repository. OpenAI recommends storing API keys securely rather than placing them directly inside source code. (OpenAI Platform)

Build a Basic LangChain Chain

Create a file named app.py:

from dotenv import load_dotenv
from langchain_core.output_parsers import StrOutputParser
from langchain_core.prompts import ChatPromptTemplate
from langchain_openai import ChatOpenAI

load_dotenv()

prompt = ChatPromptTemplate.from_template(
    "Explain {topic} to a beginner using one example."
)

model = ChatOpenAI(model="gpt-5-nano")
chain = prompt | model | StrOutputParser()

print(chain.invoke({"topic": "vector databases"}))

Run it:

python app.py

The chain has three simple steps:

Insert the topic into the prompt.
Send the formatted prompt to the model.
Convert the model response into a normal Python string.

The | operator connects each component into a readable pipeline. The current LangChain OpenAI integration is provided through the separate langchain-openai package and its ChatOpenAI class. (Docs by LangChain)

Where Should You Go Next?

Once this basic chain works, explore two important areas.

RAG, or Retrieval-Augmented Generation, lets your application retrieve information from PDFs, webpages, databases, or company documents before generating an answer.

Agents allow a model to decide when to call tools such as search functions, APIs, calculators, or custom Python functions. LangChain currently provides configurable agent-building functionality for connecting models with tools and application logic. (Docs by LangChain)

You can also turn your script into a web interface using Gradio and publish it through Hugging Face Spaces.

Here are two live examples from my own AI-tool experiments:

Start with one prompt and one model. Get that working first. Then add memory, retrieval, tools, and agents only when your application genuinely needs them.

About the author: Piyush Kumar Soni is an AI Developer, LLM Expert and Digital Growth Specialist building practical AI tools, automation systems and LLM-powered workflows.

Suggested Dev.to tags: ai, langchain, python, beginners

This article follows the Dev.to and topical-authority direction in your SERP plan, including the focus on LangChain development, Python-based AI tools and Hugging Face demos. Your supplied profile list confirms the two live Hugging Face Space URLs used above.

DEV Community: Piyush Kumar Soni

RAG for Beginners: Build a Document Q&A System with LangChain (Full Code)

RAG for Beginners: Build a Document Q&A System with LangChain (Full Code)

The Architecture

Setup

Full Working Example — Text Document

Swapping in a PDF Instead of Plain Text

Why chunk_size and chunk_overlap Matter More Than They Look Like They Should

Common Mistakes (Beyond Chunking)

Where This Goes Next

Build Your First LLM Tool: A Beginner's Guide to LangChain

Build Your First LLM Tool: A Beginner’s Guide to LangChain

Why Use LangChain Instead of Calling the OpenAI API Directly?

Install the Required Packages

Build a Basic LangChain Chain

Where Should You Go Next?

Why `chunk_size` and `chunk_overlap` Matter More Than They Look Like They Should