Artificial Intelligence applications are rapidly moving beyond simple question-answering systems. Modern enterprise AI assistants need to understand internal documents, retrieve accurate information, and provide reliable answers based on company knowledge.
I started building agentic AI systems from scratch — not to theorize about them, but to actually write the code, break it, and understand it from the inside out. This post walks through the second project in that journey: a Retrieval-Augmented Generation (RAG) system, what I built, what tripped me up, and where I'm taking it next.
What is RAG?
Retrieval-Augmented Generation combines two capabilities:
Retrieval: The system searches a knowledge base and finds relevant information related to the user's question.
Generation: The retrieved information is provided as context to an LLM, which generates a response based on that knowledge.
Instead of asking an LLM to remember everything, RAG allows the model to access external knowledge dynamically.
Technologies Used
Python, LangChain, Ollama (Local LLM), Embeddings, Vector Database, FastAPI.
What I Built
The core idea is to build a system that answers questions using only the content of a document, rather than whatever the underlying model already "knows."
Documents → Text Splitter → Embeddings → Vector Store → Retriever → LLM → Answer
Document loading — a company policy document, loaded with LangChain's TextLoader
Text Splitting — split into 200-character chunks with 50-character overlap, using RecursiveCharacterTextSplitter
Create Embeddings — generated with sentence-transformers/all-MiniLM-L6-v2 via HuggingFaceEmbeddings
Vector store — persisted in Chroma
Retrieval — top-2 most relevant chunks pulled per question
Generation — a strict prompt template that instructs the model to answer only from the retrieved context, run through Ollama's tinyllama.
Lessons Learned
Building a RAG system taught me that successful AI applications are not only about selecting a powerful LLM. The quality of the final answer depends heavily on:
Document quality
Chunking strategy
Retrieval accuracy
Prompt design
Evaluation methods
Conclusion
RAG provides a practical foundation for building enterprise AI assistants that can use private knowledge while reducing hallucination risks. This project became the foundation for my next experiments with multi-agent workflows, MCP servers, and autonomous AI systems.
Source Code: https://github.com/srirdeevi/agentic-ai-portfolio
Top comments (1)
The use of a Retrieval-Augmented Generation (RAG) system to build an AI assistant that can understand internal documents and provide reliable answers is a fascinating approach, and I appreciate the breakdown of the system's components, such as the Text Splitter and Vector Store. The emphasis on the importance of document quality, chunking strategy, and prompt design in determining the final answer's quality resonates with my own experience in working with language models. I'm curious to know more about how you evaluated the retrieval accuracy and how you plan to improve it in future iterations, especially considering the trade-offs between using a local LLM like Ollama versus more powerful but potentially more hallucination-prone models.