Hey everyone,
Over the last few months, I’ve been heavily experimenting with local LLMs. Running a model via Ollama is incredibly easy now, but I realized that building a production-ready, full-stack RAG (Retrieval-Augmented Generation) application around it—one that non-technical users or businesses can actually use without data privacy concerns—is still a massive headache.
Piecing together the backend, frontend, vector databases, orchestration, and making sure everything works smoothly in Docker takes days, if not weeks, of boilerplate setup.
To solve this for my own projects (and for clients who refuse to send their sensitive documents to OpenAI), I built VaultRAG. I wanted to share the architecture and how I put it together.
🛠️ The Tech Stack
I wanted a stack that is modern, asynchronous, and completely isolated from the internet:
LLM & Embeddings: Ollama (Running locally, mostly testing with Llama 3.1 and Qwen for text, and nomic-embed-text for embeddings).
Backend: Python & FastAPI. It handles the async API routing and document uploads seamlessly.
RAG Orchestration: LangChain. Handled the PDF parsing, text chunking, and vector store retrieval.
Frontend: React (Tailwind CSS for styling). Kept it clean, resembling a modern chat interface so enterprise users feel right at home.
Deployment: Docker & Docker Compose. Everything spins up together.
⚙️ How It Works Under the Hood
Document Ingestion: The user uploads a PDF via the React interface. FastAPI receives it, LangChain processes the text, splits it into chunks (using RecursiveCharacterTextSplitter to maintain context), and creates embeddings via Ollama.
Vector Storage: The embeddings are stored locally.
Retrieval & Generation: When a user asks a question, the backend retrieves the most relevant chunks, constructs the prompt, and streams the response back from the local LLM via Server-Sent Events (SSE) for that real-time typing effect.
🚧 Biggest Challenges
Getting Docker networking to play nicely with Ollama was initially tricky. Ensuring the FastAPI container could communicate with the host's Ollama instance (or a containerized one) required specific network configurations.
Also, tweaking the chunk size and overlap in LangChain was crucial; too small, and the LLM loses context; too large, and inference slows down significantly on local hardware.
🚀 Try It Out
I decided to package this entire architecture into a ready-to-use boilerplate. If you are building AI tools for clients, participating in hackathons, or just want a private AI for your own documents without spending a week on setup, you can use this as your foundation.
If you want to see some of the open-source components/documentation, check out the GitHub repo: https://github.com/toprakkbozbeyy/vaultrag-enterprise-starter-kit
I’d love to hear your thoughts on the stack. Has anyone else found a better way to handle local vector storage without bloating the Docker image? Open to all feedback and architectural roasting!
Top comments (0)