DEV Community

Pingredsai
Pingredsai

Posted on

I Built a Self-Hosted RAG Stack That Runs Without Cloud AI APIs

#ai #rag #selfhosted #python

Many teams want AI to answer questions from internal documents, but they cannot send those documents to a cloud service.

Legal teams have confidential case files. Clinics have patient data. Finance teams have client working papers. Manufacturers have proprietary process documentation. Some organizations operate on isolated networks.

For these teams, a cloud-first RAG stack is not an option.

I built LocalRAG to verify a simple idea:

A document question-answering system should be able to run inside a controlled environment, use local models, and return source references for every answer.

The project is open source and the current v0.1.0 release has been verified end to end.

What LocalRAG does

LocalRAG allows a user to upload documents and ask questions against them.

The verified flow is:

  1. Upload a PDF, DOCX, TXT, or Markdown document.
  2. Parse and split the document into chunks.
  3. Generate embeddings with a local bge-m3 model.
  4. Store vectors in Qdrant.
  5. Embed the user's question.
  6. Retrieve the most relevant chunks.
  7. Generate an answer with a local LLM.
  8. Return the answer with source references.

The application does not call a cloud AI API at runtime.

Initial installation still requires network access to pull Docker images and Ollama models. After that, the stack can run inside a local or air-gapped environment.

Architecture

LocalRAG uses three main containers:

Browser UI
    |
    v
FastAPI Backend
  - document parsing
  - chunking
  - RAG orchestration
  - REST API and static frontend
    |
    +--> Qdrant
    |      vector storage and search
    |
    +--> Ollama
           bge-m3 embeddings
           local LLM generation
Enter fullscreen mode Exit fullscreen mode

Why these components?

Layer Choice Reason
Backend FastAPI Easy to inspect, modify, and deploy
Vector database Qdrant One container, clean API, persistent storage
Embedding runtime Ollama + bge-m3 Local, multilingual, avoids a separate Python model stack
Generation runtime Ollama + Qwen2.5 Easy model switching from 1.5B to larger models
Frontend Plain HTML/CSS/JS No frontend build step
Deployment Docker Compose Reproducible and easy to run locally

I deliberately did not build a new model runtime, vector database, or agent engine.

The interesting engineering problem is not reinventing Ollama. It is making the complete workflow reproducible, observable, and usable in a private deployment.

Verified test

I tested the current release with:

  • Docker Desktop 4.95.0
  • Ollama 0.11.3
  • Qdrant
  • bge-m3 for embeddings
  • qwen2.5:3b for generation

The test document was a fictional employee handbook.

The system:

  1. Indexed it into three chunks.
  2. Received the question: “How many days of annual leave do full-time employees receive?”
  3. Retrieved the relevant policy text.
  4. Returned “15 days.”
  5. Displayed the source document and similarity score in the UI.

A health check returns:

{
  "status": "ok",
  "qdrant": "ok",
  "ollama": "ok",
  "llm_model": "qwen2.5:3b",
  "embed_model": "bge-m3"
}
Enter fullscreen mode Exit fullscreen mode

The repository includes screenshots and a 42-second demo video.

Running it 

git clone https://github.com/Pingredsai/local-rag.git
cd local-rag
cp .env.example .env
docker compose up -d

docker exec localrag-ollama ollama pull qwen2.5:3b
docker exec localrag-ollama ollama pull bge-m3
Enter fullscreen mode Exit fullscreen mode

Open:

http://localhost:8080
Enter fullscreen mode Exit fullscreen mode

Four problems I ran into

1. The official Ollama image was too large

The latest Docker image is convenient, but the unpacked image can approach 9 GB. In a slow network environment, that is a serious deployment problem.

I pinned the project to a smaller compatible Ollama release and documented the deployment tradeoff.

This is a reminder that runtime size and download behavior are part of the product experience, not just infrastructure details.

2. A slow Docker registry is not the same as a broken Docker installation

Docker Hub was extremely slow in my environment. The solution was a combination of a mirror, resumable pulls, and local model import in some cases.

I did not hard-code a third-party mirror into the project. Instead, the deployment notes describe the approach so operators can replace it.

3. The original streaming response did not show sources

The first implementation returned text from the streaming endpoint, while sources were only available from the non-streaming endpoint.

That made the UI look like a normal chatbot. The source references, which are essential in a business system, were invisible.

I changed the streaming endpoint to return source metadata in a response header. The frontend reads the header, streams the answer, and renders the source cards when the answer completes.

4. A 1.5B model was not reliable enough for the demo

The smaller model worked on short context but produced inconsistent responses on slightly longer documents.

I switched the demo to qwen2.5:3b and tightened the prompt and generation options.

Local AI is not automatically better just because it is smaller. Model size must match the hardware, use case, and required answer quality.

Current limitations

The first version is intentionally narrow.

It does not include:

  • OCR for scanned PDFs
  • Multi-tenant permissions
  • Multiple isolated knowledge bases
  • Audit logging
  • A reranker
  • Visual agent orchestration

The goal of v0.1.0 is to make the core path dependable:

upload -> index -> retrieve -> answer -> show sources
Enter fullscreen mode Exit fullscreen mode

I would rather state those boundaries clearly than imply that the project is already a production system.

What I plan to improve next

The next work is focused on the existing engineering path:

  • Improve PDF and DOCX parsing
  • Evaluate and refine chunking strategies
  • Improve retrieval quality and testing
  • Polish streaming answers and source rendering
  • Add clearer checks for ports, models, and data directories
  • Document backup, restore, and upgrade paths
  • Continue reducing deployment friction

Before adding more modules, I want the core path to be dependable:

upload -> index -> retrieve -> answer -> show sources
Enter fullscreen mode Exit fullscreen mode

Feedback and contributions

LocalRAG is an open-source project under active development.

If you work with local inference, RAG, or private deployment, feedback is welcome through GitHub issues and pull requests.

Repository:

https://github.com/Pingredsai/local-rag

The most useful reports are reproducible bugs, deployment issues, retrieval quality observations, and documentation improvements.

Top comments (2)

Collapse
 
sikamikanikobg profile image
Arsen Apostolov •

the part i keep hitting on my own rig is the one you put in problem one. the official ollama image unpacks close to nine gigabytes, and on a slow line that is the whole deployment, not the model. i pin a smaller release and document the tradeoff exactly the way you did, because the download behaviour is part of the product, not just the infra.

the streaming-sources fix is the detail that separates a demo from something a team would trust. a rag answer with no visible source reads like a chatbot, and the moment you put the document and the similarity score in the header, it stops being a guess. i run a small monitor that watches the ollama and qdrant containers per process (github.com/SikamikanikoBG/homelab-monitor) — the health check you show is the right shape, and the one thing i add is the vram on the embedding model, because bge-m3 is the quiet one that eats memory when the context grows.

Collapse
 
pingredsai profile image
Pingredsai •

Thanks — this is a very useful framing.

You're right that download behavior is part of the deployment experience, not just an infrastructure concern. I pinned a smaller Ollama release for the same reason and documented the tradeoff.

The source-header fix was the point where the UI stopped feeling like a generic chatbot. Once the document and similarity score are visible, the answer becomes verifiable instead of just plausible.

The per-process monitor is a good idea. Tracking embedding VRAM is especially relevant for bge-m3, since context growth is where it can quietly become expensive. I'll take a look at homelab-monitor and consider adding a diagnostic check around this.