Building a 100% Local RAG System on Kubernetes — No API Keys Required

Ahmed Nafies — Thu, 30 Jul 2026 23:00:55 +0000

Here's what we're building:

The itch I couldn't scratch

Last week I sat down to build a RAG system. You know, one of those "chat with your documents" things everyone's building these days.

I opened my first tutorial. It said: "Step 1: Sign up for OpenAI and grab your API key." Fine. "Step 2: Create a Pinecone account." Okay. "Step 3: Deploy to a managed vector database." Hmm. "Step 4: Set up your billing — here's a pricing calculator."

I stared at the calculator. Then at my $0 budget. Then back at the calculator.

Something felt off. Why does a "personal project" need a procurement department? Why does my data need to leave my laptop to answer a question about a document sitting on my laptop? And honestly — I just wanted to tinker. Not negotiate with three SaaS vendors before I could write a single line of code.

So I did what any reasonable engineer would do. I closed all twenty tabs, opened a terminal, and decided to build the whole thing from scratch. On my machine. With models running on my CPU. No cloud. No API keys. No credit card.

Here's how that ride went.

The game plan

Before writing a line of code, I sketched out what "no cloud" actually means:

Embeddings — gotta run locally. No calling OpenAI's embedding endpoint.
Vector search — needs to live inside something I already run. I'm not spinning up a separate vector database service.
The LLM — must fit in my laptop's RAM and run on CPU. I don't have a GPU lying around.
Deployment — real infrastructure patterns, not docker-compose up and pray. Something that would make sense in production, just scaled down.

What emerged was this stack:

┌────────────────────────────────────────────────────┐
│                 FastAPI Application                 │
├────────────────────────────────────────────────────┤
│  ┌──────────────┐  ┌──────────┐  ┌──────────────┐  │
│  │  Embedding   │  │ pgvector │  │  llama-cpp   │  │
│  │  Pipeline    │  │  Search  │  │  (Qwen 1.5B) │  │
│  └──────────────┘  └──────────┘  └──────────────┘  │
└────────────────────────────────────────────────────┘
                         │
                         ▼
               ┌──────────────────┐
               │   PostgreSQL     │
               │   + pgvector     │
               │   + HNSW index   │
               └──────────────────┘

The flow: someone asks a question → it gets turned into a 384-dimensional vector → PostgreSQL hunts for similar documents using cosine similarity → those results get stuffed into a prompt → the LLM answers → answer streams back. All inside Kubernetes. All local.

Every piece of this puzzle was chosen with one rule: it runs on my machine, or it doesn't make the cut.

Choosing the players

pgvector: the database you already have

I wanted vector search without installing Yet Another Database. pgvector is a PostgreSQL extension that adds vector columns, indexing, and similarity operators directly into Postgres. Just CREATE EXTENSION IF NOT EXISTS vector and you're in business.

It gives you HNSW indexes (fast approximate nearest neighbor search), a <=> cosine distance operator, and zero additional infrastructure. Your vector DB is just... your regular DB.

Qwen 2.5, living in my CPU

Here's where things got spicy. Most RAG tutorials assume you'll call GPT-4. But I had no GPU, no cloud credits, and a strong desire to keep everything local.

Enter Qwen2.5-1.5B-Instruct — a 1.5 billion parameter model that actually runs on CPU. Not fast, but fast enough. And the output quality? Genuinely surprising for its size.

The trick is serving it via llama-cpp-python, which exposes an OpenAI-compatible API at http://127.0.0.1:8001/v1. The magic of this: my application code uses the exact same OpenAIChatModel class you'd point at api.openai.com. It doesn't know — and doesn't care — that the model is running on the same machine, on a CPU, inside a container.

All it sees is a /v1/chat/completions endpoint. The fact that it's a 1.5B model crammed into a Docker container? That's between me and the container.

sentence-transformers: tiny but mighty

For embeddings, I picked all-MiniLM-L6-v2. It produces 384-dimensional vectors, runs fast on CPU, and the quality is solid for semantic search. It gets loaded once at startup and cached — no reloading per request:

from sentence_transformers import SentenceTransformer

@lru_cache
def get_model() -> SentenceTransformer:
    return SentenceTransformer("all-MiniLM-L6-v2")

def embed_text(text: str) -> list[float]:
    return get_model().encode(text).tolist()

Kubernetes on a laptop? Absolutely.

Running K8s locally sounds excessive until you try it. I used Colima — a lightweight k3s cluster that runs on 2 CPUs and 16GB of RAM. No Docker Desktop, no resource vampires. Just a clean K8s cluster on my Mac.

The benefit is real: Helm charts for declarative infra, Skaffold for the build-deploy loop, Helm hook Jobs that run migrations and seeding automatically. The same patterns I'd use in production, just at my desk.

Under the hood

Let's walk through the interesting bits. Not every line — just the parts that made me smile when they worked.

The database that thinks in vectors

The documents table stores each document alongside its 384-dimensional embedding vector. The HNSW index means similarity searches stay fast even as the corpus grows:

CREATE EXTENSION IF NOT EXISTS vector;

CREATE TABLE documents (
    id          uuid PRIMARY KEY DEFAULT gen_random_uuid(),
    title       varchar(500) NOT NULL,
    content     text NOT NULL,
    embedding   vector(384) NOT NULL,
    created_at  timestamptz DEFAULT now()
);

CREATE INDEX ON documents
    USING hnsw (embedding vector_cosine_ops)
    WITH (m = 16, ef_construction = 64);

The search that feels like magic

A stored procedure does the heavy lifting. You throw a vector at it, and it returns the closest matches ranked by similarity:

CREATE OR REPLACE FUNCTION match_documents (
    query_embedding vector(384),
    match_threshold float,
    match_count int
)
RETURNS TABLE (id uuid, title text, content text, similarity float)
LANGUAGE sql STABLE
AS $$
    SELECT
        documents.id,
        documents.title,
        documents.content,
        1 - (documents.embedding <=> query_embedding) AS similarity
    FROM documents
    WHERE documents.embedding <=> query_embedding < 1 - match_threshold
    ORDER BY documents.embedding <=> query_embedding
    LIMIT match_count;
$$;

That <=> is pgvector's cosine distance operator. 1 - distance gives us similarity. Clean, fast, entirely in SQL.

The application calls it through a parameterized SQLAlchemy query — no ORM magic, just raw SQL with safe parameters:

async def match_documents(
    session: AsyncSession,
    query_embedding: list[float],
    match_threshold: float = 0.5,
    match_count: int = 5,
) -> list[dict]:
    query = text("""
        SELECT * FROM match_documents(
            CAST(:query_embedding AS vector(384)),
            :match_threshold,
            :match_count
        )
    """)
    result = await session.execute(query, {
        "query_embedding": embedding_str,
        "match_threshold": match_threshold,
        "match_count": match_count,
    })
    return [{"id": str(r["id"]), "title": r["title"],
             "content": r["content"], "similarity": r["similarity"]}
            for r in result.mappings().all()]

The agent that thinks it's talking to OpenAI

This is my favorite part. Pydantic AI's Agent class is designed for OpenAI, but llama-cpp-python speaks the same protocol. So we just... point it at localhost:

from pydantic_ai import Agent
from pydantic_ai.models.openai import OpenAIChatModel
from pydantic_ai.providers.openai import OpenAIProvider

model = OpenAIChatModel(
    "qwen2.5-1.5b-instruct",
    provider=OpenAIProvider(
        base_url="http://127.0.0.1:8001/v1",
        api_key="not-needed",
    ),
)

agent = Agent(model=model, system_prompt=(
    "You are a helpful assistant. Answer questions based on "
    "the provided context. If the context does not contain "
    "enough information, say so."
))

The api_key="not-needed" line makes me happy every time I look at it. Take that, billing page.

Streaming: the one that nearly broke me

Server-Sent Events streaming should be straightforward. You yield chunks, the browser displays them. Simple.

Except Pydantic AI's stream_text() doesn't return deltas — it returns the accumulated text. Every chunk is the full response so far. If you send that to the browser, it keeps re-rendering the entire message.

The fix is deceptively simple. Track the previous value and compute the difference:

async def generate():
    yield f"event: sources\ndata: {json.dumps(sources)}\n\n"

    prev = ""
    async with agent.run_stream(prompt) as result:
        async for chunk in result.stream_text():
            delta = chunk[len(prev):]  # Just the new stuff
            prev = chunk
            yield f"event: delta\ndata: {json.dumps({'content': delta})}\n\n"

return StreamingResponse(generate(), media_type="text/event-stream")

Two lines of code that took two hours to figure out. But watching the text flow in, word by word, on the first successful stream — chef's kiss.

Ship it: Kubernetes all the way down

Three Helm charts, one skaffold run:

Chart	What's inside
`rag-supabase-db`	PostgreSQL StatefulSet + pgvector, 5Gi persistent disk
`rag-supabase`	FastAPI + llama-cpp running side-by-side, 3Gi model cache
`rag-supabase-ui`	Nginx serving the chat UI

The slick part: migrations and seeding aren't manual steps. They're Helm hook Jobs that fire automatically on every deploy:

apiVersion: batch/v1
kind: Job
metadata:
  name: {{ .Release.Name }}-migrate-{{ .Release.Revision }}
  annotations:
    "helm.sh/hook": post-install,post-upgrade
    "helm.sh/hook-weight": "0"
    "helm.sh/hook-delete-policy": before-hook-creation
spec:
  template:
    spec:
      restartPolicy: OnFailure
      containers:
        - name: migrate
          image: "{{ .Values.image.repository }}:{{ .Values.image.tag }}"
          command: ["uv", "run", "alembic", "upgrade", "head"]

The migrate Job fires first (hook-weight: "0"), then the seed Job (hook-weight: "10"). The seed Job is smart — it checks if the documents table already has data and skips gracefully. No more "did I already run the seeder?" anxiety.

And the LLM model? It downloads a ~1GB GGUF file on first startup and caches it in a PVC at /models. First deploy takes a few minutes while it downloads. Every deploy after that? Instant. The model survives pod restarts, rebuilds, everything.

War stories from the trenches

No project worth building comes without battle scars. Here are the ones that left marks.

The ghost in the database

After deploying, everything worked perfectly. Come back the next morning, and the app is dead. PostgreSQL connections had silently died during idle periods.

The fix: pool_pre_ping=True and pool_recycle=300 on the SQLAlchemy engine. It pings connections before using them and recycles them every 5 minutes. Simple fix, but diagnosing it involved a lot of confused staring at error logs.

The streaming trap

I already mentioned this, but it deserves its own section. I spent two hours debugging why my streams were "working" but the browser kept re-rendering the entire message every chunk. The culprit: stream_text() returns cumulative text, not incremental deltas. The chunk[len(prev):] trick is now burned into my brain forever.

Dependency hell, one version at a time

Pydantic AI 0.8.1 depends on opentelemetry._events — a module that OpenTelemetry removed in version 1.44. The fix? Pin opentelemetry-api>=1.42,<1.43 and move on with your life. Version pinning: not glamorous, but it keeps the lights on.

Skaffold, Helm hooks, and the great image tag mystery

Skaffold is amazing at building images and deploying Helm charts. But Helm hook Jobs? They reference images through .Values.image.tag, and Skaffold doesn't inject its build tags into hook templates.

The workaround: always tag your image as latest and match it in values.yaml. Not elegant, but it works. I'll fix it properly someday. (I won't.)

The first deploy is always the slowest

That first skaffold run takes longer because the container is downloading a ~1GB GGUF model. But that's the beauty of Persistent Volumes — the model sticks around. Every subsequent deploy is fast, because the model is already sitting in /models, warm and waiting.

So... does it work?

Yes. Beautifully.

Type a question. Watch the embedding get computed locally. See PostgreSQL find the relevant documents in milliseconds. Read the LLM's response as it streams in, character by character. Check the source citations to see which documents informed the answer.

And here's the thing that still gets me: all of it is happening on my laptop. No packets leaving my network. No tokens being counted by a billing system somewhere. No vendor lock-in. Just my code, my models, my data.

The pipeline handles everything end-to-end:

Natural language questions → local embedding
Semantic search → PostgreSQL with pgvector and HNSW
Retrieved context + question → local LLM via llama-cpp-python
Streaming response → back to the browser via SSE

One skaffold run deploys the entire stack. One kubectl port-forward lets you chat with it.

What I learned (and what's next)

This project changed how I think about AI infrastructure. We've gotten so used to reaching for managed services that we forget how much is possible on a single machine. A 1.5B parameter model isn't going to write your novel, but for RAG? For answering questions grounded in your own documents? It's more than enough.

The whole stack — PostgreSQL, pgvector, FastAPI, sentence-transformers, llama-cpp-python, Kubernetes — fits comfortably in 16GB of RAM. No GPU. No cloud. No API keys taped to the bottom of your keyboard.

Could it be better? Always. A better model, hybrid search (semantic + keyword), persistent conversation storage, zero-downtime deployments. That's for another weekend.

For now, I'm just enjoying the quiet satisfaction of asking my local LLM a question and watching it answer — powered by nothing but my CPU and a few hundred lines of code.

Repository: github.com/nf1s/rag-supabase

If you build something similar, tag me on Dev.to. I'd love to see what you cook up without a credit card. And if you run into the same streaming bug — chunk[len(prev):]. You're welcome.

DEV Community: Ahmed Nafies