DEV Community: Ahmed Nafies

Testing GLM-5.3 on a Budget with OpenCode Go: 10 Tasks, 4¢, Zero Failures

Ahmed Nafies — Fri, 14 Aug 2026 17:51:42 +0000

On August 14, 2026, Zhipu dropped GLM-5.3 — a post-training-only upgrade over GLM-5.2 with 128K max output tokens. I put it through its paces with OpenCode Go the same day — spoiler: it's awesome.

It's one of the pricier models on the OpenCode Go plan — a reasoning model at $1.40/M input and $4.40/M output, with a $15/month usage allowance, so every token counts. I wanted to answer a simple question: is it worth the premium over cheaper Go models like DeepSeek V4 Pro ($0.435/$0.87)? — without burning my monthly usage finding out.

So I built a tiny benchmark that cost under 4 cents total. Here's what I did and what I found.

The budget math

Before writing a single prompt, I worked out what a request actually costs:

	Price (per 1M tokens)
Input	$1.40
Output	$4.40
Cached read	$0.26

A self-contained prompt sent through opencode run (no tools, no file ops) is roughly 50 input + 100–180 output tokens. Even at the top end that's about $0.007 per request. Ten prompts ≈ 4 cents.

The trick to staying cheap: short, self-contained prompts that don't trigger agentic tool loops, and a shared system prompt so OpenCode's context stays in the cache ($0.26/M instead of $1.40/M).

Setup

One line. GLM-5.3 is already in the Go catalog under the model id opencode-go/glm-5.3, so after /connect → OpenCode Go it's just:

opencode run --model opencode-go/glm-5.3 "your prompt"

The test

A Bash harness loops over 10 prompts and runs each one through opencode run, timing it and saving the output:

#!/usr/bin/env bash
set -uo pipefail
MODEL="${MODEL:-opencode-go/glm-5.3}"
SUFFIX=$'\n\nAnswer directly. Do not run any code or use any tools. Just provide the answer.'
echo "task,wall_seconds,status" > results/summary.csv
idx=1
while IFS= read -r -d '' prompt; do
  start=$(date +%s.%N)
  opencode run --model "$MODEL" "$prompt$SUFFIX" > "results/$(printf '%02d' "$idx").md" 2>&1
  end=$(date +%s.%N)
  echo "$idx,$(echo "$end - $start" | bc -l),ok" >> results/summary.csv
  idx=$((idx+1))
done < <(python3 -c "import sys;print('\0'.join([p.strip() for p in open('prompts.txt').read().split('=====') if p.strip()]))")

The 10 prompts span the categories a coding agent actually needs: code generation, bug fixing, code review, SQL, regex, logic, math, structured output, explanation, and format-following.

Results

#	Category	Task	Result	Wall time
1	Codegen	Flatten a nested list (iterative)	✅ Correct stack-based solution	10.7s
2	Bug fix	Binary search off-by-one	✅ Caught the `return lo` bug	53.1s
3	Code review	`max()` in Go	✅ Found 3 real issues	15.4s
4	SQL	Top-3 paid per department	✅ `DENSE_RANK()` + tie handling	45.2s
5	Regex	Valid IPv4	✅ Correct	21.8s
6	Logic	8-ball / 2-weighing puzzle	✅ Correct 3-3-2 strategy	28.4s
7	Math	Derivative of `x³·ln x`	✅ `x²(3·ln x + 1)`	24.8s
8	JSON	Structured output	✅ Valid JSON, no fences	28.7s
9	Explain	JS dedup snippet	✅ Correct + 2 improvements	35.1s
10	Format	Exact bullet-list format	✅ Followed exactly	8.2s

10/10 correct. Total wall time ~4.5 minutes. Total cost ~$0.04.

Findings

1. It's a reasoning model — and it thinks a lot

This was the surprise. GLM-5.3 emits thinking tokens before answering, and the amount of thinking scales with task difficulty:

Task	Reasoning tokens	Output tokens
"List three benefits…" (easy)	37	35
SQL top-3 (medium)	95	107
Flatten nested list (hard)	749	179

On the hard task it thought ~4× more than it wrote. That reasoning is billed as output ($4.40/M), so it's the single biggest cost driver — and the biggest latency driver too.

2. Latency is the real price, not dollars

Wall times ranged from 8s to 53s. The binary-search fix (53s) and SQL query (45s) were the slowest, both reasoning-heavy. For a $4.40/M output model, the money is trivial — the waiting is what you'll notice in day-to-day use.

3. There's a hidden reasoning dial

When I hit GLM-5.3's raw OpenAI-compatible endpoint directly with default settings, it went off the rails: it burned 2,047 reasoning tokens on "flatten a list" and produced an empty answer because it hit the token cap while still thinking, with a 49-second time-to-first-token.

Passing "reasoning_effort": "low" changed everything:

low  => finish=stop  reasoning=2   total=6   content="444"

OpenCode Go's default tuning keeps the model usable out of the box, but if you're calling the API yourself, you'll want to set a reasoning effort explicitly.

4. The quality bar is genuinely high

The code review was the highlight — it caught the all-negative-input bug, the empty-slice edge case, and that the function shadows Go 1.21's built-in max, then pointed at slices.Max(). That's the kind of detail I'd expect from a flagship model.

Minor nitpicks: the Go fix used fmt.Errorf without importing fmt, and the JSON task invented a name ("Ahmed Hassan") when none was given. Both trivial.

The cost breakdown

Request type	Cost
Easy prompt (list three)	$0.0030
Medium prompt (SQL)	$0.0035
Hard prompt (flatten)	$0.0067
10-prompt suite total	~$0.04

Against a $15 monthly Go allowance, that's 0.27% of my monthly usage for a complete capability picture.

Verdict

GLM-5.3 passed every task with detail I'd call frontier-grade, at a cost that's basically free on the Go plan. The trade-off is latency: it's a deliberate thinker, so it's best for hard, non-trivial tasks rather than rapid-fire edits.

If you want a fast, cheap daily driver, DeepSeek V4 Pro on Go is the better value. If you want a model that reasons through the hard stuff and rarely gets it wrong, GLM-5.3 earns its premium — and you can test it yourself for the price of a gumball.

All prompts, the harness, and raw results are in the repo. Total spend for this entire experiment (including the failed raw-API probes): under **8 cents.

Why GPT-5.6 Luna High Is My Default for Agentic Engineering

Ahmed Nafies — Fri, 14 Aug 2026 17:38:34 +0000

I used to pick coding models the same way people pick sports cars: choose the most powerful one and pretend the fuel bill is somebody else's problem.

That worked when I was asking one question at a time. Then I started using agents for real engineering work.

An engineering agent does not answer once and disappear. It reads the repository, searches for related code, opens the wrong file, finds the right file, proposes a change, runs a test, breaks something, reads the error, fixes the change, and runs the test again. Sometimes it also writes a surprisingly thoughtful essay about the three lines it just modified.

By the time one task is finished, the model may have been called a dozen times. Suddenly, model pricing is not a footnote. It is part of the architecture.

That is how GPT-5.6 Luna with high reasoning effort became my default.

The cheap model that gets the job done

OpenAI positions Luna for cost-sensitive, high-volume work, and the pricing makes that very clear. As of August 2026, GPT-5.6 Luna costs $0.20 per million input tokens and $1.20 per million output tokens. Claude Sonnet 5 costs $2 for input and $10 for output.

So Luna's input is ten times cheaper, and its output is about 8.3 times cheaper.

Take an agent run that consumes 100,000 input tokens and produces 20,000 output tokens. Luna costs about $0.044. Sonnet costs about $0.40. Neither amount is terrifying on its own, but agents are enthusiastic employees. Give them access to a repository and they will inspect everything, call every tool, and occasionally create a strategic roadmap for changing a variable name.

Repeat that across pull requests, bug fixes, failed CI jobs, and background tasks, and the difference stops being theoretical.

Using Luna means I can let the agent iterate without feeling that every failed test needs approval from Finance.

Why I use high reasoning effort

Cheap is useful only if the model can finish the work. I do not run Luna with minimal reasoning and hope for a miracle. I use high reasoning effort.

That setting gives it enough room to trace behavior across files, understand unfamiliar abstractions, choose tools, react to command failures, and verify the result before declaring victory. In my experience, that is enough for a large part of normal engineering work.

It is not the smartest model available, and I do not need it to be. My default model does not have to solve every impossible architecture problem. It has to handle routine features, localized bugs, tests, refactoring, and repository exploration without turning each task into an expensive research project.

There is a temptation to treat "accurate enough" as an insult. I see it differently. Most engineering tasks do not need a model to contemplate the nature of software. They need it to follow the repository instructions, make a focused change, preserve unrelated code, understand the compiler output, and actually run the tests.

Luna does that well enough for me, especially when the agent harness is doing its job. Good context, sensible permissions, small changes, and real test feedback matter just as much as the model. A cheaper model inside a disciplined workflow can be more useful than an expensive model confidently wandering through the codebase.

Speed matters too. A small delay is easy to ignore in one response. Across fifteen sequential agent turns, it becomes enough time to question your career choices. Luna keeps the loop moving. It can inspect, attempt, fail, correct, and verify without making every step feel like a separate meeting.

And what about Claude Sonnet?

Claude Sonnet is a capable model. There are difficult tasks where I am happy to use it. It is simply not my default.

The obvious reason is price. The less obvious reason is output efficiency. In my experience, Sonnet often gives me more explanation than the engineering loop needs. I ask for a patch and sometimes receive the patch, a retrospective, and the opening chapter of a management book.

That can be useful when I am exploring a decision. It is less useful when the agent is supposed to edit a file and run the tests. Verbose output is not only more to read; output tokens are also the expensive tokens.

Anthropic's documentation says Claude 4.7 and later can produce roughly 30% more tokens for the same text than models using its previous tokenizer, although the exact difference depends on the workload. That is not the same as saying the model is always verbose, but it is another reminder that token behavior belongs in the engineering decision.

For autonomous execution, I care more about correct edits and passing tests than an eloquent account of the model's emotional journey.

My approach is simple: start with Luna High and escalate when the task earns it. If the architecture is genuinely ambiguous, failures remain unexplained, or the work involves subtle security or concurrency behavior, I will reach for a stronger model. That is not model loyalty. It is routing.

The best model is not always the one that wins the benchmark. For agentic engineering, I care about the cost of the completed task, how long it took, how many retries it needed, whether the tests passed, and how much cleanup I had to do afterward.

For my workload, GPT-5.6 Luna with high reasoning effort currently hits the useful middle: very cheap, very fast, and accurate enough to handle the common path.

It is not the model I want solving every problem. It is the model I want holding the wrench most of the time.

Pricing sources: OpenAI model documentation and Anthropic pricing documentation, checked August 14, 2026.

I Asked Muse Glimmer to Pair Program With Me in OpenCode

Ahmed Nafies — Tue, 11 Aug 2026 22:04:13 +0000

I wanted to use Meta's Muse Glimmer as a local agentic engineering assistant inside OpenCode.

Not as a chatbot in a browser. I wanted an agent that could inspect a repository, understand a task, edit files, run tests, and explain what it did.

The command was refreshingly short:

ollama launch opencode --model muse-glimmer

The model, however, is a 30-billion-parameter coworker who arrives carrying an approximately 18 GB suitcase and expects a 128K-context office.

The onboarding meeting

First, I checked Ollama:

ollama --version

Then I started the local service if needed:

ollama serve

I verified that the local API was alive:

curl http://localhost:11434/api/version

Then I launched OpenCode with Muse Glimmer:

ollama launch opencode --model muse-glimmer

The first run downloads the model. This is a good opportunity to make coffee, reconsider your disk space, and remember that “local AI” does not mean “tiny AI.”

To inspect installed models:

ollama ls

On Apple Silicon, Ollama also lists an MLX variant:

ollama launch opencode --model muse-glimmer:30b-mlx

The standard model is listed at around 18 GB. That is the model's luggage, not the entire hotel bill. Leave memory for the context, OpenCode, the operating system, and the repository you are asking the agent to understand.

Why OpenCode changes the experiment

The interesting part is not just running Muse Glimmer locally. It is putting the model inside an engineering workflow.

With OpenCode, I can ask the agent to:

Inspect the repository.
Find the relevant files.
Explain the current behavior.
Propose a plan.
Make a focused change.
Run the project's tests.
Read failures.
Revise the patch.
Show the final diff and verification result.

That is the difference between asking a model for code and asking an agent to help complete an engineering task. The model can still be confidently wrong. The workflow gives me more chances to catch it before the wrongness becomes a pull request.

The first repository tour

I opened the repository in OpenCode and started with a read-only request:

You are working in a local software repository. Inspect the project structure and identify:

- The application entry point
- The test command
- The main source directories
- Configuration files that affect development
- Any obvious contribution or safety constraints

Do not modify files. Report the evidence and ask before making changes.

This is a better first task than “fix everything.” It tests whether Muse Glimmer can build context without immediately turning the repository into modern art.

The key signal is whether the response points to actual files and commands. A useful agent should report evidence, not invent a project architecture because the folder name sounded familiar.

The first engineering task

After the repository tour, I gave OpenCode a small, testable issue:

Find the bug in the relevant average function. Explain the edge case, propose the smallest fix, and add a regression test. Do not change unrelated files.

The current implementation is: average(values) = sum(values) / len(values) + 1

The extra + 1 is suspicious. It is the kind of bug that enters production wearing sunglasses and saying, “Nobody will notice.”

A good OpenCode session should locate the real implementation, find the project's existing test style, make the smallest patch, and run the relevant test. A locally generated patch is still a patch. It needs review, tests, and version control like any other change.

Planning before editing

I asked OpenCode to separate investigation from implementation:

Before editing, write a short implementation plan with:

- Files you expect to inspect
- The behavior you expect to change
- The test you will add or update
- The command you will run to verify the change

Wait for approval before modifying files.

This creates a useful checkpoint. It also prevents the classic coding-agent experience where you ask for one small fix and receive a surprise framework migration, a new abstraction layer, and a README poem.

OpenCode is the interface where this loop becomes practical: inspect the plan, approve the direction, watch the diff, and ask for verification in the same engineering session.

The test-and-repair loop

The useful workflow looks like this:

Inspect -> plan -> approve -> edit -> test -> inspect failure -> revise -> test again

I tested failure recovery with an intentionally broken test result:

The test runner returned:

ERROR: assertion failed in tests/test_average.py
Expected: 2.0
Received: 3.0

Explain the failure, identify the likely cause, and propose the smallest correction. Do not edit files yet.

Then I tested a repository constraint:

The repository check returned:

ERROR: the generated lockfile differs from the committed lockfile.

Do not regenerate dependencies. Explain what should be reviewed first.

The goal is not to make the agent retry everything until the terminal turns green. The goal is to see whether it can distinguish a fixable implementation error from a situation requiring human review.

“The model usually behaves” is not an engineering control. It is something you say immediately before opening an incident ticket.

Screenshot-driven debugging

Muse Glimmer supports image input, which is useful for UI bugs, screenshots, diagrams, and terminal output. From the local OpenCode workflow, I can give the agent a screenshot and ask for evidence-first analysis:

Inspect this screenshot as a debugging artifact. List only visible symptoms, separate observations from hypotheses, and suggest which source files or browser checks should be inspected next.

“The button is clipped” is an observation. “The CSS grid is broken because of a missing min-width” is a hypothesis. That distinction saves time and prevents a screenshot from becoming a license to rewrite the frontend.

Conclusion

Using Muse Glimmer with OpenCode through Ollama is straightforward to start:

ollama launch opencode --model muse-glimmer

The interesting work begins after OpenCode starts. Ask the agent to inspect, plan, edit, test, diagnose, and explain.

Muse Glimmer can be the clever local coworker. Ollama handles the model runtime. OpenCode handles the engineering conversation. You remain the person who decides whether the patch deserves to live.

Resources:

I Let an Autonomous Agent Build My RAG Stack. Here's How the Loop Worked

Ahmed Nafies — Thu, 06 Aug 2026 17:24:49 +0000

I Let an Autonomous Agent Build My RAG Stack. Here's How the Loop Worked

I didn't write a single line of code. I didn't debug a Helm chart. I didn't stare at a traceback wondering why the connector wouldn't authenticate.

I typed one sentence and watched.

Twenty minutes later, a draft pull request landed in my private repo. Everything behind that PR — the OpenSearch cluster, the embedding models, the five init jobs in lockstep, the MCP server, the RAG pipeline, and the 30 seeded articles — was produced by an autonomous coding agent using a workflow called the loop.

The stack itself is impressive enough: OpenSearch 3.7 with hybrid search fusing semantic and full-text results, a retrieval-augmented generation pipeline powered by a local Ollama LLM, a FastMCP server exposing four tools over SSE, and a Skaffold + Helm deployment that goes from zero to running cluster with skaffold run. But the stack isn't the story.

The story is what happened between the prompt and the PR. The planning, the self-correction, the moment the agent deleted 14 model chunks it thought were duplicates, broke the entire embedding system, and then — without any help — fixed it.

This is how the loop worked.

The loop isn't just an agent. It's a contract.

The loop is a skill for opencode that turns a natural-language prompt into a working feature branch and a draft PR. It sounds like every other AI coding tool until you see what it won't do: it won't write code before you approve a plan.

Phase	What happens
proposal	Explores the codebase, analyzes the prompt, proposes a plan. No implementation.
agreement	You review. You can push back, ask questions, or say go. The agent waits.
implementation	Builds the agreed scope on a feature branch, committing early and often.
draft_gate	Self-review: lint, tests, diff inspection. Fixes issues before you ever see them.
draft_pr	Pushes the branch, opens a draft PR, reports back. Done.

Every agent should work like this. Most don't. Most just start writing code the moment you press enter, and you spend the rest of the session saying "no, not like that."

The loop carries its state in .loop-state.json — the current phase, the branch, the PR number, the plan, and a log of every phase transition. If your session crashes or you walk away for an hour, the loop resumes exactly where it left off. This alone makes it viable for multi-hour autonomous runs.

Then there's loop-police: a companion skill that rides shotgun, watching for infinite loops. Thinking loops. Tool-call loops. File-read spirals. Stagnation. When loop-police detects the agent going in circles, it interrupts: "You're stuck. Pivot." That intervention is the difference between an agent that wastes ten turns retrying the same failing deploy and one that recognizes the failure and changes strategy.

Without loop-police, the chunk-deletion disaster I'll get to later might have been unrecoverable.

The plan was one paragraph

The agent's proposal:

Build opensearch-hybrid-mcp mirroring the os-hybrid-search reference: OpenSearch 3.7 with Dashboards, MCP server (FastMCP + UV), five init jobs (template → embedding → connector → pipeline → seed), skaffold.yaml with six images, justfile, README. Private repo + draft PR.

And then a diagram — the agent drew the architecture:

macOS host                             Kubernetes (Colima)
┌───────────────┐                      ┌────────────────────────────────────────────────────────────┐
│ Ollama        │                      │  ┌──────────────────┐  ┌────────────────┐                  │
│ (Desktop)     │◄──────── :11434 ────►│  │  OpenSearch 3.7  │  │  OS Dashboards │                  │
│ qwen3.5:9b-mlx│  host.docker.internal│  │  os-hybrid:9200  │  │  :5601         │                  │
└───────────────┘                      │  └────────┬─────────┘  └────────────────┘                  │
                                       │          │                                                 │
                                       │  ┌────────▼──────────┐                                     │
                                       │  │  MCP Server       │    FastMCP + UV                     │
                                       │  │  mcp-server:8000  │    hybrid_search                    │
                                       │  │                   │    rag_query                        │
                                       │  │                   │    index_document                   │
                                       │  │                   │    cluster_health                   │
                                       │  └───────────────────┘                                     │
                                       │                                                            │
                                       │  Init Jobs (run in sequence):                              │
                                       │    1. template-job    index template                       │
                                       │    2. embedding-job   all-MiniLM-L12-v2                    │
                                       │    3. connector-job   Ollama → Qwen 3.5                    │
                                       │    4. pipeline-job    chunking/hybrid/RAG                  │
                                       │    5. seed-job        30 articles                          │
                                       └────────────────────────────────────────────────────────────┘

Ollama runs on the host, not inside the cluster — pods reach it through host.docker.internal:11434. The agent figured that out by itself. Single-node OpenSearch, 10Gi PVC, no security (it's a dev cluster), MCP server on SSE port 8000. Five init jobs that run in sequence, each waiting for its upstream dependencies through retry loops.

I said yes. The agent started building.

The implementation was fast. The recovery was where it got interesting.

Seven commits. Forty-plus files. The agent scaffolded the entire project in minutes.

Six Helm charts (OpenSearch parent, MCP server, five init jobs). A skaffold.yaml with six Docker images building in parallel. A justfile with shortcuts for skaffold run, kubectl port-forward, and job logs. Every init job is a shell script with until loops — each waits for OpenSearch, for the index template, for the embedding model to deploy, for the pipeline to be ready. If a dependency isn't there, the job pauses. If it is, the job runs. This makes the whole stack re-runnable: delete a job and re-create it, and it picks up where it left off.

There's a satisfying recursion here. The loop is a five-phase workflow of gates and retries. The jobs it wrote are retry loops with dependency checks. The agent built code that mirrors its own architecture.

The MCP server landed next: FastMCP, four tools:

hybrid_search(query, index?, k?) — neural + full-text, z-score fused
rag_query(question, index?, k?) — retrieves context, generates answer
index_document(id, title, body, index?) — chunks + embeds + indexes
cluster_health() — cluster status

The agent ran its own draft gate: helm lint and helm template on seven charts, skaffold render, sh -n on every script. Clean. It pushed the branch, opened draft PR #1, and reported back.

Technically, the loop was done: proposal → agreement → implementation → draft_gate → draft_pr. Five phases, zero handoffs.

But the stack hadn't been deployed yet. And that's where everything went sideways — then sideways again — then somehow straightened out.

Five bugs. One autonomous agent. Zero panicking.

The first skaffold run kicked off and immediately hit a wall. The agent didn't wait for me to notice. It read the logs, diagnosed the problem, patched it, and redeployed. Five times.

Bug 1 — "Connector credential is null or empty list": OpenSearch 3.7 demands a credential object even for local Ollama endpoints that don't use authentication. The agent added a dummy key (openAI_key: "ollama-local") and — crucially — switched the connector from Ollama's native /api/chat to the OpenAI-compatible /v1/chat/completions, which is what the RAG processor expects. Two fixes in one pass. Redeployed. Connector created.

Bug 2 — The one-character regex disaster: RAG queries hit a 400: "Connector URL is not matching the trusted connector private endpoint regex". The agent had written ^http://host.docker.internal:11434:.*$ — a colon after the port. The actual URL was http://host.docker.internal:11434/v1/chat/completions — a slash. The difference between : and / broke the entire pipeline. The agent found it, flipped the character, and redeployed. The kind of bug a human would spend 20 minutes on. The agent caught it in one.

Bug 3 — "Model 'ollama-qwen3.5' not found": The connector was passing the OpenSearch-registered model name to Ollama, but Ollama expects the raw model name (qwen3.5:9b-mlx). The agent traced the call chain — OpenSearch RAG processor → connector → Ollama API — and updated the MCP server to pass the correct model name in ext.generative_qa_parameters. Fixed.

Bug 4 — Memory circuit breaker: Seed-job bulk indexing hit 429 rate limits. JVM heap at 93% on 2g allocation. The agent bumped it to 4g (-Xmx4g -Xms4g, container limits 4g/8Gi), redeployed, and the jobs succeeded. The agent understood enough about JVM memory pressure to recognize the root cause without being told.

Bug 5 — The chunk-deletion disaster: This is the one where the agent almost nuked the project. While cleaning up orphaned model registrations, it deleted 14 documents from the ML system index — thinking they were duplicates. They weren't. They were the embedding model's chunks (_0 through _13). The model went to DEPLOY_FAILED. The embedding pipeline was dead.

Most agents would either not notice, or notice and keep retrying the same broken state until you killed the session. The loop did something else: it recognized the mistake. It saw DEPLOY_FAILED on the model, understood that the chunks it deleted were critical, deleted the broken base model document, let the embedding-job re-register fresh with a new model ID, and re-ran the pipeline and seed jobs. The stack healed itself.

That pattern — deploy, hit error, read logs, diagnose, patch, redeploy, verify — repeated five times. The agent operated the stack as well as it built it.

The moment it worked

After the last fix, the cluster settled. Thirty articles seeded. Both models deployed. Cluster health green.

A hybrid search returned ranked results with fused neural and full-text scores. A RAG query against the pipeline — retrieves context first, then calls Ollama through the connector, returns the answer grounded in the retrieved documents:

{
  "query": { "match": { "body": "inverted index TF-IDF BM25" } },
  "ext": {
    "generative_qa_parameters": {
      "llm_model": "qwen3.5:9b-mlx",
      "llm_question": "Based on the context, explain what an inverted index is and how TF-IDF or BM25 ranking works."
    }
  }
}

The system answered:

An inverted index is a structure that maps each unique word to the documents containing it, allowing for fast retrieval based on query terms. Regarding ranking, TF-IDF measures term frequency versus inverse document frequency, while BM25 improves upon this method by saturating term frequency.

Grounded. Accurate. Sourced from the 30 articles the agent itself had seeded. All four MCP tools exercised and verified over SSE. The agent updated the draft PR with the validation results.

What I'd do differently (and what I won't)

Loop-police is not optional. Without it, the agent would have retried that failing deploy until the heat death of the universe. With it, the agent flagged the DEPLOY_FAILED state transition, recognized it had caused the failure, and pivoted to recovery. If you're running autonomous agents, you need a stall detector. Period.

The agreement gate is the whole point. Autonomous coding agents without a plan-agreement phase are just hyperactive interns with commit access. The loop doesn't move from proposal to implementation until you say yes. That one rule eliminates the most common failure mode of AI coding: the agent building the wrong thing while you watch helplessly.

State files make long-running sessions possible. .loop-state.json tracks phase, branch, PR number, and plan history. If the session drops — and with tools running 40+ minute autonomous sequences, it will — the loop resumes without context loss. This is infrastructure, not a nice-to-have.

Retry-loop init jobs are self-healing infrastructure. Each job waits for its dependencies through until loops. You can tear down a job and recreate it — it'll wait, detect its prerequisites are met, and proceed. The loop wrote infrastructure that works the way the loop itself works. That's either poetic or recursive. Probably both.

The draft gate is the QA step you'll never do yourself. The loop ran helm lint, helm template, skaffold render, and sh -n before opening a PR. No human on my team does that for every commit. The agent does it because the workflow demands it. Fail the gate, and the loop doesn't proceed to PR. This catches broken manifests before they reach the repo — before you even see them.

The loop isn't magic. It's process.

That's the real takeaway. Autonomous coding agents don't fail because they're not smart enough. They fail because they lack structure — no plan phase, no review gate, no self-check, no stall detection.

The loop adds that structure. It turns an agent from a code generator into a collaborator: you get a plan, you approve it, the agent builds, it self-corrects, and it ships a draft PR. You review, you merge, you move on.

I didn't write a single line of code. But the loop didn't build my RAG stack by accident. It built it because the process forced it to plan, verify, and recover — the same way a good engineer would.

That's how the loop worked.

Building a 100% Local RAG System on Kubernetes — No API Keys Required

Ahmed Nafies — Thu, 30 Jul 2026 23:00:55 +0000

Here's what we're building:

The itch I couldn't scratch

Last week I sat down to build a RAG system. You know, one of those "chat with your documents" things everyone's building these days.

I opened my first tutorial. It said: "Step 1: Sign up for OpenAI and grab your API key." Fine. "Step 2: Create a Pinecone account." Okay. "Step 3: Deploy to a managed vector database." Hmm. "Step 4: Set up your billing — here's a pricing calculator."

I stared at the calculator. Then at my $0 budget. Then back at the calculator.

Something felt off. Why does a "personal project" need a procurement department? Why does my data need to leave my laptop to answer a question about a document sitting on my laptop? And honestly — I just wanted to tinker. Not negotiate with three SaaS vendors before I could write a single line of code.

So I did what any reasonable engineer would do. I closed all twenty tabs, opened a terminal, and decided to build the whole thing from scratch. On my machine. With models running on my CPU. No cloud. No API keys. No credit card.

Here's how that ride went.

The game plan

Before writing a line of code, I sketched out what "no cloud" actually means:

Embeddings — gotta run locally. No calling OpenAI's embedding endpoint.
Vector search — needs to live inside something I already run. I'm not spinning up a separate vector database service.
The LLM — must fit in my laptop's RAM and run on CPU. I don't have a GPU lying around.
Deployment — real infrastructure patterns, not docker-compose up and pray. Something that would make sense in production, just scaled down.

What emerged was this stack:

┌────────────────────────────────────────────────────┐
│                 FastAPI Application                 │
├────────────────────────────────────────────────────┤
│  ┌──────────────┐  ┌──────────┐  ┌──────────────┐  │
│  │  Embedding   │  │ pgvector │  │  llama-cpp   │  │
│  │  Pipeline    │  │  Search  │  │  (Qwen 1.5B) │  │
│  └──────────────┘  └──────────┘  └──────────────┘  │
└────────────────────────────────────────────────────┘
                         │
                         ▼
               ┌──────────────────┐
               │   PostgreSQL     │
               │   + pgvector     │
               │   + HNSW index   │
               └──────────────────┘

The flow: someone asks a question → it gets turned into a 384-dimensional vector → PostgreSQL hunts for similar documents using cosine similarity → those results get stuffed into a prompt → the LLM answers → answer streams back. All inside Kubernetes. All local.

Every piece of this puzzle was chosen with one rule: it runs on my machine, or it doesn't make the cut.

Choosing the players

pgvector: the database you already have

I wanted vector search without installing Yet Another Database. pgvector is a PostgreSQL extension that adds vector columns, indexing, and similarity operators directly into Postgres. Just CREATE EXTENSION IF NOT EXISTS vector and you're in business.

It gives you HNSW indexes (fast approximate nearest neighbor search), a <=> cosine distance operator, and zero additional infrastructure. Your vector DB is just... your regular DB.

Qwen 2.5, living in my CPU

Here's where things got spicy. Most RAG tutorials assume you'll call GPT-4. But I had no GPU, no cloud credits, and a strong desire to keep everything local.

Enter Qwen2.5-1.5B-Instruct — a 1.5 billion parameter model that actually runs on CPU. Not fast, but fast enough. And the output quality? Genuinely surprising for its size.

The trick is serving it via llama-cpp-python, which exposes an OpenAI-compatible API at http://127.0.0.1:8001/v1. The magic of this: my application code uses the exact same OpenAIChatModel class you'd point at api.openai.com. It doesn't know — and doesn't care — that the model is running on the same machine, on a CPU, inside a container.

All it sees is a /v1/chat/completions endpoint. The fact that it's a 1.5B model crammed into a Docker container? That's between me and the container.

sentence-transformers: tiny but mighty

For embeddings, I picked all-MiniLM-L6-v2. It produces 384-dimensional vectors, runs fast on CPU, and the quality is solid for semantic search. It gets loaded once at startup and cached — no reloading per request:

from sentence_transformers import SentenceTransformer

@lru_cache
def get_model() -> SentenceTransformer:
    return SentenceTransformer("all-MiniLM-L6-v2")

def embed_text(text: str) -> list[float]:
    return get_model().encode(text).tolist()

Kubernetes on a laptop? Absolutely.

Running K8s locally sounds excessive until you try it. I used Colima — a lightweight k3s cluster that runs on 2 CPUs and 16GB of RAM. No Docker Desktop, no resource vampires. Just a clean K8s cluster on my Mac.

The benefit is real: Helm charts for declarative infra, Skaffold for the build-deploy loop, Helm hook Jobs that run migrations and seeding automatically. The same patterns I'd use in production, just at my desk.

Under the hood

Let's walk through the interesting bits. Not every line — just the parts that made me smile when they worked.

The database that thinks in vectors

The documents table stores each document alongside its 384-dimensional embedding vector. The HNSW index means similarity searches stay fast even as the corpus grows:

CREATE EXTENSION IF NOT EXISTS vector;

CREATE TABLE documents (
    id          uuid PRIMARY KEY DEFAULT gen_random_uuid(),
    title       varchar(500) NOT NULL,
    content     text NOT NULL,
    embedding   vector(384) NOT NULL,
    created_at  timestamptz DEFAULT now()
);

CREATE INDEX ON documents
    USING hnsw (embedding vector_cosine_ops)
    WITH (m = 16, ef_construction = 64);

The search that feels like magic

A stored procedure does the heavy lifting. You throw a vector at it, and it returns the closest matches ranked by similarity:

CREATE OR REPLACE FUNCTION match_documents (
    query_embedding vector(384),
    match_threshold float,
    match_count int
)
RETURNS TABLE (id uuid, title text, content text, similarity float)
LANGUAGE sql STABLE
AS $$
    SELECT
        documents.id,
        documents.title,
        documents.content,
        1 - (documents.embedding <=> query_embedding) AS similarity
    FROM documents
    WHERE documents.embedding <=> query_embedding < 1 - match_threshold
    ORDER BY documents.embedding <=> query_embedding
    LIMIT match_count;
$$;

That <=> is pgvector's cosine distance operator. 1 - distance gives us similarity. Clean, fast, entirely in SQL.

The application calls it through a parameterized SQLAlchemy query — no ORM magic, just raw SQL with safe parameters:

async def match_documents(
    session: AsyncSession,
    query_embedding: list[float],
    match_threshold: float = 0.5,
    match_count: int = 5,
) -> list[dict]:
    query = text("""
        SELECT * FROM match_documents(
            CAST(:query_embedding AS vector(384)),
            :match_threshold,
            :match_count
        )
    """)
    result = await session.execute(query, {
        "query_embedding": embedding_str,
        "match_threshold": match_threshold,
        "match_count": match_count,
    })
    return [{"id": str(r["id"]), "title": r["title"],
             "content": r["content"], "similarity": r["similarity"]}
            for r in result.mappings().all()]

The agent that thinks it's talking to OpenAI

This is my favorite part. Pydantic AI's Agent class is designed for OpenAI, but llama-cpp-python speaks the same protocol. So we just... point it at localhost:

from pydantic_ai import Agent
from pydantic_ai.models.openai import OpenAIChatModel
from pydantic_ai.providers.openai import OpenAIProvider

model = OpenAIChatModel(
    "qwen2.5-1.5b-instruct",
    provider=OpenAIProvider(
        base_url="http://127.0.0.1:8001/v1",
        api_key="not-needed",
    ),
)

agent = Agent(model=model, system_prompt=(
    "You are a helpful assistant. Answer questions based on "
    "the provided context. If the context does not contain "
    "enough information, say so."
))

The api_key="not-needed" line makes me happy every time I look at it. Take that, billing page.

Streaming: the one that nearly broke me

Server-Sent Events streaming should be straightforward. You yield chunks, the browser displays them. Simple.

Except Pydantic AI's stream_text() doesn't return deltas — it returns the accumulated text. Every chunk is the full response so far. If you send that to the browser, it keeps re-rendering the entire message.

The fix is deceptively simple. Track the previous value and compute the difference:

async def generate():
    yield f"event: sources\ndata: {json.dumps(sources)}\n\n"

    prev = ""
    async with agent.run_stream(prompt) as result:
        async for chunk in result.stream_text():
            delta = chunk[len(prev):]  # Just the new stuff
            prev = chunk
            yield f"event: delta\ndata: {json.dumps({'content': delta})}\n\n"

return StreamingResponse(generate(), media_type="text/event-stream")

Two lines of code that took two hours to figure out. But watching the text flow in, word by word, on the first successful stream — chef's kiss.

Ship it: Kubernetes all the way down

Three Helm charts, one skaffold run:

Chart	What's inside
`rag-supabase-db`	PostgreSQL StatefulSet + pgvector, 5Gi persistent disk
`rag-supabase`	FastAPI + llama-cpp running side-by-side, 3Gi model cache
`rag-supabase-ui`	Nginx serving the chat UI

The slick part: migrations and seeding aren't manual steps. They're Helm hook Jobs that fire automatically on every deploy:

apiVersion: batch/v1
kind: Job
metadata:
  name: {{ .Release.Name }}-migrate-{{ .Release.Revision }}
  annotations:
    "helm.sh/hook": post-install,post-upgrade
    "helm.sh/hook-weight": "0"
    "helm.sh/hook-delete-policy": before-hook-creation
spec:
  template:
    spec:
      restartPolicy: OnFailure
      containers:
        - name: migrate
          image: "{{ .Values.image.repository }}:{{ .Values.image.tag }}"
          command: ["uv", "run", "alembic", "upgrade", "head"]

The migrate Job fires first (hook-weight: "0"), then the seed Job (hook-weight: "10"). The seed Job is smart — it checks if the documents table already has data and skips gracefully. No more "did I already run the seeder?" anxiety.

And the LLM model? It downloads a ~1GB GGUF file on first startup and caches it in a PVC at /models. First deploy takes a few minutes while it downloads. Every deploy after that? Instant. The model survives pod restarts, rebuilds, everything.

War stories from the trenches

No project worth building comes without battle scars. Here are the ones that left marks.

The ghost in the database

After deploying, everything worked perfectly. Come back the next morning, and the app is dead. PostgreSQL connections had silently died during idle periods.

The fix: pool_pre_ping=True and pool_recycle=300 on the SQLAlchemy engine. It pings connections before using them and recycles them every 5 minutes. Simple fix, but diagnosing it involved a lot of confused staring at error logs.

The streaming trap

I already mentioned this, but it deserves its own section. I spent two hours debugging why my streams were "working" but the browser kept re-rendering the entire message every chunk. The culprit: stream_text() returns cumulative text, not incremental deltas. The chunk[len(prev):] trick is now burned into my brain forever.

Dependency hell, one version at a time

Pydantic AI 0.8.1 depends on opentelemetry._events — a module that OpenTelemetry removed in version 1.44. The fix? Pin opentelemetry-api>=1.42,<1.43 and move on with your life. Version pinning: not glamorous, but it keeps the lights on.

Skaffold, Helm hooks, and the great image tag mystery

Skaffold is amazing at building images and deploying Helm charts. But Helm hook Jobs? They reference images through .Values.image.tag, and Skaffold doesn't inject its build tags into hook templates.

The workaround: always tag your image as latest and match it in values.yaml. Not elegant, but it works. I'll fix it properly someday. (I won't.)

The first deploy is always the slowest

That first skaffold run takes longer because the container is downloading a ~1GB GGUF model. But that's the beauty of Persistent Volumes — the model sticks around. Every subsequent deploy is fast, because the model is already sitting in /models, warm and waiting.

So... does it work?

Yes. Beautifully.

Type a question. Watch the embedding get computed locally. See PostgreSQL find the relevant documents in milliseconds. Read the LLM's response as it streams in, character by character. Check the source citations to see which documents informed the answer.

And here's the thing that still gets me: all of it is happening on my laptop. No packets leaving my network. No tokens being counted by a billing system somewhere. No vendor lock-in. Just my code, my models, my data.

The pipeline handles everything end-to-end:

Natural language questions → local embedding
Semantic search → PostgreSQL with pgvector and HNSW
Retrieved context + question → local LLM via llama-cpp-python
Streaming response → back to the browser via SSE

One skaffold run deploys the entire stack. One kubectl port-forward lets you chat with it.

What I learned (and what's next)

This project changed how I think about AI infrastructure. We've gotten so used to reaching for managed services that we forget how much is possible on a single machine. A 1.5B parameter model isn't going to write your novel, but for RAG? For answering questions grounded in your own documents? It's more than enough.

The whole stack — PostgreSQL, pgvector, FastAPI, sentence-transformers, llama-cpp-python, Kubernetes — fits comfortably in 16GB of RAM. No GPU. No cloud. No API keys taped to the bottom of your keyboard.

Could it be better? Always. A better model, hybrid search (semantic + keyword), persistent conversation storage, zero-downtime deployments. That's for another weekend.

For now, I'm just enjoying the quiet satisfaction of asking my local LLM a question and watching it answer — powered by nothing but my CPU and a few hundred lines of code.

Repository: github.com/nf1s/rag-supabase

If you build something similar, tag me on Dev.to. I'd love to see what you cook up without a credit card. And if you run into the same streaming bug — chunk[len(prev):]. You're welcome.