I Let an Autonomous Agent Build My RAG Stack. Here's How the Loop Worked
I didn't write a single line of code. I didn't debug a Helm chart. I didn't stare at a traceback wondering why the connector wouldn't authenticate.
I typed one sentence and watched.
Twenty minutes later, a draft pull request landed in my private repo. Everything behind that PR — the OpenSearch cluster, the embedding models, the five init jobs in lockstep, the MCP server, the RAG pipeline, and the 30 seeded articles — was produced by an autonomous coding agent using a workflow called the loop.
The stack itself is impressive enough: OpenSearch 3.7 with hybrid search fusing semantic and full-text results, a retrieval-augmented generation pipeline powered by a local Ollama LLM, a FastMCP server exposing four tools over SSE, and a Skaffold + Helm deployment that goes from zero to running cluster with skaffold run. But the stack isn't the story.
The story is what happened between the prompt and the PR. The planning, the self-correction, the moment the agent deleted 14 model chunks it thought were duplicates, broke the entire embedding system, and then — without any help — fixed it.
This is how the loop worked.
The loop isn't just an agent. It's a contract.
The loop is a skill for opencode that turns a natural-language prompt into a working feature branch and a draft PR. It sounds like every other AI coding tool until you see what it won't do: it won't write code before you approve a plan.
| Phase | What happens |
|---|---|
| proposal | Explores the codebase, analyzes the prompt, proposes a plan. No implementation. |
| agreement | You review. You can push back, ask questions, or say go. The agent waits. |
| implementation | Builds the agreed scope on a feature branch, committing early and often. |
| draft_gate | Self-review: lint, tests, diff inspection. Fixes issues before you ever see them. |
| draft_pr | Pushes the branch, opens a draft PR, reports back. Done. |
Every agent should work like this. Most don't. Most just start writing code the moment you press enter, and you spend the rest of the session saying "no, not like that."
The loop carries its state in .loop-state.json — the current phase, the branch, the PR number, the plan, and a log of every phase transition. If your session crashes or you walk away for an hour, the loop resumes exactly where it left off. This alone makes it viable for multi-hour autonomous runs.
Then there's loop-police: a companion skill that rides shotgun, watching for infinite loops. Thinking loops. Tool-call loops. File-read spirals. Stagnation. When loop-police detects the agent going in circles, it interrupts: "You're stuck. Pivot." That intervention is the difference between an agent that wastes ten turns retrying the same failing deploy and one that recognizes the failure and changes strategy.
Without loop-police, the chunk-deletion disaster I'll get to later might have been unrecoverable.
The plan was one paragraph
The agent's proposal:
Build
opensearch-hybrid-mcpmirroring theos-hybrid-searchreference: OpenSearch 3.7 with Dashboards, MCP server (FastMCP + UV), five init jobs (template → embedding → connector → pipeline → seed),skaffold.yamlwith six images,justfile, README. Private repo + draft PR.
And then a diagram — the agent drew the architecture:
macOS host Kubernetes (Colima)
┌───────────────┐ ┌────────────────────────────────────────────────────────────┐
│ Ollama │ │ ┌──────────────────┐ ┌────────────────┐ │
│ (Desktop) │◄──────── :11434 ────►│ │ OpenSearch 3.7 │ │ OS Dashboards │ │
│ qwen3.5:9b-mlx│ host.docker.internal│ │ os-hybrid:9200 │ │ :5601 │ │
└───────────────┘ │ └────────┬─────────┘ └────────────────┘ │
│ │ │
│ ┌────────▼──────────┐ │
│ │ MCP Server │ FastMCP + UV │
│ │ mcp-server:8000 │ hybrid_search │
│ │ │ rag_query │
│ │ │ index_document │
│ │ │ cluster_health │
│ └───────────────────┘ │
│ │
│ Init Jobs (run in sequence): │
│ 1. template-job index template │
│ 2. embedding-job all-MiniLM-L12-v2 │
│ 3. connector-job Ollama → Qwen 3.5 │
│ 4. pipeline-job chunking/hybrid/RAG │
│ 5. seed-job 30 articles │
└────────────────────────────────────────────────────────────┘
Ollama runs on the host, not inside the cluster — pods reach it through host.docker.internal:11434. The agent figured that out by itself. Single-node OpenSearch, 10Gi PVC, no security (it's a dev cluster), MCP server on SSE port 8000. Five init jobs that run in sequence, each waiting for its upstream dependencies through retry loops.
I said yes. The agent started building.
The implementation was fast. The recovery was where it got interesting.
Seven commits. Forty-plus files. The agent scaffolded the entire project in minutes.
Six Helm charts (OpenSearch parent, MCP server, five init jobs). A skaffold.yaml with six Docker images building in parallel. A justfile with shortcuts for skaffold run, kubectl port-forward, and job logs. Every init job is a shell script with until loops — each waits for OpenSearch, for the index template, for the embedding model to deploy, for the pipeline to be ready. If a dependency isn't there, the job pauses. If it is, the job runs. This makes the whole stack re-runnable: delete a job and re-create it, and it picks up where it left off.
There's a satisfying recursion here. The loop is a five-phase workflow of gates and retries. The jobs it wrote are retry loops with dependency checks. The agent built code that mirrors its own architecture.
The MCP server landed next: FastMCP, four tools:
-
hybrid_search(query, index?, k?)— neural + full-text, z-score fused -
rag_query(question, index?, k?)— retrieves context, generates answer -
index_document(id, title, body, index?)— chunks + embeds + indexes -
cluster_health()— cluster status
The agent ran its own draft gate: helm lint and helm template on seven charts, skaffold render, sh -n on every script. Clean. It pushed the branch, opened draft PR #1, and reported back.
Technically, the loop was done: proposal → agreement → implementation → draft_gate → draft_pr. Five phases, zero handoffs.
But the stack hadn't been deployed yet. And that's where everything went sideways — then sideways again — then somehow straightened out.
Five bugs. One autonomous agent. Zero panicking.
The first skaffold run kicked off and immediately hit a wall. The agent didn't wait for me to notice. It read the logs, diagnosed the problem, patched it, and redeployed. Five times.
Bug 1 — "Connector credential is null or empty list": OpenSearch 3.7 demands a credential object even for local Ollama endpoints that don't use authentication. The agent added a dummy key (openAI_key: "ollama-local") and — crucially — switched the connector from Ollama's native /api/chat to the OpenAI-compatible /v1/chat/completions, which is what the RAG processor expects. Two fixes in one pass. Redeployed. Connector created.
Bug 2 — The one-character regex disaster: RAG queries hit a 400: "Connector URL is not matching the trusted connector private endpoint regex". The agent had written ^http://host.docker.internal:11434:.*$ — a colon after the port. The actual URL was http://host.docker.internal:11434/v1/chat/completions — a slash. The difference between : and / broke the entire pipeline. The agent found it, flipped the character, and redeployed. The kind of bug a human would spend 20 minutes on. The agent caught it in one.
Bug 3 — "Model 'ollama-qwen3.5' not found": The connector was passing the OpenSearch-registered model name to Ollama, but Ollama expects the raw model name (qwen3.5:9b-mlx). The agent traced the call chain — OpenSearch RAG processor → connector → Ollama API — and updated the MCP server to pass the correct model name in ext.generative_qa_parameters. Fixed.
Bug 4 — Memory circuit breaker: Seed-job bulk indexing hit 429 rate limits. JVM heap at 93% on 2g allocation. The agent bumped it to 4g (-Xmx4g -Xms4g, container limits 4g/8Gi), redeployed, and the jobs succeeded. The agent understood enough about JVM memory pressure to recognize the root cause without being told.
Bug 5 — The chunk-deletion disaster: This is the one where the agent almost nuked the project. While cleaning up orphaned model registrations, it deleted 14 documents from the ML system index — thinking they were duplicates. They weren't. They were the embedding model's chunks (_0 through _13). The model went to DEPLOY_FAILED. The embedding pipeline was dead.
Most agents would either not notice, or notice and keep retrying the same broken state until you killed the session. The loop did something else: it recognized the mistake. It saw DEPLOY_FAILED on the model, understood that the chunks it deleted were critical, deleted the broken base model document, let the embedding-job re-register fresh with a new model ID, and re-ran the pipeline and seed jobs. The stack healed itself.
That pattern — deploy, hit error, read logs, diagnose, patch, redeploy, verify — repeated five times. The agent operated the stack as well as it built it.
The moment it worked
After the last fix, the cluster settled. Thirty articles seeded. Both models deployed. Cluster health green.
A hybrid search returned ranked results with fused neural and full-text scores. A RAG query against the pipeline — retrieves context first, then calls Ollama through the connector, returns the answer grounded in the retrieved documents:
{
"query": { "match": { "body": "inverted index TF-IDF BM25" } },
"ext": {
"generative_qa_parameters": {
"llm_model": "qwen3.5:9b-mlx",
"llm_question": "Based on the context, explain what an inverted index is and how TF-IDF or BM25 ranking works."
}
}
}
The system answered:
An inverted index is a structure that maps each unique word to the documents containing it, allowing for fast retrieval based on query terms. Regarding ranking, TF-IDF measures term frequency versus inverse document frequency, while BM25 improves upon this method by saturating term frequency.
Grounded. Accurate. Sourced from the 30 articles the agent itself had seeded. All four MCP tools exercised and verified over SSE. The agent updated the draft PR with the validation results.
What I'd do differently (and what I won't)
Loop-police is not optional. Without it, the agent would have retried that failing deploy until the heat death of the universe. With it, the agent flagged the DEPLOY_FAILED state transition, recognized it had caused the failure, and pivoted to recovery. If you're running autonomous agents, you need a stall detector. Period.
The agreement gate is the whole point. Autonomous coding agents without a plan-agreement phase are just hyperactive interns with commit access. The loop doesn't move from proposal to implementation until you say yes. That one rule eliminates the most common failure mode of AI coding: the agent building the wrong thing while you watch helplessly.
State files make long-running sessions possible. .loop-state.json tracks phase, branch, PR number, and plan history. If the session drops — and with tools running 40+ minute autonomous sequences, it will — the loop resumes without context loss. This is infrastructure, not a nice-to-have.
Retry-loop init jobs are self-healing infrastructure. Each job waits for its dependencies through until loops. You can tear down a job and recreate it — it'll wait, detect its prerequisites are met, and proceed. The loop wrote infrastructure that works the way the loop itself works. That's either poetic or recursive. Probably both.
The draft gate is the QA step you'll never do yourself. The loop ran helm lint, helm template, skaffold render, and sh -n before opening a PR. No human on my team does that for every commit. The agent does it because the workflow demands it. Fail the gate, and the loop doesn't proceed to PR. This catches broken manifests before they reach the repo — before you even see them.
The loop isn't magic. It's process.
That's the real takeaway. Autonomous coding agents don't fail because they're not smart enough. They fail because they lack structure — no plan phase, no review gate, no self-check, no stall detection.
The loop adds that structure. It turns an agent from a code generator into a collaborator: you get a plan, you approve it, the agent builds, it self-corrects, and it ships a draft PR. You review, you merge, you move on.
I didn't write a single line of code. But the loop didn't build my RAG stack by accident. It built it because the process forced it to plan, verify, and recover — the same way a good engineer would.
That's how the loop worked.
Top comments (1)
I'm impressed by the level of autonomy demonstrated by the loop, particularly its ability to self-correct and recover from errors, such as the incident with the 14 model chunks. The introduction of loop-police as a companion skill to prevent infinite loops and stagnation is a clever solution to a common problem in autonomous coding agents. The fact that the agent can resume its state from
.loop-state.jsoneven after a session crash or interruption is also a significant advantage. I'm curious to know more about the potential applications and limitations of this technology, especially in terms of handling complex, multi-component systems like the OpenSearch cluster and RAG pipeline described in the article - what are some potential use cases where the loop could be particularly valuable, and what are some challenges that it may still face?