DEV Community: paul_h

Loop Engineering: Building an Agent Loop with agent-runbook

paul_h — Wed, 17 Jun 2026 09:02:40 +0000

Recently, another interesting new term has appeared in the AI industry.

Loop Engineering.

If you follow the AI space, you've probably seen it everywhere in the past couple of days. It's all over X, all over various social media, and quite a few people are discussing it in group chats too.

Recently Addy Osmani formally organized this concept into Loop Engineering — the fourth Engineering after Prompt Engineering, Context Engineering, and Harness Engineering.

What is a Loop? Here's a concrete scenario:

You have a project with 16 failing tests. Previously you'd do this: run the tests, see what failed, tell Claude "fix this", it fixes it, you run the tests again, find new issues, say something again... back and forth, you are the person driving the loop.

The idea behind Loop Engineering is: you no longer manually drive it round by round. You define the goal (all tests pass), define what to do each round (run tests → fix code), define constraints (can't modify test files), then let go. The system runs on its own until the goal is met.

/goal Is Not Enough

At this point you might say: doesn't Claude Code already have the /goal command? Can't I just /goal "all tests pass" and be done?

On the surface, yes. /goal gives you a completion condition, and Claude works on its own until it's satisfied. But after using it a few times you'll notice the problem — the goal is defined, but the agent still won't work properly. Because you only told it "what counts as done", you didn't tell it "what to do each round".

/goal "all tests pass" — what did it do:

Tells the agent "keep going until this condition is met"
At the end of each round, an independent model judges whether the goal is satisfied
The agent has complete freedom in what it does each round

What it doesn't do:

Doesn't define the internal structure of each round. In /goal the agent does whatever it wants each round. Maybe the first round it runs tests + fixes code, the second round it suddenly goes refactoring, the third round it modifies test files.
No iteration-level constraints. /goal only has a termination condition. There's no guardrail like "only modify one file per round", and you can't control when the agent goes out of bounds.
Not reusable. /goal "all tests pass" is gone once you type it. Next time you switch repos or switch people, you have to type it all over again.
Not auditable. When your boss asks "what's the logic of this automated fix workflow", you can't show them /goal.

To summarize: /goal solves "keeping the agent from stopping", but doesn't solve "making the agent follow the rules".

What you need is a place to write down the loop's structure, constraints, and goals — not a one-time command typed into the terminal, but a file that can be committed to the repo, where anyone who gets it can run it and get the same behavior.

agent-runbook: The Contract Format for Loops

This is what agent-runbook does.

agent-runbook is an open source project (github.com/KnoxOps/agent-runbook), it's not the execution engine for loops, but rather the contract format for loops. You use YAML to declare "what to iterate on, when to stop, what the constraints are for each round", and the compiler generates a SKILL.md for you — this is the reusable instruction format for Claude Code and Codex, put it in your project and it can be directly invoked with claude --skill.

A loop step has three elements:

body: what to do each round (the rhythm of observe → act → verify)
goal: when to stop (must be a machine-verifiable condition)
max_iterations: safety boundary (exceeding this number means the design has a problem, prevents burning tokens)

There's also one more key thing: quality_check. This is an iteration-level guardrail — after each round it checks whether the agent went out of bounds (e.g. modified files it shouldn't have). If blocking: true, the round doesn't count as complete if the check fails.

Hands-on: Building an Automated Test Fix Loop

Here's a simple example to show you how we use agent-runbook to build an agent loop.

We're going to build an automated test fix Loop. This loop is simple, the goal is 100% unit test pass rate. Each iteration has only two steps:

run_tests - run the tests, see which ones are still failing
fix - launch a clean context agent to fix the discovered issues

Beyond that, we also need to define our safety boundary: max_iterations. I wonder if any readers here have had the experience of burning through all their tokens with the /goal command — max_iterations is what prevents that.

Here's the full runbook, defined in structured YAML:

name: fix-failing-tests
description: Iteratively fix all failing tests until the test suite is green

steps:
  - id: fix_loop
    type: loop
    description: "Run tests, analyze failures, fix source code, repeat until green"
    goal: "pytest exits with 0 failures (all tests pass)"
    max_iterations: 10
    depends_on: []
    body:
      - id: run_tests
        type: script
        command: "cd examples/fix-loop && python3 -m pytest tests/ --tb=short 2>&1 | tail -60"
        depends_on: []
      - id: fix
        type: agent
        prompt: |
          Look at the pytest failures from run_tests.
          Pick ONE source file that has failing tests and fix the bugs in that file.

          Rules:
            - Only modify files in src/, NEVER modify test files
            - Fix exactly ONE file, then stop immediately
            - Do NOT read or modify any other source files
        depends_on: [run_tests]
        quality_check:
          blocking: true
          rules:
            - "Only files in src/ were modified, not test files"
            - "Exactly one source file was modified"

  - id: present
    type: inline
    prompt: |
      Generate a markdown report summarizing the fix loop results.
      Include:
        - Total iterations taken
        - What was fixed in each iteration (file + bug description)
        - Final test results
        - How cascading dependencies caused failures to clear automatically
      Write the report to fix_report.md
    depends_on: [fix_loop]

From YAML to Executable SKILL.md

Next we need to compile the YAML into a SKILL.md that Claude Code/Codex can directly execute. The generation command is simple:

python3 -m agent_runbook generate runbook.yaml -o output/

The generated SKILL.md looks like this:

---
name: fix-failing-tests
description: ">-"
  Iteratively fix all failing tests until the test suite is green
user-invocable: true
---

## Execution Flow

### Task Context

Before starting execution, initialize `task_context.json`:

```json
{
  "task_id": "<task_id from input>",
  "current_step": 0,
  "current_step_id": null,
  "status": "running",
  "steps": {
    "fix_loop": "pending",
    "present": "pending"
  },
  "updated_at": "<ISO timestamp>"
}
```

Update this file after each step completes. On error, set step status to `"failed"` and overall `status` to `"failed"`.

### Step 1: fix_loop

**Type:** loop
**Description:** Run tests, analyze failures, fix source code, repeat until green

## Iteration Loop

**Goal:** pytest exits with 0 failures (all tests pass)
**Max Iterations:** 10

> This step executes as a loop. The body steps repeat until the goal is met or max iterations reached.

## Loop Body (repeats each iteration)

#### Body Step 1: run_tests

**Type:** script

**Execution:** Execute the following command:
```bash
cd examples/fix-loop && python3 -m pytest tests/ --tb=short 2>&1 | tail -60
```

#### Body Step 2: fix

**Type:** agent

**Execution:** Launch an independent agent with the following prompt file:

Look at the pytest failures from run_tests.
Pick ONE source file that has failing tests and fix the bugs in that file.

Rules:
  - Only modify files in src/, NEVER modify test files
  - Fix exactly ONE file, then stop immediately
  - Do NOT read or modify any other source files


## Goal Evaluation

After all body steps complete, evaluate:

**Goal:** pytest exits with 0 failures (all tests pass)

1. If goal IS met → mark this step completed, proceed to next step.
2. If goal NOT met and iterations remain → reset body steps, start next iteration.
3. If max iterations reached → mark step completed with status "max_iterations_reached", report what remains.

Append a summary to `iteration_history` after each iteration.

### Progress Tracking

After completing this step, update `task_context.json`:
- Set `current_step_id` to `"fix_loop"`
- Set `steps.fix_loop` to `"completed"`
### Step 2: present

**Type:** inline

## Execution
Follow these instructions:

Generate a markdown report summarizing the fix loop results.
Include:
  - Total iterations taken
  - What was fixed in each iteration (file + bug description)
  - Final test results
  - How cascading dependencies caused failures to clear automatically
Write the report to fix_report.md


### Progress Tracking

After completing this step, update `task_context.json`:
- Set `current_step_id` to `"present"`
- Set `steps.present` to `"completed"`

What does the generated SKILL.md contain? It translates the contracts you declared in YAML into execution instructions that the agent can understand:

iteration_history: requires the agent to record what was done each round and whether the goal was met, forming structured iteration memory
goal evaluation: the judgment logic after each round — if met then stop, if not met then continue, if limit reached then report
progress tracking: tracks overall progress through task_context.json, supports checkpoint resume

Running It: 3-Round Convergence

Now we can trigger this skill to run in Claude Code:

The run included three iterations:

Iteration 1: calculator fix → 6 failures disappeared

Iteration 2: validator fix → 5 failures disappeared

Iteration 3: formatter fix → all green

Finally, this is also what we defined earlier in the runbook — a fix_report.md to be produced after the loop.

Key Points for Designing a Good Loop

Choose the right task. Not all tasks are suitable for loops. A good loop task has two characteristics: objective feedback signals (test results, lint output, whether compilation passes), and the ability to make incremental progress building on the previous round. Fixing tests, code migration, and performance optimization are all good candidates. Tasks requiring one-time creative decisions (architecture choices, naming) are not suitable.
Write the goal as a decidable end state. "pytest exit 0" is a good goal, "better code quality" is not. The agent must be able to determine true or false on its own through tool output, otherwise the loop never knows whether it should stop.
Keep the body in an "observe—act" rhythm. First use script steps to see the current state clearly (run tests, run lint), then use agent steps to make decisions and modifications. Don't let the agent observe, act, and verify all in one round — split them up, each step has clear responsibilities, and when something goes wrong it's easier to locate.
Leave an exit for failure. max_iterations is not the number of rounds you expect, but a safety valve for "exceeding this number means the approach has a problem". A normal loop should converge well below the upper limit. If it maxes out, it means the goal is too hard or the body design has flaws, and human intervention is needed.

agent-runbook: More Than Just Loops

Due to the AI product I'm developing, I frequently need to write many long-running, as-error-free-as-possible DevOps skills for SREs.

During debugging I often encounter two types of problems:

One is agents not following instructions — you tell it to only restart the service, and it goes ahead and changes the configuration too.
The other is in a complex multi-step skill, agents not collaborating according to the established norms, where the output from the previous step isn't read by the next step at all, or it's read but the format is wrong.

Based on these problems, I developed agent-runbook: a contract-based skill generation tool, where the generated SKILL.md can be directly used as a skill integrated into the Claude Code/Codex ecosystem.

Its core philosophy is: use contracts to constrain agent collaboration, instead of relying on prompts and hoping for the best.

This table gives you a quick sense of how agent-runbook differs from /goal:

	/goal	agent-runbook
Per-round structure	Agent does whatever it wants	Body declaratively defines each round's steps
Iteration constraints	None, only a termination condition	quality_check guardrails, out-of-bounds doesn't count as complete
Inter-step communication	Relies on LLM context passing	JSON Schema files, inspectable, parallel-readable
Error recovery	Start over	Checkpoint & Resume, pick up from where it crashed
Build-time checks	None	DAG cycle detection, schema reference validation, contract closure checks
Reusability	Gone once you type it	Commit to repo, anyone can run it with the same behavior

Loop is a step type added on top of this foundation — when your task requires iteration, use the same contract-based approach to define the loop's body, goal, and constraints.

You don't have to start from scratch either. open-devops-skills is a production-grade DevOps skill library built on agent-runbook, currently featuring infrastructure/cloud resource cost optimization skills, with more DevOps scenarios to be expanded in the future. You can use them directly, or use them as reference for designing your own skills.

It's also worth mentioning that agent-runbook itself is not limited to DevOps. Any scenario requiring multi-step orchestration, inter-agent collaboration, and long-term reliable operation is suitable — code migration, security auditing, documentation generation, data pipeline validation. As long as your task can be broken down into "steps + contracts + dependencies", it can be expressed with a runbook.

The repo is at github.com/KnoxOps/agent-runbook, feel free to try it out and give feedback. If you have a workflow where you're repeatedly prompting agents manually, try writing it as a runbook — you'll find that once it becomes a contract, the cost of debugging and reuse drops significantly.

I Asked Claude to Map My Infrastructure. Then I Asked a Purpose-Built Tool.

paul_h — Mon, 15 Jun 2026 07:13:09 +0000

I manage a small stack. Three Linux VMs, one Kubernetes cluster, maybe 20-something services total. Not big. But underdocumented — the kind of environment where you SSH in and discover things you forgot were running.

Last week I ran the same task through two different AI tools: "tell me what's running, how it connects, and what looks risky." One is a general-purpose LLM (Claude). The other is a purpose-built AI SRE tool. Same environment, same ask. The results were... instructive.

The task

Simple brief: infrastructure discovery. I want a full picture — services, dependencies, topology, risks. The kind of thing a new hire would spend their first week piecing together from wikis that haven't been updated since 2023.

Claude Code (Opus model)

My prompt:

"I manage a small infrastructure — 3 Linux VMs (172.30.0.41, 172.30.0.42, 172.30.0.43) and a Kubernetes cluster. SSH access is already configured. Help me understand what's running across this environment — I want a full picture of my services, dependencies, and topology."

I'm running Claude Code locally with the Opus model — their flagship tier. Claude didn't ask questions. It just started SSH-ing in.

" width="800" height="354">

Five minutes later it handed me a report. And honestly? It was better than I expected.

What Claude delivered:

Identified all three VM roles correctly (API Gateway, Order Processing, Data Tier)
Drew an ASCII topology showing Nginx routing to backend services with canary weights
Built a full service table — host, port, tech stack, notes
Mapped the Redis Sentinel cluster including a stale replica on a decommissioned node
Enumerated every K8s namespace and workload
Traced the observability pipeline (node_exporter → Prometheus, OTel → Jaeger, Datadog agents)
Flagged four real issues: dead Redis replica, broken image pulls in aigc-app, active canary split, multiple knoxd versions

Five minutes. No hand-holding. For a "quick, what's running here?" sweep, this is genuinely useful.

Where it stops

Here's what I noticed after the initial "wow, that was fast" wore off.

The output is a wall of markdown. Accurate, mostly. But flat. Everything has the same weight — a critical single-point-of-failure sits next to a cosmetic naming inconsistency. No severity. No priority.

More specifically:

No topology visualization. I got an ASCII diagram. It's readable for 6 machines. At 60 machines, it's unreadable. At 600, impossible.

No business grouping. Claude listed every service but couldn't tell me which ones form the e-commerce flow vs. the logistics flow vs. the platform layer. That requires domain context it doesn't have.

No risk assessment. Four issues found, but no severity classification. The dead Redis replica and the cosmetic knoxd naming thing are presented with equal weight.

No quality gate. Nobody verified whether Claude's topology was actually correct. It connected things confidently — but was the canary weight really 90/10? I'd need to go check.

No persistence. Close the chat window. The report is gone. Tomorrow I'd run it again and get a slightly different exploration path, slightly different findings.

No depth control. I can't say "that Business Island looks risky, go deeper on it." It's all-or-nothing.

This maps to a pattern I keep seeing across industries. In legal tech, people noticed the same thing — general LLMs are good at summarizing contracts but can't do precision clause verification. In finance, ChatGPT can describe how to post a journal entry but can't actually post one. The dividing line is consistent: general AI is a thinking tool; specialized AI is an acting tool.

When the task is "reason about this data and explain it to me" — general tools are great. When the task shifts to "build a structured, persistent, verifiable model of my environment" — you've crossed into territory they weren't designed for.

Purpose-built tool, same task

For comparison, here's what happens when I send one line to Knox (our purpose-built AI SRE tool — yes, this is our product, stating that upfront):

"Run a full infrastructure discovery on our production environment."

Shorter prompt. No need to explain the environment — it already has connectors configured.

Twenty minutes later:

" width="800" height="450">

The differences that matter:

Visual topology — not ASCII art, an interactive service relationship graph
Business Islands — services auto-grouped by business function with criticality labels
Risk Triage — findings ranked by severity with a distribution chart
Persistence — results stored in a graph database, queryable later
Depth on demand — "Deep Analysis Available" button for any Business Island

How it got there — a team of agents, not a single model:

8" width="800" height="450">

This is the work process, not a deliverable. Multiple specialized agents collaborated — one coordinated the task, one did the actual discovery, one quality-checked the findings — flagging 9 uncertain items for human review instead of presenting everything with equal confidence.

The scale question

We ran this on 5-6 machines. The gap is already visible. But this is the minimum-gap scenario.

At 60 servers across multiple environments, Claude's context window fills up. You'd need multiple sessions, manual stitching, and the "flat markdown" problem becomes unbearable. The gap doesn't grow linearly — it compounds.

That's not a knock on Claude. A Swiss Army knife is great. But when you need surgery, you reach for a scalpel.

What's your environment look like? At what scale did you find general AI tools hitting their ceiling for ops work?

If you want to try the purpose-built approach: knoxops.app

Agentic Ops: How I Shipped My Vibe-Coded Game to Production

paul_h — Sat, 30 May 2026 07:31:08 +0000

Over the weekend, I vibe coded a cooking game. You combine random ingredients, and the game generates a dish with a score and a snarky review — stuff like "This tastes like regret and too much butter." I'd wanted to build this for a while. Eventually I'll hook it up to an AI model to generate more combinations and even harsher critiques.

One Prompt, One Hour

I opened Claude Code and typed a single prompt:

"Create a cooking game where players combine ingredients to discover recipes..."

An hour of coding and debugging later, I had a working version running on localhost.

The Wall

Then came the real problem: deploying it so my friends could actually play.

AI has collapsed the barrier to building software. But no matter how low the entry gets, even the most seasoned SRE can't rattle off HTTPS configs, domain setups, and nginx routing rules from memory. As a vibe coder, what was I supposed to do next?

The Plan

I spun up an AWS VM, installed a Knox Daemon (Knox is an AIOps product), and connected it to my GitHub repo. Then I told it:

"How I Shipped My Vibe-Coded Code to Production"

It started exploring my codebase. It discussed the task with me, asked clarifying questions, and came back with a full plan — five stages covering pre-checks, building the game, requesting certificates, updating nginx routes, final verification, and documenting what it learned for next time. Nothing would execute until I approved it.

The Execution

I reviewed the plan and hit approve. The agents kicked off in parallel — one checking the environment, one executing changes, another validating the output of each stage. They ran efficiently, every step visible. It looked exactly like a human SRE team at work.

When it was done, the agent handed me a report. I clicked the URL in the report and — there it was. My game. Live. Someone could play it.

30 Minutes

I was doing other things throughout the deployment, so I wasn't always quick to respond when the agent needed input — requirement discussions, plan approval, execution confirmations on my AWS box. Total time from start to live: about an hour. If I'd been fully focused, probably 30 minutes.

The whole experience was striking. More and more people are building things in the AI era. They think about product design and development, but then what? How do you deploy? How do you keep the service running?

I think this is what agentic ops means.

Agentic ops gives you the same answer: describe what you want, and an agent operates the server. Same loop as vibe coding. The output just isn't code anymore — it's a running service.

The endpoint of vibe coding shouldn't be localhost:3000. It should be a link you can drop in a group chat.

AI Agents Mapped My Legacy Production Environment in One Hour.

paul_h — Thu, 28 May 2026 03:55:11 +0000

I inherited a black box.

Three VMs. A hundred-something microservices. Redis, ClickHouse, MySQL, some homegrown database nobody could name. Kafka and Zookeeper thrown in because of course they were.

Nobody knew how the services connected. The original team was gone. The architecture lived entirely in oral tradition, and the last person who could recite it had left six months ago.

This is not a metaphor. This is Tuesday for anyone who's done SRE work long enough.

Setup: 30 seconds, zero footprint

I already had Teleport for daily ops. SSH access, session recording. It worked, I didn't want to break it.

What I did:

Installed knoxd on my Teleport proxy (not on the servers)
AI agent team auto-configured a Teleport connector

That's it. Nothing new on my production machines. The agents ride the Teleport session I already had, with the permissions I'd already defined.

Non-invasive — not in the "we promise it's lightweight" sense. In the "there is literally nothing new running on your production machines" sense.

How it actually works

The agents SSH in through Teleport. Plain SSH commands, same ones you'd type yourself.

What makes this safe rather than terrifying:

	Auto-run	Requires human approval
Read-only	`ps`, `ss`, `cat /proc/net/tcp`, `nginx -T`	—
Mutating	—	`kill`, `systemctl restart`, `rm`

The sandbox: strict AST parsing + default-deny whitelist. The agents can look at everything but touch nothing without asking.

What the agents discovered

Step 1: OS inventory — kernel, distro, packages. All 3 VMs in parallel.

Step 2: Process mapping — ps aux, parsed. Hundreds of processes tagged with binary path, resource footprint, parent-child relationships.

Step 3: Process → Service resolution

Check name service first
If unregistered (most weren't — legacy system), infer from install path
Flag for human confirmation before writing anything back

The AI doesn't hallucinate service names into your architecture map. It asks.

Step 4: Service → Business Island grouping

A business island = logical grouping by business function (billing, user auth, order processing). The thing that exists in every architect's head but never in any document.

Step 5: Connection mapping — four evidence sources, cross-referenced:

Source	What it reveals	Example
Network connections (`ss -tnp`)	Live TCP dependencies	Port 6379 → Redis, port 9092 → Kafka
Config files	Declared dependencies	`kafka.brokers: kafka-01:9092` in YAML
Access logs	Actual call patterns	Who calls whom, how often
LB configs (nginx)	Ingress chain	Domain → LB → real server

Cross-reference. Resolve conflicts. Draw edges.

One hour.

What I got

Architecture diagrams — topology maps of each business island, services as nodes, dependencies as edges, data flows labeled. The kind of diagram you'd pay a consultant a week to produce.

High-risk report:

Single points of failure
Circular dependencies
Kafka topics with no visible consumer group
One Redis instance holding session state for 6 business islands, zero isolation

Things I needed to know. Things dashboards would never show me.

The cost

Zero.

Knox gives free credits on signup. Enough for a small cluster for a long time. No credit card. No trial-that-converts-to-paid. One binary on a jump host.

Why this matters

Most AIOps tools treat metrics as the final answer. They're not. They're the starting point.

Real outages hide in blind spots:

System logs nobody tails
Manual changes nobody tracked
Config drift APM tools don't see

To find root cause, you have to log into machines and build an evidence chain. That's what humans do. That's what these agents do.

Monitoring tells you a metric crossed a threshold. It doesn't tell you:

Service X and Y form a circular dependency that will cascade
Your session store is a single point of failure for half the platform

Those aren't metric problems. They're structure problems. LLMs are uniquely good at structure — if you give them a way to see it without breaking anything.

Safety model

Letting AI touch production should sound terrifying. That's why:

AST-parsed command validation — not string matching, actual syntax tree analysis
Default-deny whitelist — everything blocked unless explicitly allowed
Human-in-the-loop — any destructive action requires a plan + approval
Connector model — agents use paths you already trust (Teleport, SSH, AWS, Prometheus)

The agents never need their own access path. They never open a new hole in your security posture.

That's the difference between an agent you'd let near production and one you wouldn't.

What I'm building

It's called KnoxOps. Core idea: infrastructure is an object graph, not a flat list of resources. Model it that way and LLMs can reason like a senior SRE — tracing dependencies, calculating blast radius, finding what dashboards miss.

The goal: delegate routine SRE toil so developers can focus on building.

More connectors coming. The principle stays the same: use the access paths you already trust.

If you've inherited a system nobody understands — I'd like to hear from you.

I'm the founder of KnoxOps. Currently in open beta — use code DEVTO26 for 10,000 free credits on signup.