DEV Community: Jason Zhou

Context Engineering: The Complete Guide (2026)

Jason Zhou — Fri, 24 Jul 2026 05:49:00 +0000

Manus measured their production agents: 100 input tokens for every 1 output token. Read that again. The model's "intelligence" in an agent system is mostly a function of those 100 tokens - what got included, what got cut, how it's arranged. The Karpathy framing: the LLM is a CPU, the context window is RAM, and you are the operating system deciding what loads into memory.

Prompt engineering asks "how do I phrase the ask?" Context engineering asks "what does the model see at this exact moment?" The second question is where production agents live or die. Here's the field in one pass: why context degrades, the four management strategies, the four failure modes, and the cache economics underneath it all.

Prefer to watch? Here's the full walkthrough of putting this into practice with a structured .agent folder in Claude Code:

Why Long Context Degrades: Context Rot

Bigger windows didn't fix this. Two structural reasons:

Attention is quadratic and finite. Every token attends to every other token - n² relationships. More tokens means each one gets a thinner slice of attention. Anthropic's framing: context is a resource with diminishing marginal returns. Each appended token spends attention budget.

Performance decays before the window fills. The measured phenomenon is called context rot. Galileo's benchmark: GPT-4o at 98.1% accuracy on short contexts dropped to 64.1% at scale - same model, same task, more tokens. This affects every transformer. It's a gradient, not a cliff, but a 34-point slide will absolutely ruin your agent's afternoon.

And tool definitions compound it. Berkeley's function-calling leaderboard found a quantized Llama 3.1 8B that worked fine with 19 tools and fell apart at 46. The window had room; the model's discrimination didn't. Complexity, not length, was the killer.

What "Context" Actually Includes

Wider than most people draw it - four layers, each failing differently:

Layer	Contains	Failure mode
Instructions	System prompt, rules, few-shot examples	Bloated rules dilute everything downstream
Knowledge	Retrieved docs, user prefs, facts	Too much = lost model; too little = confabulation
Tools	Definitions, call results, errors	Tool overload (the 46-tool collapse)
History	Prior messages, decisions, trajectory	Stale conclusions steering current work

Context engineering is budget allocation across these four. Spend where the task needs it; cut everywhere else.

The Four Strategies

1. Offloading - the filesystem is free memory

Don't carry what you can fetch. Manus's pattern: scrape a webpage, keep a 500-token summary plus the URL, drop the 50,000-token body. 100:1 compression, fully reversible - the agent re-fetches if it actually needs the details.

The other offloading move: the agent maintains a todo.md and re-reads it before each step. External memory doubles as an attention anchor - rewriting current goals into the recent end of context keeps long tasks from drifting. Same reason you write a checklist instead of memorizing it.

2. Retrieval - just-in-time beats preloading

Classic RAG retrieves once, up front. Agentic retrieval hands the model lightweight pointers - file paths, query templates, URLs - and lets it pull what each step needs. Claude Code is the reference implementation: it never loads your database into context; it writes a query, stores results, reads slices with head. The pointer is cheap; the data stays on disk until summoned.

Hybrid is the practical answer: preload the stable essentials (CLAUDE.md), let the agent explore the rest. Where you draw the line is the craft.

3. Isolation - don't share everything with everyone

The trap in multi-agent systems: a giant shared context everyone reads. Now every agent wades through everyone else's noise and bills for the privilege.

Treat inter-agent communication as interface design: a research sub-agent returns verified findings and decision points - not its browsing history, dead ends, or internal monologue. Anthropic's multi-agent research system proved the economics: separate contexts per sub-agent cost 15x the tokens but delivered 90.2% better task performance. Isolation is what you're buying. (This is the entire thesis of sub-agents.)

4. Compression - shrink without losing the constraints

When the window fills, summarize and restart - Claude Code triggers this automatically at 95% utilization. The subtle art is what survives compression. Importance is lagged: the detail you drop at step 5 becomes load-bearing at step 50. Priority goes to facts that constrain future action - what failed, what got created, what's been ruled out, what's still unknown.

Cheapest variant: clear old tool results. A file dump from 40 turns ago is almost never needed verbatim - it's the lowest-risk deletion in the window.

The Four Failure Modes

Name them and you'll start seeing them everywhere:

Poisoning - one hallucination enters context and compounds. DeepMind's Gemini-plays-Pokémon logged the canonical case: a false game-state claim landed in the goal section, got re-read every turn, and the agent chased an impossible objective for dozens of cycles. Errors that enter context self-confirm.
Distraction - past ~100K tokens, models start pattern-matching their own history instead of reasoning fresh. Gemini 2.5's report describes the agent repeating past actions rather than planning. Databricks found worse: distracted models default to summarizing the context - ignoring your instruction entirely.
Confusion - irrelevant material degrades output even when it's ignorable. The 46-tool collapse, again. Models don't ignore noise; they pay attention tax on it.
Clash - contradictions across turns derail reasoning. Microsoft/Salesforce sharded benchmark tasks across multi-turn conversations: average performance fell 39%. Their line: "when LLMs take a wrong turn in a conversation, they get lost and do not recover."

The Money Layer: KV-Cache

Agent contexts share a shape - stable prefix, growing tail - and inference engines cache the prefix computation. Claude Sonnet pricing makes the stakes plain: cached input $0.30/M tokens, uncached $3.00/M. At a 100:1 input ratio, cache hit rate basically is your unit cost. Three rules:

Freeze the prefix. One changed token invalidates everything after it.
Append-only. Never edit history. And serialize deterministically - JSON key order isn't guaranteed in most languages, and nondeterministic serialization silently breaks caching.
Mask, don't remove, tools. Tool definitions sit early in context; add/remove mid-session and the cache resets. Keep the definition set stable and constrain choices via logit masking instead.

Two Counterintuitive Practices

Leave failures in. The instinct after a failed tool call is to clean up and retry on a fresh slate. Wrong - the error trace is evidence. Models that see their own failure update away from repeating it; models with scrubbed history walk into the same wall. Manus treats error recovery as one of the clearest signals of real agentic behavior, and notes benchmarks systematically under-measure it.

Inject structured variety. A context full of similar action-observation pairs becomes accidental few-shot pressure - the model mimics the pattern even when the situation changed. Varying serialization templates and phrasing breaks the rhythm.

When to Bother

If your agent is single-turn Q&A: don't. Strategy overhead beats no strategy. The escalation triggers: context regularly past ~30K tokens, more than ~20 tools, or 1,000+ sessions/day. Then add strategies in order of pain - usually compression first, retrieval second, isolation when you go multi-agent.

One more trendline worth holding: stronger models make context engineering more valuable, not less - capability unlocks longer tasks, longer tasks mean more context pressure. The window is RAM. The model is the CPU. You're the OS - and the OS is where agents are won.

Originally published on AI Builder Club, where the full free agent-engineering curriculum lives.

MCP Internals: STDIO, SSE, and JSON-RPC Explained

Jason Zhou — Thu, 23 Jul 2026 05:49:00 +0000

You've pasted this config a dozen times. Can you explain what it does?

{
  "mcpServers": {
    "filesystem": {
      "command": "npx",
      "args": ["-y", "@modelcontextprotocol/server-filesystem", "~/Downloads"]
    }
  }
}

Most builders run MCP servers daily without being able to answer. That's fine until something breaks - a server won't start on Windows, tools mysteriously don't appear, or you want to build your own client and realize the magic was never magic. This is the wire-level tour: transport, message format, and the exact six-step loop behind every MCP tool call.

What That Config Actually Is

It's a command line, disassembled into JSON. The client reassembles and runs it:

npx -y @modelcontextprotocol/server-filesystem ~/Downloads

npx - Node's run-a-package-without-installing tool (uvx is the Python twin)
-y - skip the "install this package?" prompt, because no human is present to answer
the package name - an ordinary npm package that speaks MCP
~/Downloads - an argument the server itself defined (here: the directory it's allowed to touch)

That's the entire trick. An MCP server is a process your client spawns, no different in kind from anything else you'd launch from a shell. Which means you can skip npx entirely and run a local copy - "command": "node", "args": ["/path/to/server.js"] - fully offline, immune to upstream version changes, and (per the security article) the safest way to run anything important.

The Windows gotcha that bites everyone once: Windows' default shell can't exec Unix-style commands directly, so the config needs wrapping - "command": "cmd", "args": ["/c", "npx", ...]. If your server "just doesn't start" on Windows, it's this.

Transport: STDIO and SSE

MCP defines what messages say, not how they travel. Two transports:

STDIO - for local servers. The client writes JSON to the child process's stdin; the server answers on stdout. The same plumbing as cat or any Unix pipe - you can literally drive a server from your terminal:

npx -y @modelcontextprotocol/server-filesystem ~/Downloads \
  <<< '{"method":"tools/call","params":{"name":"list_directory","arguments":{"path":"~/Downloads"}},"jsonrpc":"2.0","id":1}'

A directory listing comes back as JSON. No network stack, no ports, no auth handshake - process isolation is the security model. This is why local MCP servers feel instant.

SSE - for remote servers. Client connects over HTTP; the server pushes via Server-Sent Events:

{ "mcpServers": { "browser": { "url": "http://localhost:8000/sse" } } }

Use it when the server lives elsewhere - a shared team service, a SaaS endpoint, anything not on your machine. Trade-offs are the usual networked ones: latency, availability, and a real authentication story to care about.

The Message Format: JSON-RPC 2.0

Every MCP message is JSON-RPC - a request/response convention from long before LLMs:

// request
{ "jsonrpc": "2.0", "id": 1, "method": "tools/call",
  "params": { "name": "query_db", "arguments": { "table": "users" } } }

// response (id matches)
{ "jsonrpc": "2.0", "id": 1,
  "result": { "content": [{ "type": "text", "text": "..." }] } }

id pairs responses to requests so multiple calls can be in flight. Errors come back in a structured error field with codes. The methods you'll see constantly: tools/list (what can you do?) and tools/call (do it). The protocol also specs resources, prompts, and sampling - but tools are ~90% of real-world traffic today.

The Full Loop, Six Steps

Here's a complete trace of "which courses does teacher Zhang teach?" against a database server (two tables: teachers, courses - so two tool calls). This pattern is identical across every client:

Init. Client spawns/connects to each configured server, calls tools/list, collects every tool's name + description + parameter schema into a catalog.
Prompt assembly. Your question + the tool catalog go to the LLM.
Model decides. Returns a structured tool call: search_teachers({ name: "Zhang" }). Crucially - the model returns intent, JSON describing a wish. It has no hands.
Client executes. Translates the wish into an actual tools/call to the right server, gets the teacher's ID back.
Loop. Result is appended to the conversation; model sees it, requests call #2: search_courses({ teacherId: ... }). Steps 3-4 repeat until the model has enough.
Synthesis. Model writes the human answer: "Zhang teaches Advanced Mathematics."

A two-hop question = 13 log entries in a typical client: one tools/list, four LLM round trips, two tool executions, plus bookkeeping. Multi-step "agentic" behavior is this loop, repeated. Nothing else is happening. (If you've read the agent loop, you've recognized it - MCP just standardizes the tool side.)

How the Model Knows About Tools: Two Schools

Packet-capture different MCP clients and a fault line appears - there are two ways to teach a model its tools:

School 1: native Function Calling. The client passes the tool catalog through the API's tools parameter. The model emits structured tool_calls, reliability backed by constrained decoding. Clean, robust - but only works with models fine-tuned for tool use, which is why some clients gray out MCP for certain models.

School 2: system-prompt convention. The client writes the entire tool protocol into a giant system prompt - tool list, XML-ish call format, usage rules - and parses the model's text output for tool invocations. Measured in the wild at ~42,000 characters of system prompt before you've said a word. Burns tokens, fragile to format drift - but works with any model that can follow instructions, no tool-use fine-tuning required.

Same protocol underneath; the difference is purely how the client talks to its LLM. This single distinction explains both why some tools support every model and why those tools cost more per request.

Build a Toy Client - It's One Loop

Nothing demystifies MCP faster. The whole thing in pseudocode:

1. spawn configured servers, collect tools/list into a catalog
2. loop:
   - send messages + catalog to LLM
   - if response is text → print, await user
   - if response is tool call →
       route to owning server via tools/call,
       append result to messages,
       continue

An afternoon's work against the official SDK (the MCP 101 guide covers the server side). Log every message as JSON while you're at it - watching your own client's traffic teaches more than any diagram, and it doubles as the security audit habit of knowing exactly what your servers send and receive.

What to Keep

Four load-bearing facts:

A server is a spawned process (or an HTTP endpoint). The JSON config is a disassembled command line.
Transport is STDIO locally, SSE remotely. Messages are JSON-RPC 2.0 either way.
The model only ever outputs intent. The client executes. Every "AI did something" is this handoff.
Tool awareness comes via Function Calling or via system prompt - which one your client uses determines model compatibility and token cost.

The protocol's genius was never sophistication - it's that it standardized something dumb enough for everyone to implement. USB-C for AI tools: boring on the wire, transformative in the ecosystem.

Originally published on AI Builder Club, where the full free agent-engineering curriculum lives.

Agent Memory Systems: The Complete Guide (2026)

Jason Zhou — Wed, 22 Jul 2026 05:48:00 +0000

You spent 30 minutes teaching your agent the project: TypeScript, Vitest, Supabase, deploys on Vercel. It nailed the work. You closed the terminal. Tomorrow it knows nothing.

This is not a bug - it's the architecture. An LLM's only "memory" is the context window: the text you send with each request. New session, empty window, total amnesia. The in-conversation memory you experience is just prior messages being re-sent every turn.

For one-shot questions, fine. For an AI that works with you across weeks - remembering your preferences, your stack, last Tuesday's debugging conclusion - you need a memory system bolted on from outside. Here's the full map: the three memory types, where memories physically live, the lifecycle that keeps them useful, and how MemGPT, Mem0, and Claude Code each solved it differently.

Prefer the video version? Watch the full breakdown of whether agent memory is actually solved:

Three Kinds of Memory (Borrowed From Your Brain)

Cognitive science gave agent design its most useful taxonomy. Three types, three jobs:

Type	Answers	Agent examples
Episodic	What happened?	Past conversations, tool-call traces, "approach A failed last time, B worked"
Semantic	What's true?	"User prefers terse answers", "project uses Supabase", "deploys are us-east-1"
Procedural	How do I do it?	Learned workflows: the password-reset sequence, the standard debug routine

A mature agent needs all three. Episodic-only over-indexes on anecdotes. Semantic-only never learns from experience. Procedural-only breaks on anything novel. Most builders start semantic (it's the easiest win - facts and preferences), add episodic when cross-session learning starts mattering, and find procedural memory emerging in things like Skills - which are essentially procedural memory you wrote by hand.

Where Memory Lives: Pick Your Layer

A 107-page 2025 survey (Hu et al., Memory in the Age of AI Agents) sorts storage into three forms. The practical conclusion is short:

Token-level - the layer you can actually use. Memory stored as readable text - facts, summaries, profiles - in files or databases, injected into the prompt when needed. Every production system you've heard of lives here: Mem0, Letta/MemGPT, Zep, Claude Code's CLAUDE.md. The reason is brutal and simple: commercial APIs accept text and nothing else. You can't reach into Claude's attention layers. Your entire design space is what goes in the prompt.

Within token-level there's a complexity ladder: flat (entries + vector search - Mem0's original design), graph (entities and relations for multi-hop queries - Zep's temporal knowledge graph), hierarchical (raw entries → cluster summaries → global abstractions - HippoRAG). Benchmark reality check: flat retrieval ties or beats fancier structures on standard tests. Start flat; graduate only when you observe actual multi-hop retrieval failures.

Implicit (KV-cache tricks, learned memory tokens) - requires model internals; commercial APIs don't expose them. Skip unless you self-host.

Parameter-level (fine-tuning knowledge in) - durable but can't update incrementally, and edits risk catastrophic forgetting of neighbors. Batch domain adaptation only; not a memory system.

The Lifecycle: Write, Maintain, Retrieve

Writing - quality ceiling gets set here

Garbage written = garbage retrieved, regardless of how clever retrieval is. Two workhorse methods:

Extractive: an LLM pulls discrete facts from conversation - "user prefers dark theme", "API limit 100/min". Precise, occasionally misses context. Mem0's core pipeline.
Summarative: compress conversations into running or chunked summaries. Holds context; risks semantic drift over repeated updates.

Use extraction for facts, summaries for conversational context. The bar for writing anything: does this constrain future reasoning? Preferences, decisions, recurring patterns - yes. Everything else - high threshold. A memory system that faithfully saves everything is a garbage heap with an API.

Maintaining - the step everyone skips

Merge: "prefers concise answers" and "likes brief replies" are one memory, not two
Update: "we migrated Postgres → Supabase" must supersede the old fact - Zep soft-deletes with timestamps so history survives but stale facts stop surfacing
Forget: decay by age, prune by access, judge by importance - with one landmine:

Retrieving - where good memories go to waste

Query rewriting fixes the dirty secret of semantic search: questions and stored facts have different shapes. "How's auth handled?" sits far from "JWT validation in auth.ts middleware, 24h expiry" in embedding space. HyDE is the counterintuitive fix: have the model hallucinate an answer first, embed that, search with it. The fake answer is wrong but answer-shaped - and shape is what embedding distance measures.
Hybrid search - BM25 keyword matching catches exact terms (auth.ts); embeddings catch paraphrases ("login system" → "authentication middleware"). Either alone leaves recall on the table.
Filter hard. Three highly relevant memories injected beat ten half-relevant ones. Over-injection is just context pollution with extra steps.

Three Production Architectures

MemGPT / Letta - agent as memory manager. The 2023 OS metaphor: context window = RAM, external store = disk, and the agent itself pages data between them via tools (core_memory_append, archival search). Maximum flexibility; memory quality rides entirely on the model's judgment.

Mem0 - automated pipeline. Every exchange flows through extract-then-reconcile: LLM pulls facts, vector similarity decides new/update/merge. Later added a graph layer for relational queries. On the LOCOMO benchmark, Mem0 beat full-context stuffing - better accuracy, fewer tokens. Precise extraction + precise retrieval > brute force, officially measured.

Claude Code - files, that's it. CLAUDE.md at the project root (auto-loaded, stable conventions), ~/.claude/CLAUDE.md for cross-project preferences, plus topic-split memory files behind an index. Radical properties: fully human-readable, git-versionable, zero infrastructure, agent edits its own memory with file tools. Weakness: no semantic search - retrieval is file names and index discipline. At large memory volumes it strains.

	MemGPT/Letta	Mem0	Claude Code
Managed by	The agent	Pipeline	Agent + you
Storage	Tiered (core/recall/archival)	Vectors + graph	Markdown files
Transparency	Medium	Low	Total
Infrastructure	Framework	Vector DB	None
Fits	Complex memory reasoning	Products needing auto-memory	Dev tools, local agents

The Minimum Viable Memory System

Three steps, one afternoon, no vector database:

A save_memory tool. The agent calls it when something deserves persistence. Writes to a file or a Supabase table - either works at this scale.
Index injection at session start. Put memory titles + one-line summaries in the system prompt; the agent requests full entries when relevant. This is Claude Code's index-then-read pattern, and it's load-bearing: inject summaries, not bodies.
Scheduled hygiene. Merge duplicates, expire stale facts. A monthly manual pass is genuinely enough at the start.

This takes you from "amnesia every morning" to "remembers what matters." Add embeddings, HyDE, graphs when you hit a real retrieval failure - not before.

The One-Sentence Version

Memory ≠ context: context is RAM (visible now, gone at session end), memory is disk (persistent, useless until retrieved into context) - and the bridge between them, retrieval, is where memory systems are actually won. Build the smallest one that stops the amnesia, then earn each layer of complexity with a real failure. Done right, your agent gets better every week you use it. Skipped, every morning is a first date.

Originally published on AI Builder Club, where the full free agent-engineering curriculum lives.

Claude Code Worktrees: File Isolation for Parallel Agents

Jason Zhou — Tue, 21 Jul 2026 05:48:00 +0000

Two agents, one package.json, zero survivors. Run parallel Claude Code agents in the same directory long enough and you will watch one agent's half-written file get clobbered by another's, or an npm install from Agent A break Agent B's running build. Messaging between agents doesn't fix this. Physical separation does.

That's what worktrees do: every agent gets its own complete working directory, backed by the same git repo. Work in parallel, merge at the end, let git referee the conflicts. Here's how Claude Code wraps it and when the overhead pays off.

Git Worktree in 60 Seconds

Worktree is a native git feature, older than AI agents and underused by humans. Normally one repo = one working directory, and that directory can sit on exactly one branch at a time. Mid-feature and need to hotfix main? Stash, switch, fix, switch back. Annoying.

git worktree breaks the limit: spin up additional working directories from the same repo, each on its own branch. Shared .git history, independent files. A house with one bookshelf and several desks - every desk holds a different draft.

Three properties matter for agents:

Each worktree has its own physical files
All worktrees share one git history
Different worktrees can sit on different branches simultaneously

Why Agents Need This More Than You Do

A human does one thing at a time; one directory is fine. Parallel agents shred that assumption in three specific ways:

Failure	What happens
Write conflicts	Two agents write the same file. Last writer wins, or the file ends up interleaved garbage
Dirty reads	Agent B reads a file Agent A has half-written. Every downstream decision builds on a broken premise
Environment pollution	Agent A installs a package or edits config. Agent B's build explodes for reasons it can't see

Human teams solve this with conventions and chat. Agents execute too fast for that - the gap between "decides to write" and "wrote" is milliseconds. You cannot coordinate at that speed. You can only isolate.

How Claude Code Wraps It

Start a session with the flag:

claude --worktree

Claude Code then:

Creates a fresh git worktree from your repo
Points the session's file operations at that directory
Runs the whole session inside the isolated copy
On exit, handles merge or cleanup

You work normally. The only difference is where edits land - a private copy instead of your main checkout.

With SubAgents: sub-agents already get context isolation (own message history, scoped tools, summary-only returns). Worktrees add the fourth wall: file isolation. Let a sub-agent attempt a risky refactor in a worktree; if it faceplants, delete the worktree. Your main directory never knew anything happened. Cheapest undo button in the business.

With Agent Teams: each teammate gets its own worktree. Even when backend and frontend agents both touch package.json, they edit private copies. Git merges at the end - and merge conflicts at integration time are reviewable; runtime overwrites are just lost work.

Configuration

Lives under .claude/:

baseRef - where a new worktree starts from:

"fresh": latest remote main. For independent tasks.
"head": your current local commit. For building on uncommitted work in progress.

.worktreeinclude - the gotcha file.

List what must travel:

# .worktreeinclude
.env
.env.local
supabase/.temp

(node_modules is usually better reinstalled than copied - let the agent run the install.)

Lifecycle hooks - WorktreeCreate and WorktreeRemove events fire on creation/deletion. Wire hooks to auto-run npm install or seed a local database whenever a worktree spins up.

What It Solves, What It Doesn't

Solved:

Write conflicts - physically impossible across worktrees
Environment pollution - each agent installs and configures privately
Cheap rollback - bad experiment? git worktree remove, done
Confidence to parallelize - tasks you'd serialize out of fear now run together

Not solved:

Merge conflicts. Isolation defers conflict, doesn't delete it. Two agents rewriting the same function still collide at merge - but as a reviewable git conflict, not silent data loss.
Semantic conflicts. Agent A changes a function's return type; Agent B writes callers against the old one. Files merge clean, runtime breaks. Only inter-agent communication catches this - it's why worktrees pair with Agent Teams messaging instead of replacing it.
Disk. Every worktree is a near-full copy of working files (the .git database is shared). A 2GB repo with five worktrees is 10GB+. Mind the laptop.

Cleanup

Claude Code manages the lifecycle so you don't accumulate orphans:

Session ends with zero changes → worktree auto-deleted
Session ends with changes → you choose: merge to main, keep as a branch, or discard
Manual: list active worktrees, prune as needed

The Three-Layer Isolation Model

Worktrees complete a stack, each layer answering a different question:

Layer	Isolates	Question it answers
SubAgent	Context	"How do I keep exploration noise out of the main conversation?"
Agent Teams	Roles	"How do specialized agents coordinate?"
Worktree	Files	"How do parallel agents not trash each other's work?"

Compose them based on what the work needs:

Agents only read code → SubAgents alone. No write risk, no worktree needed.
Agents write, with clean file ownership → Agent Teams messaging covers it.
Agents write, ownership unclear or overlapping → add worktrees and stop worrying.

When to Skip It

One agent working - no conflict exists, pure overhead
Read-only workloads (reviews, codebase Q&A)
Multi-GB repos on small disks
Sub-10-minute tasks - worktree create/merge overhead eats the win

Try It This Week

Solo first: claude --worktree on a normal task with file edits. Feel the create → work → merge loop.
Risky refactor in a sandbox: point a sub-agent at your scariest "I've been meaning to restructure this" module, worktree on. Worst case costs one git worktree remove.
Full team: three teammates, three worktrees, one real feature. The merge step at the end is where you'll learn the most.
Tune: set baseRef, fill .worktreeinclude with whatever the agent complained about missing.

The mental shift is the actual product here. Without worktrees, you ration parallelism because you fear collisions. With them, "don't touch each other's files" stops being your problem and becomes git's. You stop planning around conflicts and just let the agents run.

Originally published on AI Builder Club, where the full free agent-engineering curriculum lives.

How to Evaluate AI Agents: What Works in 2026

Jason Zhou — Mon, 20 Jul 2026 05:47:00 +0000

Ask your agent "did you complete the task correctly?" and the answer is yes. It is always yes. Anthropic measured this while building long-running coding agents: models evaluating their own output skew systematically optimistic, and the fuzzier the criterion - design quality, completeness, "is this actually good" - the worse the skew. The agent isn't lying. It's grading homework it just wrote, with the same brain that wrote it.

Evaluation is the harness component builders skip most, and it's the exact line between "demos well" and "deployed." Here's the toolkit that works: separated evaluators, trace debugging, eval sets grown from real failures, and the judge pitfalls that silently corrupt your numbers.

Why Self-Evaluation Fails (It's Not the Model's Fault)

Three structural reasons, none fixable by prompting harder:

Confirmation bias, mechanized. The reasoning that produced the answer is sitting right there in context. Asked to verify, the model re-walks its own steps and finds them - surprise - reasonable. Same failure as a developer reviewing their own PR ten seconds after writing it.
Shared blind spots. Whatever misunderstanding produced the bug also evaluates the bug. If the agent misread the spec, its self-check applies the same misreading and passes.
"Looks done" vs "is done." Generation optimizes for plausible-looking output. Code that reads correct and code that runs correct are different claims - and only one of them can be checked by reading.

The fix follows from the diagnosis - and it's a rule older than AI: production and acceptance must be different parties.

Pattern 1: The Generator-Evaluator Split

The architecture Anthropic landed on for autonomous coding runs:

Generator builds the thing
Evaluator - a separate agent with fresh context - checks it. It never sees the generator's reasoning, so it can't inherit the blind spots
Failures route back as concrete feedback; the loop continues until the evaluator passes it

The detail that makes it work: the evaluator operates the output instead of reading it. It opens the app in a browser, clicks the flow, runs the test suite, checks the console - behavioral verification, not code review. "Looks done" can't survive contact with a clicked button that doesn't work.

You can run this pattern today at three budget levels: a sub-agent with a review prompt and read-only tools (cheapest), an Agent Teams reviewer teammate with browser access (the multi-perspective version), or a Stop hook that runs lint+tests before the agent is allowed to finish (deterministic floor - the agent literally can't hand over broken code).

Pattern 2: Traces, or Debugging the Middle

A 15-step agent run that produced a wrong answer is not one failure - it's one failure somewhere in 15 steps, and without traces you're guessing which. Outcome-only evaluation tells you that it failed; traces tell you where: the retrieval that fetched the wrong doc at step 3, the tool error silently swallowed at step 7, the goal drift after the context filled at step 12.

Minimum viable tracing: log every step's input, tool calls with arguments, results, and decision - structured (JSONL), replayable, and greppable. The audit-log hook gives you this for Claude Code in one config entry. For your own agents it's an afternoon of plumbing that pays back the first time a run goes sideways. Two metrics worth computing from traces beyond pass/fail: steps-to-completion (rising step counts = degrading efficiency, even while pass rate holds) and cost-per-success (the number that decides if the agent ships).

Pattern 3: Eval Sets Grown From Failures

Public benchmarks (SWE-bench, GAIA, τ²-bench) tell you which model to pick. They tell you nothing about whether your agent handles your tasks - your data shapes, your edge cases, your users' weird phrasings. For that you need your own eval set, and the cheapest way to build one is to never waste a failure:

Start embarrassingly small - 20 real tasks with verifiable expected outcomes
Every production failure becomes a new eval case (the agent equivalent of regression tests)
Run the set on every prompt change, model swap, or harness tweak
Track pass rate, steps, cost over time

This is eval-driven development, and it converts agent work from vibes ("feels better after the prompt change?") into engineering ("pass rate went 71% → 84%, cost per success down 12%"). LangChain's Terminal Bench climb - top 30 to top 5 without touching the model - was exactly this loop applied relentlessly to the harness.

Pattern 4: LLM-as-Judge, With Its Three Corruptions

Many criteria can't be asserted in code - "is this summary faithful," "is the tone right." An LLM judging outputs against a rubric scales where human review doesn't. It works, if you dodge three documented biases:

Bias	What happens	Mitigation
Position	Pairwise comparisons favor the first option	Judge both orders, average
Verbosity	Longer outputs score higher at equal quality	Length-cap or explicitly instruct against it
Self-preference	Models rate their own family's prose higher	Judge from a different provider than the generator

Two rules regardless: rubrics must be specific ("does it cite a source for every numeric claim - yes/no per claim", not "rate quality 1-10"), and a 5-10% human spot-check sample stays forever. The judge is a scaling tool, not an unsupervised authority - calibrate it against humans before trusting it, and re-calibrate when you change the rubric.

The Maturity Ladder

Where most builders are vs where production agents need to be:

Level 0 - vibes. Run it, eyeball it, ship it. Every demo you've seen.
Level 1 - deterministic gates. Tests + lint via hooks. An afternoon. Catches the embarrassing class.
Level 2 - separated evaluator. Fresh-context agent operates the output. Catches "looks done."
Level 3 - eval set + traces. Regression suite from real failures, every run debuggable. Changes become measurable.
Level 4 - continuous. Production sampling scored by calibrated judges, alerts on drift, cost-per-success on a dashboard.

Each level catches what the previous one structurally cannot. Most builders sit at 0; level 1 is one hook away; level 3 is where "we think it works" becomes "we know what changed." And the cultural shift underneath is the same one harness engineering keeps teaching: stop asking the model to be more trustworthy, and build the system that doesn't need to trust it. It's also the same lesson behind loop engineering - in any agent loop, the evaluator (the verifier) is the bottleneck, not the model.

Originally published on AI Builder Club, where the full free agent-engineering curriculum lives.

Claude Code Hooks: Complete Guide with 7 Recipes (2026)

Jason Zhou — Sun, 19 Jul 2026 06:47:00 +0000

You told Claude Code to run Prettier after every file edit. It did - twice. Then it forgot. You reminded it. It complied. Next session, forgot again. By the fourth reminder you are not pair programming anymore, you are babysitting.

The fix is not a better prompt. The fix is a hook: a rule wired into Claude Code's execution pipeline that fires every single time, no matter what the model thinks. One user ran a force-push blocker for nine months. It fired 8 times - every one was Claude deciding "cleaning up git history seems like a good idea."

This guide covers the full hooks system: lifecycle events, matchers, handler types, and the 7 hooks worth installing first.

What Are Claude Code Hooks?

A hook is an external command that runs automatically at a specific moment in the agent's lifecycle. File about to be written? Your hook fires first. Command finished? Your hook post-processes the result.

The key property: hooks are deterministic. Instructions in CLAUDE.md are suggestions the model usually follows. Hooks are pipeline machinery the model cannot skip. The difference matters most for the things you can never let slip:

Need	CLAUDE.md instruction	Hook
"Prefer English comments"	Fine - occasional misses are harmless	Overkill
"Format after every edit"	Misses 1 in 10 times	Fires 100%
"Never force-push to main"	One miss = disaster	The only safe option
"Log every bash command for audit"	Will drift over a long session	Guaranteed complete

Rule of thumb: if a miss costs you real money, real data, or real trust, it belongs in a hook, not a prompt.

The Lifecycle Events: Where Hooks Attach

Claude Code exposes events covering every key moment in a session. The six you'll actually use:

Event	When it fires	What it's for
`PreToolUse`	Before any tool call executes	The only event that can block an action
`PostToolUse`	After a tool call completes	Post-processing: format, lint, test
`Stop`	When Claude is about to finish responding	Quality gates: don't let it stop with failing tests
`Notification`	When Claude needs your attention	Forward to desktop/Slack so you can context-switch
`SessionStart`	New session begins	Inject env vars, check dependencies, start logging
`UserPromptSubmit`	You send a message	Attach extra context before the model sees it

There are more (SubagentStart/SubagentStop, PreCompact, WorktreeCreate/WorktreeRemove) but the six above cover 90% of real usage.

The Three Things a Hook Can Do

Every hook returns one of three decisions:

Allow - do nothing, let the pipeline continue. Pure logging hooks live here.
Block - stop the action. Only available in PreToolUse. Claude receives an error explaining the action was rejected by an external rule.
Inject - add information into the conversation. Example: your linter found 3 warnings after a file write, inject them so Claude sees and fixes them.

That's the whole API surface. Three verbs, composed across lifecycle events, cover almost every automation you'll want.

Matchers: Scoping When Hooks Fire

You rarely want a hook on every tool call. Matchers filter by tool name using regex:

{
  "matcher": "Bash"           // only shell commands
}
{
  "matcher": "Write|Edit"     // only file modifications
}
{
  "matcher": ""               // everything (audit logs)
}

Typical scoping: security rules match Bash, formatting matches Write|Edit, audit logging matches everything.

The Four Handler Types

Type	What runs	When to use
Command	A shell command or script. Receives event JSON via stdin, returns decision via stdout.	90% of hooks. Fast, deterministic, free.
HTTP	A request to a URL	Push events to audit systems or internal APIs
Prompt	A Claude model evaluates the event	Grey areas a regex can't judge ("is this code security-sensitive?"). Costs tokens.
Agent	A full agent instance investigates	Heavyweight checks needing multi-step reasoning. Rare.

Start with Command. Reach for Prompt only when the rule genuinely can't be expressed as a regex or script - and accept that you've traded determinism for judgment.

Where Configuration Lives

Hooks go in settings.json at three levels, all additive:

File	Scope	Use for
`~/.claude/settings.json`	All your projects	Security rules, audit logs, notifications
`.claude/settings.json`	This project, committed to git	Team-shared rules: this repo formats with Prettier
`.claude/settings.local.json`	This project, gitignored	Personal preferences, machine-specific paths

Multiple hooks on one event all run in order. Type /hooks inside Claude Code to see everything currently registered - useful when a hook misbehaves and you forgot where you defined it.

7 Hooks Worth Installing First

1. Auto-format after every edit

PostToolUse, matcher Write|Edit. Extract the file path from the event JSON, run your formatter on it:

{
  "hooks": {
    "PostToolUse": [
      {
        "matcher": "Write|Edit",
        "hooks": [
          {
            "type": "command",
            "command": "jq -r '.tool_input.file_path' | xargs npx prettier --write --ignore-unknown"
          }
        ]
      }
    ]
  }
}

You never mention formatting again. Neither does Claude.

2. Block force-push to main

PreToolUse, matcher Bash. Script checks the command string for push --force or push -f targeting main/master, returns block:

#!/bin/bash
cmd=$(jq -r '.tool_input.command')
if echo "$cmd" | grep -qE 'push\s+(--force|-f).*(main|master)'; then
  echo '{"decision": "block", "reason": "Force push to main is not allowed. Use a feature branch."}'
  exit 0
fi
echo '{"decision": "allow"}'

3. Protect secret files

PreToolUse, matcher Write|Edit. Block writes to .env, credentials.json, anything matching your secrets patterns. Even if you ask Claude to "update the .env", the hook says no - which is exactly the point. Secrets get edited by humans.

4. Quality gate on Stop

Stop event. Run your linter and test suite when Claude tries to finish. Errors found? Inject them back: "ESLint reports 2 errors in src/api/checkout.ts." Claude sees the failures and keeps working instead of handing you broken code. This single hook changes the default from "looks done" to "verified done."

5. Desktop notification when input needed

Notification event. On macOS:

osascript -e 'display notification "Claude Code needs your approval" with title "Claude Code"'

Stop staring at the terminal while long tasks run. Go do something else; the notification pulls you back.

6. Full audit log

PreToolUse or PostToolUse, no matcher. Append tool name, input, and timestamp to a JSONL file. After any session you have a complete record of what was read, written, and executed. Costs nothing until the day you really need it.

7. Supabase migration guard

PreToolUse, matcher Bash. If the command contains supabase db reset or a destructive migration against your production project ref, block it. Agents are enthusiastic about "fixing" databases. Production is not the place to learn this.

Hooks vs the Permission System

Claude Code already has permissions (allow/ask/deny per tool). Where's the line?

Permissions are a gate: an action passes or it doesn't. Hooks are programmable machinery: they can block, but also transform, inject context, trigger external systems, and post-process results.

Use permissions for coarse access control ("always ask before bash"). Use hooks for everything with logic in it ("block bash only when it touches main"). They stack: permissions filter first, hooks refine.

Practical Rules From the Field

Keep hooks dumb. Check a path, match a regex, read an exit code. A hook's value is certainty - a "smart" hook that calls an LLM gives that certainty back. Sometimes worth it, but know the trade.

Watch latency. A 3-second hook on PreToolUse means every tool call waits 3 seconds. Claude Code makes dozens of tool calls per task. Keep synchronous hooks under ~200ms; push slow work (remote logging) to async.

Test the blocker hooks. A formatting hook that silently fails costs you nothing. A force-push blocker that silently fails costs you a main branch. Trigger each blocking hook deliberately once and confirm it actually blocks.

The Bigger Picture

Hooks compose with everything else in the Claude Code stack: give sub-agents a test-runner hook, give Agent Teams a quality gate on task completion, auto-initialize environments when a worktree is created.

The model handles judgment. Hooks handle certainty. Agents are probabilistic - they do the right thing most of the time. Hooks exist for the things where "most of the time" is not good enough.

Start with one: the auto-formatter. Five minutes of setup, and you will never type "can you format that" again.

Originally published on AI Builder Club, where the full free agent-engineering curriculum lives.

The 6 Components of a Production Agent Harness

Jason Zhou — Sat, 18 Jul 2026 05:46:00 +0000

Same model, same API, same price - one team's agent acts like a seasoned employee, the other's like an intern with amnesia. The difference has a location, and it isn't the weights. LangChain's equation names it: Agent = Model + Harness, so Harness = everything that isn't the model. The labs set your model ceiling. The harness is entirely yours.

"Everything that isn't the model" is too vague to build from, so here's the working decomposition - six components, what each does, and the characteristic failure that appears when it's missing. The last column is the useful one: agents fail in patterns, and the pattern points at the gap.

#	Component	Question it answers	Missing → failure looks like
1	Context management	What does the model see?	Inconsistent quality, forgotten constraints
2	Tool system	What can it touch?	Hallucinated facts, wrong tool, no tool
3	Orchestration	What happens next?	Half-finished pieces, no coherent whole
4	State & memory	What persists?	Every session starts from zero
5	Evaluation & observability	Was it right?	Confident garbage, undebuggable runs
6	Constraints & recovery	What if it breaks?	One bad step kills the run

1. Context Management: What the Model Sees

The most counterintuitive truth in agent work: most "dumb model" complaints are information environment complaints. Same intelligence, curated inputs, transformed output.

Three sub-jobs. Boundary definition - role, goal, success criteria. "Write an article" and "write a technical explainer for builders who know APIs but not transformers, optimizing for clarity over completeness" are different tasks; only one of them is the task you meant. Selection and exclusion - the senior-engineer move of handing over three relevant docs instead of forwarding the whole wiki. Relevance in, noise out, because noise costs attention even when ignored. Structure - fixed rules, current task, run state, external evidence in stable layers, so the model never loses the constraint in the pile.

This component is deep enough to be its own discipline - the context engineering guide covers rot, poisoning, and the four management strategies.

2. Tool System: What It Can Touch

Without tools a model only predicts text. Tools are where prediction becomes action - and where three design questions hide:

Which tools? Scope to the job. A writing agent and a security-audit agent should have disjoint toolsets; the universal-toolbelt design is how you get the 46-tool collapse, where selection accuracy degrades just from menu length. When to call? Both failure directions are real: the agent that searches the web to answer "what's 2+2," and the agent that confidently invents your API's auth flow instead of reading the docs sitting in its toolset. Calibrating tool-reach is harness work - prompts, examples, and tool descriptions that draw the line. What happens to results? Ten raw search results dumped into context is pollution; the harness distills - extract the claims, keep the citations, drop the boilerplate - before the model reasons over it.

3. Orchestration: What Happens Next

The gap between "can do every step" and "does the whole job" is orchestration. An unorchestrated agent freestyles: does some things, in some order, stops at some point. The output is jagged - brilliant fragments, missing connective tissue, no verification pass.

Mature orchestration makes five things explicit: step decomposition (the task as a sequence, not a vibe), decision points (where the path branches and on what), intermediate artifacts (each step's output is the next step's input - so it gets named and stored), termination conditions (what "done" means, mechanically), and escalation rules (which failures invite a retry and which summon a human). The structure can be code (a LangGraph graph), convention (Claude Code's explore-plan-code-commit), or a task list with dependencies - what matters is that the agent walks a track rather than wandering a field.

4. State and Memory: What Persists

A stateless agent is Groundhog Day with API costs - re-explaining the project every morning, re-discovering Tuesday's conclusion on Thursday. Statefulness splits into three tiers, and conflating them is the classic mistake:

Run state - where am I in this task? Done, doing, blocked. (The Task system's territory.)
Session memory - what happened in this conversation that later steps need?
Long-term memory - what survives across sessions? Preferences, conventions, decisions. (Three memory types, write/maintain/retrieve lifecycle - its own discipline.)

Tier-mixing produces both failure flavors: agents that forget the project context (under-persistence) and agents whose memory is a junk drawer of expired facts steering current work (over-persistence, no hygiene).

5. Evaluation and Observability: Was It Right?

The harness layer most often skipped, and the one that separates demos from production. Two halves:

Evaluation - because models grade their own homework generously. Anthropic measured this directly: self-evaluation skews optimistic, especially on fuzzy criteria like design quality. Their fix - a separate evaluator agent with fresh context that actually operates the output (clicks the UI, runs the tests) rather than admiring the code - is the generator/evaluator pattern, and the production results that followed are that article's whole story. The principle is older than AI: production and acceptance must be different parties. When you run agents in a loop, this evaluator is the verifier that decides "good enough" - and it, not the model, is the bottleneck.

Observability - a 15-step run with 8 tool calls that produced a wrong answer is undebuggable without structured traces: which step drifted, what the tool returned, where reasoning forked wrong. Log every step's inputs, outputs, and decision; make runs replayable; alert on no-progress loops. Boring. Indispensable. The first thing you'll wish you had.

6. Constraints and Recovery: What If It Breaks

Demos run the happy path; production runs the other ones. Real runs hit expired tokens, malformed files, rate limits, and instructions the model creatively reinterprets. Three sub-layers hold the line:

Constraints - what's off-limits regardless of model opinion: permission boundaries, deterministic hooks on dangerous patterns, sandbox walls underneath everything. Hard rails, not suggestions.
Validation - pre-handoff checks: does the output answer the ask, match the format, pass the tests? Catches "confidently wrong" at the cheapest possible moment.
Recovery - the difference between a stumble and a dead run: classify the error, retry transients with backoff, reroute hard failures to an alternate path, drop back to the last good state when the branch is poisoned. (One nuance from the context guide: keep the evidence of failure in context while recovering - models that see their own error stop repeating it.)

Using This as a Gap Check

The six aren't a maturity ladder you climb in order - they're load-bearing walls, and your agent's weirdest behavior points at the weak one. Run the diagnostic backwards from symptoms: quality varies run-to-run with identical inputs → context. Makes things up despite having the right tool → tool calibration. Great pieces, incoherent whole → orchestration. Asks you things it knew yesterday → state. You discover failures only when users do → evaluation. One flaky API call kills forty minutes of work → recovery.

Most builders, honestly scored, are strong on tools, passable on context, and near-zero on evaluation and recovery - which is exactly why most agents demo beautifully and deploy badly. The labs keep shipping better engines. The six walls are the house, and the teams getting production results are the ones who built all six. And if you're unsure where these six walls end and the agent's loop begins - the goal, the verifier, the stop condition - the loop vs harness comparison draws that boundary precisely.

Originally published on AI Builder Club, where the full free agent-engineering curriculum lives.

Crabbox: Cloud Sandboxes for Parallel Coding Agents

Jason Zhou — Thu, 16 Jul 2026 14:00:00 +0000

Crabbox: Isolated Cloud Sandboxes for Parallel Coding Agents

When you run ten or fifteen coding agents in parallel, writing code stops being the bottleneck. Merging it becomes the bottleneck. Every agent produces a pull request, and every PR carries the risk of breaking the system, so the real work shifts from "make the agent write code" to "let the agent prove its code works before a human ever looks at it."

That is the problem Crabbox - the new project from OpenClaw creator Peter Steinberger - is built to solve. Here is the full walkthrough, why parallel testing breaks on your laptop, and the open-source codebase harness that productizes the fix.

The merge bottleneck is the new bottleneck

Once your team sets up loops - agents that get triggered on their own, pick up work, and ship it - you end up with a lot of sessions running at once. At almost any given time I have at least ten agent sessions going; Peter has posted screenshots of fifteen-plus in parallel. The result is a massive volume of PRs that simply was not possible before.

That volume is the problem. Each PR has to be reviewed, and each one might break production. So the constraint moves up the stack:

   OLD bottleneck            NEW bottleneck
 ┌────────────────┐        ┌────────────────────┐
 │ writing code   │  ───►  │ getting code MERGED │
 │ (agents solved │        │ verifying + trusting│
 │  this)         │        │ 15 parallel PRs     │
 └────────────────┘        └────────────────────┘

The fix is not "review faster." It is to give each agent the tools to verify its own work and attach evidence - a passing test, a screenshot, a screen recording - to the PR, so a human is reviewing proof, not guessing. That is the first job of a codebase harness: make the repo agent-ready so agents can run, test, and verify before they hand work back.

Why parallel testing breaks on your laptop

Giving one agent a browser tool and a dev server works fine. The setup falls apart when many agents run at once, because they share your machine:

One database, one schema. If a local Supabase or Postgres instance is shared, one agent trying a new schema migration can break every other session at once.
Hardcoded ports. Most repos are not set up to run multiple instances side by side - a port is pinned for a good reason, and two dev servers collide.
One Docker daemon, one OS. Parallel sessions share the same daemon and resources, so they step on each other.
Resource limits. A modern production repo is heavy. Running many full instances locally is slow and often impossible.

Git work trees solve the code isolation problem - each agent gets its own checkout to modify - but they do not solve the runtime problem. Agents still need a live dev server and database to actually test against, and that is what collides.

The fix: one isolated sandbox per work tree

The right architecture is to stop running every session on your local machine and instead give each agent its own isolated cloud environment - its own box, its own database, its own dev server - that cannot affect any other session.

            LOCAL (work trees: code isolation only)
   agent A ─┐   agent B ─┐   agent C ─┐
            │            │            │
            v            v            v
   ┌─────────────┐ ┌─────────────┐ ┌─────────────┐
   │ CLOUD BOX A │ │ CLOUD BOX B │ │ CLOUD BOX C │   runtime isolation
   │  own DB     │ │  own DB     │ │  own DB     │
   │  own server │ │  own server │ │  own server │
   └─────────────┘ └─────────────┘ └─────────────┘
       test           test            test
   (no shared schema, ports, or daemon - they can't break each other)

Building that pipeline by hand is real work: spin up a machine on spot, mount a disk, copy the code in, install dependencies, start the app, open a browser to test, then tear it all down afterward. My team built exactly this, and it unlocked a lot - but one rough edge remained: when an agent finds a bug mid-test, getting the local fix back into the cloud box is awkward. The normal commit-push-CI flow floods the repo with throwaway commits, and you do not want to rebuild the box every time.

What Crabbox does

Crabbox closes that last gap. It lets an agent warm up a cloud box, sync the uncommitted diff straight from the local work tree, and run tests against it in real time - so you can change a file locally and retest in seconds.

The workflow an agent runs:

Warm up a box - spin up an isolated cloud sandbox on demand.
Run any command in the cloud - the agent runs bash commands as if local, but they execute in the box. Each run first syncs the dirty diff from your local work tree (no commit needed - the folder just has to be git-initialized), then runs the command against the latest code.
Generate evidence - collect screenshots, record video of the run, and publish artifacts to storage (such as an S3 bucket) so they can be posted inline as PR comments.
Stop the box - tear it down and delete it when the task is done.

  agent finds a bug ─► fixes file LOCALLY ─► next `run` auto-syncs the diff
        ▲                                              │
        └──────────────── retest in the cloud ◄────────┘
   no extra commits, no box rebuild - always testing the latest version

The whole thing is configured with three files:

A Dockerfile that encapsulates everything your local machine has - Node, Docker, the Supabase CLI, any tool the app needs.
A crabbox.yml that defines the sandbox provider (the video uses Daytona for fast startup via prebuilt snapshots), the default work root, which folders to exclude from sync (heavy or unneeded ones - node_modules and build dirs are usually gitignored already), and environment variables to pass. Those vars are pushed to the box over an encrypted SSH connection, so the data plane stays relatively safe.
A setup.sh so the agent runs one script to install dependencies and bring the dev server up, instead of stepping through it each time.

Exact command names and flags live in the project and the video above - treat the steps here as the shape of the workflow, not copy-paste syntax.

The codebase harness that productizes this

You do not have to wire all of this together from scratch. We packaged the setup my team runs into an open-source Claude Code plugin: the AI Builder Club skills repo. It ships two flagship skill sets:

Codebase harness - make any repo agent-ready so agents can run, test, verify, and ship, including an isolated cloud box per agent so loops can ship code in parallel.
Loops - spin up compounding agent loops on a shared, file-based knowledge base.

Install it in Claude Code:

/plugin marketplace add AI-Builder-Club/skills
/plugin install skills@ai-builder-club

Then the two entry points:

/setup-codebase-harness - run it in the repo your agents work in, so they can run, test, and verify their own work (this is the harness Crabbox plugs into).
/new-loop - run it where your agent's memory should live; it bootstraps the shared knowledge base and scaffolds your first compounding loop.

If you want the full walkthrough of the concept - the four ingredients of a loop and how the harness fits - the loop engineering guide is the written companion, and AI Agent Reliability and Cost Control covers why agent-generated evidence is what makes parallel work safe to merge.

Go deeper:

AI Agent 101 Course - build agents that run, test, and verify their own work, then deploy them
Claude Code 101 Course - CLAUDE.md, hooks, subagents, and work trees, the foundation a codebase harness sits on

Want the step-by-step build alongside a team running this in production? Join AI Builder Club.

Start Here

Pick the repo your agents already work in and run /setup-codebase-harness from the AI Builder Club skills repo. Get one agent running its own tests and attaching evidence to a PR before you scale to ten. The merge bottleneck only gets worse with volume, so solve verification first.

For the full build alongside a team running this in production, join the AI Builder Club.

Join AI Builder Club

Originally published on AI Builder Club, where the full free agent-engineering curriculum lives.

Codebase Memory MCP: Give Your Coding Agent a Map (2026)

Jason Zhou — Thu, 16 Jul 2026 05:46:17 +0000

Codebase Memory MCP: Stop Making Your Coding Agent Read Your Repo as Flat Text

Ask a coding agent to change something in a big codebase and you know what happens. It greps, gets a wall of matches, opens 20 files one by one, and still misses half the places that might break.

Here is the thing though: your codebase is already a map. Every import is an edge. Every function call is an edge. Your agent throws that structure away and reads the whole thing as flat text.

Codebase Memory MCP fixes this by indexing your repo into a persistent code graph - functions, classes, call chains, routes, cross-file and cross-service links - that the agent queries instead of grepping. It is written in C, parses 158 languages via tree-sitter, indexes most repos in seconds, and answers structural queries in under a millisecond. When I ran the same architecture question with and without it on my own monorepo, token consumption dropped from ~38,000 to ~11,000. Over a two-question session: 64,000 vs 33,000. About half.

Prefer to watch? Here is the full walkthrough:

Why grep burns your context window

The default exploration loop every coding agent runs on a structural question looks like this:

Grep for the identifier. Get matches across dozens of files.
Read the files one by one. Each read dumps hundreds or thousands of lines into context.
Follow the imports. More reads.
Try to hold the reconstructed structure in the context window while doing the actual work.

Every step pays full price because grep returns text, not structure. The agent has to rebuild the relationships between symbols from scratch, on every task, by reading source. Claude Code has subagents to contain this (the search runs in a separate context), which helps with pollution but is slow, and the cost still gets paid somewhere.

This is context engineering's ugliest corner: in production agents, roughly 100 input tokens are consumed for every output token, and on large-repo tasks a huge share of those inputs is just re-discovering how the code fits together. The project's arXiv evaluation across 31 real-world repositories puts numbers on it: graph-based exploration answered structural questions with 10x fewer tokens and 2.1x fewer tool calls than file-by-file exploration, at 83% answer quality. The README's extreme benchmark: five structural queries cost ~3,400 tokens via the graph versus ~412,000 via grep-and-read.

What Codebase Memory MCP actually is

Codebase Memory MCP is an open-source (MIT) MCP server that extracts every function, class, method, route, and import from your code and builds a relationship graph out of them. Cross-file, cross-package, even cross-repo: if service A calls an HTTP route in service B, that edge is in the graph.

Two design decisions make it different from the pile of codebase-index tools you have already seen and forgotten:

1. No LLM in the indexing pipeline. Earlier projects ran a language-model pass to write a "knowledge map" of your repo. Those maps cost money to build, took minutes to hours, and went stale the moment the code changed. Codebase Memory MCP parses with tree-sitter grammars in pure C. Indexing is purely programmatic: an average repo indexes in seconds, and the Linux kernel - 28 million lines across 75,000 files - indexes in about 3 minutes. Rebuilding on change is cheap enough that the graph just stays current.

2. It meets the agent where it already works. More on the hook pattern below, because it is the most copyable idea in the project.

The agent gets a toolset that replaces the grep-read-repeat loop:

Tool	What the agent uses it for
`get_architecture`	Quick overview: languages, packages, routes, hotspots, clusters
`search_graph`	Locate the node for a symbol by name, label, or file pattern
`trace_path`	Follow the call chain: who calls this, what does it touch
`detect_changes`	Map a git diff to affected symbols, with risk classification
`query_graph`	Cypher-like queries, e.g. "callers of handle_order with no test coverage"
`get_code_snippet`	Pull just the source of one function by qualified name

That last pairing is the token win in miniature: instead of reading a 2,000-line file to see one function, the agent asks the graph where the function is and pulls only its body.

detect_changes is the sleeper feature for teams. Point it at a PR diff and it returns the blast radius - every symbol the change can reach - which turns agent code review from "read everything and hope" into a graph query.

The hook pattern: why this MCP works when others get ignored

Here is the failure mode that killed most code-search MCPs: the agent forgets to use them. You install a beautiful semantic search tool, and the agent... greps anyway. Claude Code and Codex are heavily optimized around their built-in search tools, and no system prompt reliably overrides that habit.

Codebase Memory MCP acknowledges this instead of fighting it. On Claude Code it registers a PreToolUse hook: when the agent calls its normal Grep or Glob tool, the hook intercepts the call, runs the graph lookup for matching symbols, and injects the structured context - call chains, relationships, definitions - alongside the ordinary grep results. The grep still runs. The agent just gets the map with the matches, whether or not it remembered the MCP exists.

If you are building MCP tools yourself, steal this. Do not rely on the model choosing your tool over its built-ins; use hooks to enrich the tools it already prefers. It is the difference between shipping a tool and shipping a behavior.

The real test: tracing a hidden lock through a monorepo

I indexed the codebase for Superdesign, our vibe-design platform. It has a backend function called createDesignDraftNode - the tool an agent calls to add a design to the infinite canvas. Because multiple agents can generate designs on the same canvas at once, everything flows through a queue guarded by a canvas lock.

The catch: the lock is invisible from where the work starts. createDesignDraftNode never calls it directly; the protection is delegated down through layers. Grep that file for "lock" and you get nothing. Any agent (or human) reading the entry point would conclude the protection does not exist.

So I asked two questions, in two sessions - one with the graph, one with MCP explicitly disabled:

Question	With Codebase Memory MCP	Without (default grep)
"Trace the createDesignDraftNode and canvas lock flow"	~11,000 tokens, full flow returned in seconds	~38,000 tokens
"What breaks if I change the lock?" (cumulative)	~33,000 tokens, all 13 call sites identified	~64,000 tokens

Same model, same repo, same questions. Half the tokens, and the graph version found every call site - which matters more than the cost, because the expensive failure is the new engineer (or agent) who decides the lock "slows things down," tweaks it, and ships a race condition. Ask "what will break if I change this?" and the graph answers with the actual dependency list instead of a best-effort sample.

Those are my numbers on one repo, measured from Claude Code's context inspector; your ratio will move with repo size and question type. The pattern held across every structural question I threw at it: the bigger and more cross-cutting the question, the wider the gap.

How to set it up (2 commands, ~10 minutes)

1. Install. One command on macOS/Linux:

curl -fsSL https://raw.githubusercontent.com/DeusData/codebase-memory-mcp/main/install.sh | bash

Want the graph visualization UI too:

curl -fsSL https://raw.githubusercontent.com/DeusData/codebase-memory-mcp/main/install.sh | bash -s -- --ui

The installer auto-detects your installed agents (Claude Code, Codex CLI, Gemini CLI, Zed, OpenCode, Antigravity, Aider, KiloCode, VS Code, OpenClaw, Kiro) and configures the MCP entries and hooks for each. Single static binary, zero runtime dependencies.

2. Index. Open your coding agent in the repo and tell it: "Help me use codebase memory MCP." It will run the indexing, and if your repo already has decent documentation it applies ignore filters on its own - mine correctly skipped the legacy folders. My semi-complex monorepo indexed in a few seconds.

3. Verify. Ask it to trace a function you know is load-bearing. You should see search_graph and trace_path calls come back with the full chain, no file-by-file reading.

4. (Optional) Look at your graph. With the UI install: codebase-memory-mcp --ui=true plus a port gives you a web view of the whole graph. Is it useful? Honestly not sure. It does look really cool.

Where this fits: the codebase harness

A code graph is one ingredient, not the whole meal. The repos where agents ship reliably have a full harness: a way to run the app locally in one command, end-to-end tests that gate PRs, isolated sandboxes for parallel work, and now a structural map of the code.

That is why we added Codebase Memory MCP to /setup-codebase-harness in the open-source AI Builder Club skills repo. Run the skill in your repo and it sets up the MCP and index alongside the rest: e2e test gates, the script toolkit that gets your local server running, and Crabbox for parallel agent testing in remote sandboxes. One command, and your codebase goes from flat text to something an agent can actually navigate, run, and verify against.

If you are optimizing agent costs more broadly, the graph attacks the input side; reducing Claude Code API costs covers the rest of the bill.

Start Here

Install Codebase Memory MCP in the repo where your agent does real work, index it, and ask one question you already know the answer to: "what breaks if I change X?" Compare what comes back against what you know. That single test tells you whether your agent has been navigating your codebase or guessing at it.

Then make it part of the harness: run /setup-codebase-harness from the AI Builder Club skills repo so every agent session starts with the map.

For the step-by-step build alongside a team running this in production, join the AI Builder Club.

Join AI Builder Club

Originally published on AI Builder Club, where the full free agent-engineering curriculum lives.

Loop Engineering Guide (2026)

Jason Zhou — Sun, 05 Jul 2026 11:41:52 +0000

Loop engineering is the discipline of designing the loop an agent runs inside - what it does between tool calls, when it checks its own work, and how it decides it's finished - instead of hand-writing each prompt. It's the 2026 successor to prompt engineering. The model writes the prompts now. The scarce skill is defining what "good" and "done" mean, and the part almost every explainer skips is this: in any loop, the verifier is the bottleneck, not the model.

A few mornings ago I typed one sentence into Claude Code and went to make coffee. By the time I got back, something like forty agents had spun up, written code, checked each other's work, thrown away the bad attempts, and left me three pull requests to review.

I didn't write forty prompts. I wrote one loop.

If you've felt the ground shift under "prompt engineering" lately, this is why. Here's the decode - not the hype version, the version you can use this afternoon.

AI Jason breaks down the same shift in his "Loop Engineer: Systemization and Artifacts" video - what the term actually means, the four core ingredients, and how his team designs loops that compound:

How Did We Get From Prompt Engineering to Loop Engineering?

It helps to see the line this sits on:

Year	Era	What you do	Your role
2024	Prompt engineering	Write a good prompt, get a good output	Operator
2025	Parallel agents	Stop babysitting one chat, run several at once	Manager
2026	Loop engineering	Build the loop that runs the agents for you	System designer

The people behind Claude Code have said the quiet part out loud: some mornings they aren't really writing the prompts anymore - another model is - and they're managing hundreds, even thousands, of agents at a time. That sounds like a flex until you see what it implies: if you're not in the loop pressing enter between every step, something else has to decide when the work is good enough. That something else is the whole game.

What Is a Loop, Exactly?

Strip the jargon and a loop is four moves on repeat:

discover → plan → execute → verify → (repeat until a condition is met)

You used to be the loop. You were the thing standing between the agent's steps, reading the output, catching the mistake, deciding what happens next, telling it to try again. Loop engineering is the discipline of stepping out of that inner cycle and up to designing the track the agent runs on.

Here's the simplest possible version. Give the agent a goal and a stopping condition, and let it run:

Goal: get the test suite passing.
Loop: run the tests, read the failures, fix the most likely cause, run again.
Stop when: all tests green, or you've tried 6 rounds (then summarize what's left).

That's a loop. And it genuinely works on a simple task. But point a loop at something open-ended - "improve this app," "make this page better," "research X and write it up" - and it either produces something great or it quietly turns into a very expensive slop machine. The difference between those two outcomes is the part nobody talks about.

Loops Nest: Andrew Ng's Three-Loop Model

The discover → plan → execute → verify cycle above is the inner loop. Zoom out and you'll find it wrapped inside slower loops. Andrew Ng laid this out cleanly in a June 2026 Batch letter - and the fact that he wrote it is itself the signal: "'Loop engineering' is a hot buzzphrase after mentions of it by Boris Cherny (Claude Code's creator) and Peter Steinberger (OpenClaw's creator) went viral." When Andrew Ng devotes a whole letter to a term, it has left buzzword territory.

His model is three loops running at different speeds:

Loop	Cadence	Who runs it	What it decides
Agentic coding loop	Seconds to minutes	The agent	Write code, test it, iterate until it's bug-free and meets the spec (and evals, if you have them)
Developer feedback loop	Tens of minutes to hours	You	Review the product, steer the agent, update the spec
External feedback loop	Hours to weeks	The world	Alpha testers, A/B tests, production data - feeds your vision back into the spec

  ┌──────────────────────────────────────────────────────┐
  │  EXTERNAL FEEDBACK LOOP            (hours → weeks)     │
  │  alpha testers · A/B tests · production data          │
  │   ┌────────────────────────────────────────────────┐ │
  │   │  DEVELOPER FEEDBACK LOOP     (minutes → hours)  │ │
  │   │  you review · steer · update the spec          │ │
  │   │   ┌──────────────────────────────────────────┐ │ │
  │   │   │  AGENTIC CODING LOOP    (seconds → mins) │ │ │
  │   │   │  generate → test → verify → repeat       │ │ │
  │   │   └──────────────────────────────────────────┘ │ │
  │   └────────────────────────────────────────────────┘ │
  │              vision ──► spec ──► agent                 │
  └──────────────────────────────────────────────────────┘

Watch what changes as you move outward: the loop gets slower, and the verifier gets more human. The inner loop can verify itself with tests. The outer loops can't - they need you. Ng makes the sharpest point right here, and it's a friendly amendment to the "taste is the reward function" idea coming up next. People call the human contribution "taste," he writes, but he prefers "context advantage": "So long as the human knows something the AI does not, human-in-the-loop is needed to inject that knowledge." Taste sounds innate and unteachable. "Context advantage" tells you exactly what to encode into the loop: the things you know about your users and your problem that the model doesn't. That's your verifier, written down.

And here's his tell for when to graduate from eyeballing output to a real verifier: "If you find that the system repeatedly runs into certain problems, building a set of evals for the agent becomes useful." Same lesson as the landing-page example below - the moment you're checking the same thing twice, write it into the loop.

Why Is the Verifier the Bottleneck, Not the Generator?

Every loop has two halves. The generator produces work - that's the model, and models are now extremely good. The verifier judges whether that work is good. Put it plainly: a loop is just a generator wired to a verifier, and the generator was never the bottleneck. The verifier is.

For two years we obsessed over the generator. We tuned prompts, swapped models, argued about temperature. But in a loop, the generator runs over and over for nearly free. The thing that decides whether all that motion produces value is the verifier.

And the freer you let the loop run, the more everything rides on the verifier. A loop with a weak "good enough?" check doesn't fail loudly. It succeeds at producing garbage, confidently, hundreds of times.

This is the same shift Addy Osmani keeps pointing at: the bottleneck moved from writing code to proving it works. Review, judgment, taste, knowing what "correct" looks like - that's now the most leveraged skill an engineer has. It's the same lesson agent evaluation teaches at the production layer: a model grading its own homework always gives itself an A. In a loop-engineering world, your taste isn't a soft skill anymore. It's the reward function.

Open Loop vs Closed Loop: Which Should You Build?

Once you accept that the verifier is the point, the engineering decision gets clear. Every loop sits somewhere between two poles.

	Open loop	Closed loop
What it is	Give a goal and loose conditions, let it explore a wide space	Pin success criteria in advance, evaluate every step, define an explicit stop
Upside	Genuinely novel output. Surprising solutions live here	Runs on a normal budget. Predictable. Safe to leave alone
Downside	Burns tokens. Degrades into slop fast with loose criteria	Won't surprise you. Does what you specified, not more
Lives or dies by	The verifier (even more so)	The verifier

The actual engineering is two decisions:

Choose open or closed for this specific task. Roughly: how much do I need novelty, and how much budget am I willing to risk?
Write the verifier that matches. A closed loop needs hard, checkable passes. An open loop needs an even better verifier, because it's the only thing standing between exploration and slop.

What Does a Loop With a Weak Verifier Actually Produce?

The most useful thing isn't a loop that works. It's the same task run two ways.

Here's Take 2 written out:

Goal: improve landing-page conversion clarity.
Done when ALL pass:
  - Lighthouse accessibility score >= 95
  - Exactly one primary CTA above the fold
  - Hero headline states the value prop in <12 words
  - No layout shift (CLS < 0.1)
Loop: propose a change -> run the checks -> keep it only if every check still passes
      -> stop when all green or after 5 rounds.

Now the loop converges, because every iteration clears a bar you defined. Then comes the trick that rescues open loops too: keep those hard checks as the floor, and add one open instruction ("surprise me with the headline"). Now you get exploration that can't degrade below your standard.

The lesson isn't "closed loops are better." It's: the verifier is what makes either kind ship.

What Tools Can Run the Loop For You?

You don't have to build the loop machinery from scratch. There's a clear lineage, from rigid to autonomous:

Tool / pattern	How it decides "done"	Best for
"Ralph" loop	A bare shell `while` loop re-fires the same prompt until you stop it - no smart "done," you are the stop button	Mechanical, repetitive tasks. Crude but bulletproof
Claude Code `/goal`	A small fast model judges the stop condition after every turn	Fuzzy "done" that still needs evaluating
Goal-tracking setups	Agent tracks its own progress in files, defines "done" up front	Long runs that need to stay oriented
Self-hosted agents (Hermes-style)	Runs continuously on its own infrastructure, keeps state across sessions	Always-on agents you set up once and leave running

/goal is the cleanest entry point: you type a completion condition once, and after each turn a Haiku-class evaluator reads the transcript and decides yes or no, looping until it holds (or you hit your turn cap). One catch worth knowing - the evaluator only sees what Claude already printed, so your condition has to be provable from the agent's own output.

Pick the least autonomous tool that does the job. Autonomy is not the goal. A shipped result is the goal. When the work fans out wide and repetitive, push the loop into a script with dynamic workflows instead of babysitting it.

What Does a Real Loop System Look Like?

A single loop is useful. A system of loops that share a brain is where the leverage compounds. This is what AI Jason runs inside his own company.

Start with the simplest one: a support loop. Every 30 minutes a cron wakes an agent. It pulls every support ticket, replies to the ones it can answer, and logs the frictions and product ideas it spots into a shared folder called signals. That loop alone earns its keep.

Now make it compound. The same loop doesn't just log a bug - it spawns a coding agent to fix it, monitors whether the fix held, and tells the customer it shipped. If people still hit the issue, that means it wasn't fixed at the root, so the loop tries again.

The real trick is that every loop reads and writes the same shared folders. Jason runs several at once:

Loop	Trigger	What it does	What it writes
Support	Every 30 min	Answer tickets, spot friction	`signals`, engineer tasks
SEO	Daily, 9am	Pull data, research topics, publish pages	pages, conversion-gap `signals`
Product growth	Daily	Prioritize experiments from analytics + signals	tasks
Reddit	Scheduled	Draft on-brand comments	comment artifacts

Because they share one file system, the SEO loop's "this keyword converts but we have no organic content" signal feeds the content loop. The support loop's repeated-bug signal gets picked up by the product loop. Each loop runs every hour or every day, reading what the others learned. The shared brain is what makes it compound - and Jason's quoted output from this setup is 20 to 40 high-quality pages a day driving traffic without him looking at it.

The 4 Ingredients of a Loop That Compounds

Jason's framework boils down to four parts. Most people nail the first three and skip the fourth - which is the one that actually decides whether autonomous work is possible.

Triggers. What wakes the agent. A cron job, a webhook, another agent, a server incident. The point is the agent runs without you pressing enter.
File structure. The most important design decision. Where artifacts, contracts, and logs live (covered below).
Tools and connectors. The skills and scripts that let the agent do real work - Intercom to fetch tickets, Stripe to check subscriptions, Supabase to debug, Playwright to test.
An agent-ready codebase. The setup that lets many agents work in parallel and verify their own output. This is the one everyone misses.

How Do You Make a Codebase Agent-Ready?

Before any loop works, the environment has to let an agent operate solo. Three properties:

Legible - the agent can find where to change what. Keep AGENTS.md / CLAUDE.md as a ~100-line index that points to deeper docs (OpenAI keeps theirs around 100 lines). Then bake rules into custom lints so the agent gets a warning automatically instead of you hoping it reads the right doc. Example: lint-fail any import from a legacy folder you don't want touched.

Executable - the agent starts work with the dev server already up, at near-zero token cost. Write a dev local script so it doesn't burn 3-5 minutes booting the app every run. Make the repo worktree-friendly so five parallel agents each spin up their own server without colliding. Add scripts that jump to a specific state to test scenarios fast.

Verifiable - give the agent tools to test and prove it worked. The Playwright CLI is the standout: it drives the browser and records a video clip you can attach to the GitHub PR, so review takes seconds. Back it with end-to-end tests on the flows you never want broken - sign-up, upgrade, core action.

One hard rule from the video, and it lines up exactly with the verifier point above: don't let an agent self-verify. It doesn't work well. Jason's PR skill always spawns a separate read-only verifier agent with the detailed spec. Generator and verifier stay different agents.

The File System: Artifacts, Contracts, Logs

This is ingredient #2, and it's the heart of the system. Three file types, three jobs:

Artifacts - the shared knowledge layer. The output of each loop's work. Types include docs, signals, tasks, tickets, even campaigns for an ads loop. Each artifact type gets its own folder with a README defining what goes in, what doesn't, the process for adding an item, and the schema. Each artifact file carries front-matter metadata, a body, and a timeline of every change.

A signal is the unit that makes loops compound. It captures a product idea, a friction, or a missed opportunity, links to its raw sources (a support ticket, a customer quote), and any loop can read or write it.

Contracts - one per loop, usually a README in the loop's folder. It states the goal, the workflow, the boundaries, the outstanding backlog, and a timeline. Every time the loop fires, it reads its contract first - goal, workflow, what happened last time - then takes the next best action.

Logs - a single global work-log file. Different from the timelines because your day mixes reviewing loop output with hands-on copilot work. Before an agent starts a big task it reads the last 5-10 entries; when it finishes it appends what it did. That's how cross-domain context survives between sessions.

The workflow to stand one up: run the loop manually once as a test, calibrate the workflow with the agent, then ask it to write the contract and register the trigger. Test run first, loop second.

Why Writing a Verifier Is Like Defining a Reward Function

If you've touched reinforcement learning, this clicks instantly. In RL you don't tell the agent every move - you define the reward, and the agent iterates toward it on positive and negative signal.

That's exactly what you're doing here. You are not training the model. You are defining the reward: the end goal, and what counts as good. Your domain knowledge - knowing what correct looks like in your problem - is the moat. The model is a commodity. The reward function is yours.

This is also why the field keeps drifting toward harness engineering: the leverage isn't in the phrasing anymore, it's in the system around the model - context, tools, state, and the evaluation loop that decides when to stop. Loop engineering is that same move, named from the loop's point of view.

And watch where the vocabulary goes next. Once you accept that the verifier is the point, the natural follow-up question is how do you write a good one - which is exactly the ground "evals" cover. Ng's own advice ("build a set of evals when the system keeps hitting the same problems") points straight at it, and the term already circulating for this next layer is eval engineering: formalizing the verifier into a versioned dataset you can measure against. If loop engineering is designing the loop, eval engineering is designing the bar the loop clears. Same lineage, one level deeper.

Which is why I keep saying it: writing the verifier is the new prompt engineering.

Is Loop Engineering Just a New Buzzword?

I'd be doing you a disservice not to flag the counter-take, because in the three weeks after the term went viral, the skeptic case grew real teeth. The strongest version has four distinct arguments, not one:

The vocabulary mockery. One widely-shared post maps the whole lexicon back to CS primitives - "a while loop becomes 'loop engineering'... unit tests become 'evals'" - and it's now a genre: a 1,800-comment Hacker News thread arguing agents are "just a while loop with an LLM call," and an entire site (extra-steps.dev) dedicated to the bit, both rounded up in PostHog's "WTF is loop engineering". Anyone who's written a CI pipeline or a Kubernetes reconciliation loop has been "designing loops" for years.
The autonomy check. The Register's June 24 take: the latest AI buzzword "still needs humans in the loop." The demos hide how much steering real work still takes.
The economic critique. Ed Zitron's version: the trend amounts to "celebrating and evangelizing autonomous token consumption" - spending the model vendors would very much like to stimulate. And he has a live exhibit: Uber reportedly capped engineers at $1,500/month for agent tooling after burning through its annual AI budget in four months. An unattended loop with a weak verifier doesn't fail loudly; it fails at token prices, all night.
The sampling-bias point. The subtlest one, from developer @_kboy_ on X: Claude Code solving "writing software" is real, but that doesn't "give anyone the evidence to start defining how everyone should develop software" - the vendors' data comes from people already using their product.

Both things are still true at once, and that's the honest read. The word is riding a wave and the shift under it is real. The tell is who's using it seriously: Boris Cherny (Claude Code's creator), Peter Steinberger (OpenClaw's creator), and Andrew Ng - who wrote an entire letter mapping out his three loops - are not chasing a hashtag. When the people building the tools and the person who taught half the industry machine learning independently converge on the same frame, the frame is load-bearing. (And the wave is still rising: three weeks in, "Loop Engineer" is already a module title in GenAI bootcamp curricula, and Jensen Huang was echoing "prompt engineering is dead" on stage.)

So here's your filter. Ignore the word. Look at whether the underlying shift is real:

Are you increasingly designing systems that run agents, instead of prompting agents directly? Yes.
Is the scarce, valuable part now defining "done" and "good," rather than phrasing the request? Yes.
Does a loop with a weak verifier reliably produce expensive garbage? Also yes - see Uber's invoice.

Here's the thing to notice: every serious criticism above attacks weak loops, not the discipline. Loops that still need un-budgeted human steering. Loops that burn tokens with no halt condition or spend cap. Loops graded on the vendor's own telemetry. The skeptics aren't refuting loop engineering - they're describing what happens when you skip the verifier, which is the entire argument of this guide. The label is optional. The shift is not. Whether you call it loop engineering or just "building agents that don't waste my money," the move is the same: stop perfecting prompts, start writing verifiers.

Your Loop Engineering Starting Checklist

If you build one loop this week, run it through this:

Define "done" in measurable terms before you write a single instruction.
Pin the passes - the tests, the rubric, the eval - up front, not after.
Choose open vs closed by need-for-novelty x budget-you'll-risk.
Always attach the run data to whatever you hand back to a human.
Use the least autonomous harness that gets the result.
Set the trigger - cron, webhook, or another agent - so it runs without you.
Give it shared folders - artifacts, a loop contract, a global log - so the next run, and every other loop, can build on this one.

Do that and you've crossed the line from someone who prompts AI to someone who engineers the system that does the work. That's the whole skill. The rest is reps.

Start Here

Pick one repetitive task this week. Before you write a single instruction, write down what "done" means in measurable terms. Then pin the checks, choose open or closed, and let the loop run against your bar instead of the model's.

For closed-loop templates, verifier checklists, and teardowns of loops that shipped (and loops that burned money), join the AI Builder Club - come ship something real.

Join AI Builder Club

Originally published on AI Builder Club, where the full free agent-engineering curriculum lives.