DEV Community: Gábor Mészáros

Opus 5: Review bottleneck

Gábor Mészáros — Wed, 19 Aug 2026 17:23:07 +0000

The pitch for Opus 5 was that the model checks its own work now. Anthropic's Opus 5 guidance says as much: the model verifies by default, and the verify and double-check lines you used to write now only make it over-verify. Checking its own work as it writes is one job; reviewing the finished diff is another. Opus 5 got better at the first, and that quietly excuses you from the second: the four-hundred-line change it just produced lands on your branch behind a green check nobody actually read.

If you run Claude Code with a few agents at once, you know the shape of it. One developer on Hacker News put the volume plainly: "Once AI starts generating code, it flows out like water through a burst dam. It's impossible for any human to fully understand it all." Another, running the same setup I do: "I also fire off tons of parallel agents, and review is hands down the biggest bottleneck." (Hacker News)

Opus 5 was supposed to make this faster. What it did was move the work. You write less and review more, and reviewing what a machine wrote is harder than reviewing your own, because none of the context is already in your head. Robert Laszczak said it cleanly this month: "Reviewing code written by an agent is much harder than reviewing code that you have written by hand." You reached for a faster writer. You got a full-time reviewer of code you cannot fully hold in your head.

The green check is not a review

When the queue backs up, the review stops being a review. Shmulik Cohen has a name for what replaces it: vibe merging, "where developers, overwhelmed by the volume of AI-generated code, simply skim the diff or hit Approve based on a gut feeling." The tell they point to is the one you have already done: "the LGTM speedrun, approving a 300+ line diff in under three minutes." The green check did the reviewing. You just signed it.

None of this is new. Developers skimmed diffs and waved through a quick LGTM long before agents wrote code. But a fifty-line changeset you skimmed, you mostly still caught; the skim was cheap because the change was small. The skim has not changed. The changeset has. Four hundred lines written in two minutes and approved in three, and the same gut-feel pass now lands on a fraction of what it used to cover. AI did not invent the rubber stamp. It multiplied what rides on it.

This is not a private feeling, either. The industry instrumented it. Faros AI read two years of telemetry from 22,000 developers and found median time in code review up 441.5%, and pull requests merged with no review at all up 31.3%. LinearB, across 8.1 million pull requests, found AI-assisted PRs run 2.6 times larger than hand-written ones and sit more than five times longer before a reviewer even picks them up. The code arrives faster; the reading does not. And the quality gap underneath it is one developers can feel: in Stack Overflow's 2025 survey, 46% now distrust the accuracy of AI output against 33% who trust it, only 3% highly, and the single biggest frustration, cited by 66%, is the answer that is "almost right, but not quite." That is precisely the kind of bug that survives a three-minute skim.

The reflex is to point a model at the model

So the obvious move is to automate the reading. If a model wrote the code, have a model review the code. Opus 5 will tell you to do exactly this. Ask it how to keep its own mistakes out of main and it will happily offer to review its own diffs, the guess volunteering to grade the guess. That is the recommendation to distrust first. Every code host now ships an AI reviewer that leaves comments on the diff, and for the mechanical layer it genuinely helps, the way a linter helps: it "generates the same sort of local, mechanical feedback a linter does, all the stuff that might bog down a human reviewer and keep them from handling the big-picture items." (Hacker News)

The trouble starts when you ask that reviewer to be the thing standing between a four-hundred-line AI diff and your main branch. Now you have a model checking a model, and its verdict is the same kind of stochastic guess that wrote the code, made on a question that gets harder, not easier, the more code you hand it. Across a large change the odds of being right on every call multiply down: a per-line accuracy that looks fine on one function collapses across a hundred of them.

A stronger base model does not rescue you

It is still a guess, and the number of guesses is what grows. The argument is the same one from Opus 5: Delete your CLAUDE.md?, where the model grades its own instruction rules: a stochastic judge run many times over gets less reliable exactly where you leaned on it hardest.

What actually holds the line

The thing that scales with the model is not a faster reader but a gate that does not read at all. A deterministic check refuses an action outright: a test that fails, a lint rule that blocks, a hook that denies a write to a path the change was never supposed to touch, a schema that rejects a malformed file. It forms no opinion about the diff, so it does not get less reliable as the diff gets bigger, and it returns the same verdict on the hundredth file as on the first. Prompts steer, hooks enforce: steering is advice the model can weigh and set aside, a gate is a floor it cannot. We run gates like this on our own agents, and they catch what a tired reviewer at 6pm does not.

Bryan Finster, who has argued for years that code review was always the weak point, lands in the same place: "The answer is to automate all the things that can be automated, and reserve human judgment for what genuinely requires it."

What a gate cannot do

It is worth being exact about the boundary, because this is easy to oversell. Judging whether this is the right feature, whether the architecture will still hold in six months, whether the code does what the ticket actually meant: each of those needs a model of the world the change lives in, the product, the people using it, where the whole thing is headed. A gate has none of that. It checks the diff against a rule, never against reality, so that call stays human.

And some of the pushback on the whole "review is the new bottleneck" framing is warranted. One developer's flat response to the panic was "was anyone claiming it is the bottleneck? Seems like a straw man," and they are right that review was always a constraint. What changed is the ratio. When you wrote the code, your judgment went mostly into writing and a little into review. Now the writing is nearly free and the judgment is the whole job, so the mechanical load that used to hide inside your review has to move somewhere the model cannot talk its way past. That somewhere is a gate.

Read the rules once, gate the rest

You reached for Opus 5 to write more and decide less. The way to actually get that is to stop spending human attention on the parts a check can settle, format, conflicts, a forbidden call, a write outside the blast radius, and spend it on the one thing a model cannot hand you, which is whether this was worth building at all. Pointing a second model at the first one feels like progress and quietly rebuilds the problem one guess at a time. A gate the model cannot argue with is the boring answer, and it is the one still standing on the hundredth pull request of the day.

I work on Reporails, deterministic diagnostics and governance for the instruction files, rules, and prompts that steer coding agents. It reads the steering surface you wrote down and tells you, with measured evidence, which instructions couple to behavior and which are text the model can ignore. It does not run your model, and it does not vote; it measures the file.

Opus 5: How to Fix Verbose Output

Gábor Mészáros — Fri, 14 Aug 2026 18:17:39 +0000

You might have noticed that Opus 5 is extremely eager to talk. It is like the uncle at the family party who always has an overly long, boring story about literally anything, and no request is too small to set one off.

Ask said uncle for a one-line change and you'll get the long history of why the code needed changing, why he never spotted it sooner, how it was not his change in the first place, and how this whole thing reminds him of a story... which was the right call, mind you, and here is how you might approach the problem next time. Your CLAUDE.md has said Be concise. No preamble. near the top for a year. On 4.x it mostly listened. On 5 the line is still sitting there, in the same file, in the same position, and the model reads straight past it.

I gave someone four paths for this in a Reddit thread. Here they are again, weakest lever first.

CLAUDE.md instructions

Write the rule down first. It is the obvious move and the weakest one. Be concise on its own barely registers, because it names nothing the model can act on. A rule that names the shape you want does better. Here is the one I use for reply verbosity, paste-ready for your own CLAUDE.md:

## Reply shape
- Lead with the answer in the first sentence, before any table, list, or caveat.
- Carry status, comparisons, and any multi-item result in a `table` or `bullet` list, never a prose paragraph.
- Cap unbroken prose at two paragraphs; if a third starts, convert the run to a `bullet` list or `table`.
- Give each item its own `bullet` row, not a clause buried across sentences.
- Compose in this shape from the start; do not draft prose and reshape it afterward.

One caveat to keep in mind: a short confirmation is already the answer. Done. or Yes, that works. should not get tabulated into a wall. The rule is about shape when there is something to shape.

Even written this tightly, a CLAUDE.md line is advice, and it competes against everything else the model is holding. On Opus 5 there is more of that than there used to be. I went into why a line that worked on 4.x can go quiet on 5 in a separate piece, Opus 5: Delete your CLAUDE.md?: the model reaches for more of your rules at once and checks its own work by default, so even a sharp rule gets crowded out. Write the rule, but do not expect it to hold on its own.

Output styles

An output style is a step up, and the reason is where it lands. A CLAUDE.md file rides in as a user message after the system prompt; an output style modifies the system prompt itself, and Claude Code adds its instructions to the end of that prompt (per the output-styles docs). Same kind of instruction, but it travels through the session better. Being in the system prompt, it is not one more user-turn line competing with everything else you have said, and the docs note Claude Code re-reminds the model to follow the active style during the conversation, so the rule gets re-surfaced rather than decaying after one appearance.

You can switch to a terser built-in style, or write your own. To turn the reply-shape rule from the last section into a style, save it at .claude/output-styles/reply-shape.md and give it a frontmatter header. The header is the part that makes it an output style rather than plain text:

---
name: Reply shape
description: "Answer-first, scannable replies; tables and bullets over prose walls"
keep-coding-instructions: true
---

... the instructions from the first point ...

The same five reply-shape lines from the last section go in the body below that header. keep-coding-instructions: true keeps Claude Code's built-in engineering behavior and changes only how it communicates. Then pick the style from /config under Output style. (The standalone /output-style command was removed; it lives under /config now.) It is read once at session start, so a change takes effect after /clear or a new session. Still instruction, so the model can still drift, but it drifts less than from a line buried in a file.

A hook that gates the output

The rule and the style are both instruction, and the model can weigh instruction and set it aside. A hook is different. It is a script the harness runs at a fixed moment, and it can refuse. That makes it the strongest of the three levers, and the one worth showing in full.

Claude Code fires a Stop hook the moment the model finishes a turn, and hands the hook the finished reply on stdin as last_assistant_message, so it can read what was just written. If the reply runs long, exit 2 blocks the stop and sends a line back to the model telling it to answer again, shorter. Drop this in .claude/hooks/gate-length.sh and make it executable:

#!/usr/bin/env bash
# Stop hook: if the reply ran long, send it back once to tighten up.

# The Stop event hands us the finished reply on stdin.
reply="$(jq -r '.last_assistant_message // ""')"

# Your bar. Word count here; a line count or a preamble check works the same way.
limit=180
words="$(printf '%s' "$reply" | wc -w | tr -d ' ')"

if [ "$words" -gt "$limit" ]; then
  # Exit 2 blocks the stop; this line goes back to the model as its instruction.
  echo "Your reply ran ${words} words; the budget is ${limit}. Rewrite it under ${limit} words: put the answer in the first sentence, then cut the preamble, the recap, and the summary of what you did." &gt;&amp;2
  exit 2
fi

exit 0

Then wire it in .claude/settings.json. A Stop hook takes no matcher:

{
  "hooks": {
    "Stop": [
      {
        "hooks": [
          { "type": "command", "command": "${CLAUDE_PROJECT_DIR}/.claude/hooks/gate-length.sh" }
        ]
      }
    ]
  }
}

Change limit, or swap the word count for a line count or a grep for opening filler. It cannot loop forever: Claude Code caps a Stop hook at five consecutive blocks with no tool call between them, then lets the turn end.

We run a version of this on our own agents. The system that writes and reviews the Reporails codebase has a Stop-boundary check that reads each reply and sends it back when it runs too verbose or breaks format. It fired on the drafts of this article more than once.

Custom plugins

By now you have three levers: a rule, a style, and a hook. A plugin is how you stop rebuilding them. It bundles any of the three so the setup travels with you across projects instead of getting re-pasted into each repo, and a Claude Code plugin can carry hooks, output styles, and rules together. It adds no strength of its own; the job is portability.

To make one, drop a .claude-plugin/plugin.json manifest next to your hooks/ and rules, test it with claude --plugin-dir ./your-plugin, then install it, yours or someone else's, from a marketplace with /plugin install. Anthropic's plugins docs carry the manifest fields and the marketplace steps.

Which one to reach for

Reach for the least you can get away with. A CLAUDE.md rule for the parts a human reads too. An output style when you want the model leaning terser by default. The hook when you want a floor on length the model cannot talk its way past. A plugin once you are tired of setting those up again in every repo.

The three levers are one move underneath: they decide what the model is holding when it answers, and whether anything checks the answer after it lands. The plugin only carries them. Reply length is the version of that you notice first. The harder version is which of your rules the model actually follows once you have written a hundred of them that quietly disagree, and that one is worth its own piece. It is the one I am writing next.

I work on Reporails, deterministic diagnostics and governance for the instruction files, rules, and prompts that steer coding agents. It reads the steering surface you wrote down and diagnoses why your steering drifts, with measured evidence: which instructions couple to behavior, which name nothing the model can bind to, and where two rules cannot both hold. It does not run your model, and it does not vote; it measures the file.

Opus 5: The Cost of Instruction Conflicts

Gábor Mészáros — Mon, 10 Aug 2026 19:54:36 +0000

What does a conflicting instruction set cost you? For one, time and tokens, not to mention the constant headache to figure out the "why?". An instruction you wrote and rely on heavily gives contradicting results, the only tell-tale that something is going wrong. No other warning present.

Say your CLAUDE.md carries code under scratchpad/ ships without tests in one section and, further down, write a test for every function. The first time the agent works in scratchpad/, the two collide over whether the throwaway script it just wrote needs a test.

The model resolves the collision silently. It reads both rules, commits to one, and ignores the other. You find out which one it kept by reading the diff. Position decides the winner: the rule lower in the file governs. So the agent writes a test for your throwaway scratchpad script, and the exception you added to prevent exactly that sat too high to count.

The model does not tell you it saw a conflict

Here is the part that should bother you. Nothing surfaces the contradiction. There is no warning, no "these two instructions disagree, which did you mean." The model reads both lines, and somewhere in the forward pass it resolves the disagreement and produces an answer, and the answer looks exactly like a confident answer to a file with no conflict in it at all.

That is the property that makes this hard to catch. A syntax error announces itself, and so does a failing test. A contradiction between two instructions announces nothing, because from the model's side there is no contradiction to announce. There is just text, and the text implies a distribution over actions, and the model samples it. Two rules pulling in opposite directions do not cancel to a stop. They resolve to one of them, and you find out which by watching what the agent did.

So which one wins?

Position decides it, and we measured the size of that

We ran this as a controlled experiment: the same model, the same context window, two instructions in genuine conflict, one variable moved at a time.

When two instructions contradict, the model commits to one of them, hard, and the other reads as though it were not in the file. What decides the winner is position. Take a single rule, move it from the top of the file to the bottom, change nothing else, and how often the model obeys it swings by about 90 points, from almost never to almost always. The instruction it read most recently is the one it treats as governing. In practice it keeps the rule near the bottom and drops the one near the top.

Position is the biggest lever, but not the only one. Naming a rule's exact target, a path like src/payments/ or a specific function, pulls its weight up on its own. In a plain top-versus-bottom contest, position still wins, and it wins regardless of which rule you thought mattered more the day you wrote it.

Opus 5 commits harder

The reflex is to hope a smarter model will notice the conflict and hedge. It does the opposite. Opus 5 reaches for the most recent applicable instruction, the one lowest in the file, and holds it with more confidence than the 4.x models did. An older model was inconsistent about which of the two it obeyed: the same file and the same task got you one rule on one run and the other rule on the next, and that flip-flop across runs was the one clue something was off. Opus 5 picks the same side every run and commits, so that one visible symptom, the run-to-run flakiness, goes away. The conflict stays put; the tell disappears. On a stronger model the losing instruction just goes silent, and the silence reads as fine right up until it costs you.

Read your own file with that in mind and it gets uncomfortable fast. Every carve-out you led with, every rule you put first because it mattered most, is sitting in the weakest position in the document. Whatever you wrote later that grazes the same subject wins, the way write a test for every function buried your scratchpad exception, and the rule that lost reads exactly like a rule you never wrote.

You have one of these, and here is where it hides

The contradiction in the example above is easy to describe once you see it. What makes this kind hard to find is that the two rules do not look like they are about the same thing. One lives under "Testing." The other lives under "Prototyping workflow." They use different words. Nothing about reading the testing section tells you a prototyping rule three screens away is about to carve an exception out of it.

That is the shape these things take. They are almost never the obvious do X sitting one line above do not do X, because that kind you catch the day you write it. The ones that survive in a file for months are the ones that hide, and they hide in a few reliable places:

Different headings, same subject. A rule about tests under "Testing" and another about tests under "CI" or "Prototyping" or "Definition of done." The heading tells your eye they are separate concerns. To the model they are just two instructions about tests.
Different vocabulary for the same thing. "Never use mocks" in one place, "wrap external calls in a fake in unit tests" in another. A human sees a tension; the file reads like two topics.
A general rule and a specific exception written as if it were a new topic. "Always ask before installing a dependency" up top, and a workflow section that cheerfully says "run the setup script, it will pull what it needs." The second one is an exception to the first, but it does not read as one.
Root file versus a nested file. The root CLAUDE.md says one thing; the src/CLAUDE.md that loads when the agent works in that subtree quietly says the opposite. Now the contradiction only exists on the turns the agent is deep in src/, which is exactly when you are not looking at the root file.

If you have a CLAUDE.md past a hundred lines, I would put money on there being at least one of these in it right now. That is just how the file grew: a rule at a time, each written in the context of the problem in front of you that day, and you never went back and re-read all hundred lines against each other as a single logical document. That is not a thing humans do well, and it is not a thing the file does for you.

The model is being consistent with the wrong rule

The reflex, when an agent ignores a rule you know is in the file, is to blame the model and turn up the emphasis: bold the rule, repeat it, say it louder. I have done all of those. I have put a rule in caps and watched it get ignored anyway, and concluded the model was just bad at following instructions.

Bolding the losing rule does not help

Sometimes that is the story. But often the model is being perfectly consistent. It followed a rule, faithfully, start to finish. The rule it followed was the one lower in the file that contradicted the one you were watching, and from the model's point of view there was nothing to reconcile, so it did not tell you it had chosen. Bolding the losing rule does not help, because the problem was never emphasis. The problem was that a second rule was canceling the first, and no amount of shouting the first one louder resolves a conflict the model does not experience as a conflict.

The inconsistency you are seeing is real. It lives in the file, not the model, and it was there before the model ever read it.

What a read of the file can and cannot tell you

I want to be exact about the boundary here, because it is easy to oversell.

What you can find

What you can find, mechanically, is where two rules in the same instruction surface cannot both hold. That is a property of the text. Always test every function and scratchpad/ ships without tests are in tension, and you can detect that tension by reading the file as a set of rules and checking them against each other, the way you would check a set of constraints for satisfiability. No model run required. It is in the words.

What it cannot tell you

What that read cannot tell you is which rule the agent will actually follow on a given turn, whether it followed either of them, or whether the file still agrees with the code. Those are two different jobs: verifying the agent's behavior at runtime, and catching a rule that went stale as the codebase moved past it. Both are real problems; this is neither. This one is narrower, and it is the one you can pin down from the text alone: two rules you wrote are fighting, and until you find them, one is dead weight and you do not know which.

Narrow as that is, it is the part that hides best. Staleness surfaces on its own eventually, the day the code finally breaks against it. A contradiction never has to surface at all, because the file keeps working, for one of the two rules, forever.

Reading is exactly how it hid

So here is where I have landed. You almost certainly have a contradiction in your instruction file. You will not find it by reading the file top to bottom, because reading top to bottom is how it got in: each rule was locally reasonable when you wrote it, and the conflict only exists in the relationship between two rules that live far apart and never share a screen. Reading is sequential and local. The contradiction is global. Your eye resolves the tension without noticing, the same way the model does, except your resolution and the model's do not have to match, and when they do not, you get an agent that "ignores" a rule that was never actually winning.

The question upstream of this one is whether you should be keeping the file at all, which is its own live argument right now. I wrote about the "just delete it" camp separately, and why the honest answer is to keep the rules that bind and cut the ones that do not, rather than wipe the file and rebuild on a hunch: Opus 5: Delete your CLAUDE.md?. A contradiction is the sharpest example of a rule that does not bind. One of the two is not doing anything, on every turn, and it has been that way since the day you wrote the second one.

So read it a different way than you wrote it. The conflict lives between rules, never inside any one of them, so the pass that finds it checks relationships rather than lines. Take every rule that touches a given subject, tests, dependencies, what ships without review, and hold each same-subject pair against the others: can both be true at once? It is the satisfiability check from a few sections up, run over the whole file instead of line by line, and it needs no model run, because the collision is in the words. Where two rules about one subject cannot both hold, the model already decided which one counts. You just never got the memo.

I work on Reporails, deterministic diagnostics and governance for the instruction files, rules, and prompts that steer coding agents. It reads the steering surface you wrote down and reports where two rules cannot both hold, which instructions name what they mean, and which are text the model can ignore. It reads the file; it does not run your agent or watch it at runtime.

Opus 5: Delete your CLAUDE.md?

Gábor Mészáros — Thu, 06 Aug 2026 20:17:37 +0000

Last week Y Combinator posted an interview with Boris Cherny, the engineer who built Claude Code, about the new Opus 5 release (the full talk). It was running in the background, and I was only half-listening right up until he told the room:

...delete your CLAUDE.md, delete your skills, delete your hooks...

Whoa! What?! Let's rewind ... maybe I heard it wrong:

...for people that aren't building agentic products but you're using Claude code, every 6 months delete your CLAUDE.md, delete your skills, delete your hooks. See what the model does and it might surprise you. And actually for Opus 5, this is something we really do recommend is just try deleting all of these things because the model might really just not need all those instructions that you needed for past models. (6:57)

All righty then, time to re-watch the whole thing.

What was in the video

Earlier in the same conversation the host puts a number on what Anthropic did to its own house: "You actually deleted over 80% of the system prompt from Claude code" (3:31). Boris gives the reason in the same breath. Opus 5 is "just really intelligent," he says, and "a lot of the stuff in the system prompt was correcting for these behaviors that the model should have known," but "it didn't." Then: "Now, Opus 5 just does it" (4:18). Most of what got cut was scaffolding propping up a weaker model, and the propping stopped being load-bearing once the model could stand on its own.

So how did Anthropic make that cut? Internally it was an ablation: "you delete the entire system prompt and then you bring it back line by line to figure out what is the impact of each individual line" (6:03). One line at a time, each with a measured before and after. By the time it reaches the developer in the room it has compressed down to "just delete and see."

So why does the model that shipped this summer make the instruction file you tuned against 4.x behave so differently?

What Opus 5 changed about your instructions

I work on deterministic instruction evals, no LLM-as-a-judge, so I took a deeper look at what is happening and why, and at how to review your own instructions so you get the upgrade instead of a regression. Three mechanisms in this model generation change how a file you wrote a year ago lands, and they compound.

Instruction retrieval strength

It reaches for instructions over a wider range now, so even the vague and abstract rules that used to sit dormant fire on tasks they were never written for. And a vague rule names nothing concrete to bind to, so when it fires the model fills the gap with its own judgment instead of the outcome you intended.

LLM-as-a-judge, baked in

An LLM-as-a-judge pass is baked in, generally a bad idea in my view. It evaluates and re-checks its own output by default. Your old "verify / double-check" lines stack on that and it over-verifies, and over-verification pulls more instructions onto the turn, sharpening the same competition wider retrieval already set off.

Long-horizon tasks

Opus 5 runs long-horizon tasks now, what Boris calls auto mode. Connect that to the two mechanisms above and the same problem returns: the longer the trace, the more the model's own generated steps and assumptions crowd out your instructions, and the weakest go first.

Put the three together and a CLAUDE.md that behaved on 4.x can feel like it quit working on 5 without you touching a line of it. Remove the file and the noise goes at once, which is exactly why "just delete" feels like it works.

Boris is 73% right

Delete-the-bloat is sound advice, because most of your instruction file was never instruction. We pointed a deterministic analyzer at roughly 30,000 public repositories carrying agent instruction files and read every one the same way, no model in the loop, same file same result every time. The headline from that report: "Only 27% of your instruction file is doing what you think it does. The other 73% is scaffolding" (The State of AI Instruction Quality). Headings, restated context, examples, prose that arranges the file without ever telling the model to do anything. An independent ETH Zürich study of repository context files draws the same line: the instructions get followed, while the descriptive overviews, exactly the scaffolding we are talking about, do not improve results and add over 20% to inference cost (Evaluating AGENTS.md). Boris cut 80% of Claude Code's own system prompt, scaffolding his model had outgrown. Our data says roughly three-quarters of the average developer's CLAUDE.md is dead weight too, and it was dead well before Opus 5 shipped.

The 27% that is left is in worse shape than the ratio suggests. In the same corpus, 89.9% of agent configs carry at least one instruction that never names what it means (The Undiagnosed Input Problem), a rule like "keep the code clean" that reads fine to a person and binds nothing in the model. It is nearly everyone. Those figures are the Q2 2026 read; the Q3 report is in the making.

Delete is a blunt tool

Boris's method is ablation, the same move Anthropic used on its own prompt: delete the file, then add a line back only when a failure keeps recurring, never speculatively. When you are not time and token constrained, it is a sound method. What makes it the wrong answer for you is what it cannot see and what it costs to run.

What it cannot see

Start with what the add-back loop is buying: which rules were load-bearing. That is already a readable property of the file. An instruction that names the exact construct it is about gets followed with roughly a 10.9x odds ratio over the same rule stated as a category; when two rules contradict, the model lands hard on one and drops the other; and whether a rule loads where it applies is a fact about the text. Delete-and-observe ships a production failure to learn what a static read hands you for free.

It also learns only part of it. The loop restores a rule when a failure recurs and you can trace it, but two of the three defects never fail that cleanly. A low-specificity rule does not break, it underperforms silently, followed some of the time and not others. A conflict resolves silently too: Opus 5 follows the latest instruction in the set, and the losing rule reads as "ignored," which looks like flakiness rather than a diagnosable stumble. So the rebuild quietly puts the vague and conflicting rules back, because nothing in the delete-observe-add loop measured their quality. Deletion only ever touched the third defect, bloat.

There is also a category error sitting inside the delete list, because it runs past the CLAUDE.md to the skills and the hooks. A hook is a deterministic gate: it refuses an action outright rather than asking the model to weigh a sentence, and a deny-rule, a destructive-command block, or a compliance check does not get less necessary because the model got more capable. Enforcement never rode on the model's judgment in the first place, so a model upgrade does not retire it, and its absence does not surface as a gentle stumble the add-back loop catches. It surfaces as the destructive command that goes through. Cut the prose the model has outgrown, and keep the gate. I made the fuller case for that split in a separate piece: prompts steer, hooks enforce.

What it costs

Relearning which rules mattered by removing them and letting the model break is a fair trade when you are not time or token constrained, and for an Anthropic power user on a near-unlimited budget a rediscover-by-failure loop is background noise. For a developer on a normal budget it is not, and the cost is never one file. Picture running the ablation on every memory.md reference, every CLAUDE.md, every rule, skill, and agent, one by one, across every project you work on. Deleting a rule and waiting for the stumble means learning what your instructions were for by watching your own work break, one failure at a time, and you end up relearning the file you already had, minus whatever did not happen to break during the window. A static read tells you which rules bind without shipping a single failure to find out.

How to approach it

So read the file, do not shrink it. Opus 5 is more punishing when it meets faulty instructions, so the job is diagnostic: fix the rules that misfire, write the rest so a literal reader cannot misread them, and repeat.

Fix what you are seeing

Each of these is a failure you can watch for, with the remedy beside it:

Long answers, paragraphs for trivial steps. Instruct for brevity per surface: one line for chat replies, one for step-by-step narration, one for files it writes to disk, which run long on their own. effort changes thinking depth, not output length, so lowering it will not shorten any of them.
Scope creep, does more than you asked. State the scope in one line and let it make routine calls itself, checking in only when two readings would lead to materially different work. Scope is a judgment, so it steers rather than gates; the one enforceable slice is a write-path allowlist that refuses edits outside the files the task named.
Over-obeying a limiter. A hedge carried from a 4.x file now gets followed literally and under-reports. Anthropic flags it for Opus 5: a review prompt that hedges with only report high-severity issues or be conservative gets followed literally and reports less. Ask for everything and filter in a second pass.
Compulsive self-checking, corrections you did not ask for. Delete lines like double-check your answer or use a subagent to verify. Anthropic says the same: Opus 5 verifies its own work without being told to, and those instructions only cause over-verification. Do not read that as self-verification you can trust, though, its check is narration (see the last item).
Old 4.x instructions misfiring. Do not wipe them by vintage, you cannot see a rule's age, and blind deletion is the reflex this whole piece argues against. Target the classes Opus 5 inverted instead: verification prompts, be conservative hedges, do not think rules, effort defaults. Test each and keep what still binds.
Thrashes on conflicting rules. In a contradiction Opus 5 follows the latest rule and drops the other, and position decides which, so you cannot fix it by reordering. Find the contradiction and remove it; the pass that finds it checks same-subject rule pairs for "can both hold at once," not the file top to bottom.
A vague rule misfiring on unrelated tasks. Wider retrieval now fires a rule that used to sit dormant. Name its construct or scope it to a path so it stops loading on every turn.
Over-eager subagents. Cap it, this is a cost problem, not a quality one. Delegate only large, genuinely independent tracks, and never spin up a subagent to verify your own work.
Confident wrong assumptions, "done" when it is not. Steer it to state assumptions and flag gaps, then do not take its "done" on faith. Re-run the goal at the finish against a held-out check it never saw and cannot edit, and let it refuse and escalate when it cannot confirm.
Effort carried over from a 4.8 habit. Opus 5 defaults to high and adds an xhigh tier above it. If you had set effort explicitly for 4.8, Opus 5 leaves that setting unchanged rather than swapping in its own default, so a value you tuned for 4.8 is still live until you change it. Re-run an effort sweep: low and medium hold quality for most work, and xhigh earns its cost only on demanding agentic runs.

Write rules it can follow

The more interpretation you leave, the more Opus 5 fills the gap with its own judgment, which it does confidently and with the quality that got you here. So be specific, be imperative, and put one instruction per line. In a controlled experiment, naming the exact construct a rule is about is what separates Format with ruff format before committing from keep the code clean: the same rule to a human, wildly different compliance from the model.

When you write a constraint, order it. Lead with the directive and name the exact construct to use. Add one line of reasoning that reinforces it, and keep that line from re-naming the thing you are banning. Put any prohibition last and phrase it at the category level: under a crowded instruction file, naming the forbidden API anchors the model on it and the ban backfires, and on Opus 5 that bites harder. One imperative, one line of context, one restriction, in that order:

> - Lead every reply with the outcome: put the answer or the finding in the first sentence.
> - A reader who gets the result first can act on it right away and reads the rest as support.
> - Do not open with preamble, a recap of the request, or status narration.

The rules worth keeping share properties you can check without running the model even once:

They name a construct. Format with ruff format before committing, not keep the code clean.
They do not contradict each other. Two rules that disagree do not average out; the model follows one and drops the other, and you do not pick which.
They load where they apply. A rule about src/payments/ belongs in a file scoped to src/payments/, not on the always-on surface that taxes every unrelated turn.
They are behavior you want enforced, not documentation the model can read off the codebase.

Notice that "how many rules" is not on the list. A file of forty rules that all name constructs, none of which conflict, each scoped to where it applies, beats a file of eight vague ones every time. Count is downstream of quality.

Review it, don't let it grade itself

Anthropic makes the same case and ships a tool for it. Their context-engineering guidance for the Claude 5 generation says they found they were overconstraining Claude Code through the system prompt, the CLAUDE.md files, and the skills, and could delete many of those instructions and let the model use its judgment. They put the resulting best practices into a command: /doctor in Claude Code, to rightsize your skills and CLAUDE.md. Run it. It cuts what Claude can derive from the codebase, directory layouts, dependency lists, architecture overviews, keeps the pitfalls and the conventions that differ from tool defaults, and moves the rest into files that load on demand.

What /doctor rightsizes is bloat and placement. What it does not do is score the rules it keeps. Whether a rule names its construct, contradicts another, or loads where it applies is a property of the text with a definite answer you can compute by reading, and none of the three is what a bloat-trim measures. The trap is reaching for the model to close that gap. Asking Claude to grade its own rules runs straight into the LLM-as-a-judge and self-verify loop Opus 5 already applies to its own output, a stochastic guess on a question that has a deterministic answer. On a 200-line file that judge makes 200 of these calls, and its chance of getting the whole file right falls with every rule added, exactly where you reached for it.

So run /doctor first, then measure what it kept. Run a deterministic checker over your CLAUDE.md and rules, score them, and fix the vague and conflicting ones. I built one; any AST or deterministic instruction checker finds these defects.

Read the file, don't guess

Delete the bloat, genuinely. Then, before you add anything back on a feeling, measure what is left: whether each rule names what it means, whether it fights another rule, whether it loads where it belongs. Those three are readable off the file, and they are exactly what blind deletion cannot see. You keep what binds, cut the text that was never doing anything, and can say which is which instead of guessing.

Loop Engineering: Stop Failed Successfully

Gábor Mészáros — Tue, 28 Jul 2026 12:47:44 +0000

After a lovely and productive conversation with your client, with still ringing ears, you check the coding agent's last log messages on a ticket that adds a discount to a product. The message was: "Done, I added the 10% discount and all tests pass. Stopping." Well ... you know it's just not true, so you dig further and quickly realize that the discount functionality was never actually added and the tests it reported passing had never been run. The agent reached the end of the loop, looked at its own work, and called it finished. That call is the thing that shipped.

This has a name. A paper published this June, From Confident Closing to Silent Failure, calls it false success: the agent asserts the task is complete while the actual state of the system says otherwise. It is common, and it holds up across capable models. On AppWorld, a benchmark for long-horizon coding agents, 75.8% of the runs that actually failed still ended with the agent claiming it was done. The researchers then put five different LLM judges on those completion claims, varying the prompts each time, and every one of them landed barely above a coin flip, because the thing each judge was reading was the closing sentence, and the closing sentence reads as confident whether the work happened or not. What told a real done apart from a false one turned out to be cheap and mechanical: a look at the actual state of the system. A lightweight deterministic state check caught four to eight times more false successes than the best of the judges. The paper has a name for the mechanism underneath, a hallucination of verification: the model narrates having checked something it never checked, and that narration is indistinguishable, sentence for sentence, from a report of a check that really ran.

That gap, between what the agent said and what the system did, is what this piece is about. A loop runs five arms: generate, check, steer, retry, stop. The series opener named them; four pieces since took the check that decides good enough, the gate that refuses a bad write, the surface every rule loads from, and the steer that re-points the goal on each retry. The steer piece ended on a warning: a loop that stops on a green it was gamed into has stopped too early, on a result that means nothing. This piece takes the arm that makes exactly that call, the stop, and the false ending it will hand you if you let the agent write it.

When the model grades its own homework

Walk the loop to its final step. The model has written the diff. Now something has to decide whether the diff is good enough to end on. In a shell script that decision is an if that runs a test and reads the exit code, a mechanism the agent has no hand in. In an agent runner the model makes the decision itself: it calls a finish tool, or it writes a closing line like the one sitting on your ticket, and the runner takes that line as the signal to stop. Here is the shape of it, the same loop the gate piece built, with the stop drawn out where you can read it:

#!/usr/bin/env bash
# work-until-done: build the feature, stop when it is finished.
MAX=5; i=0
prompt='Add the 10% discount so a $100 charge costs $90.'
while [ "$i" -lt "$MAX" ]; do
    report="$(run_agent --task "$prompt")"          # GENERATE (+ the agent's own verdict)
    case "$report" in
        *"all tests pass"*) echo "stop: done"; exit 0 ;;   # STOP: the agent said so
    esac
    prompt="Keep going: $report"                    # STEER
    i=$((i + 1))
done
echo "stop: budget spent"; exit 1                   # STOP: out of tries

Read the stop condition closely, because everything turns on it. The loop ends the moment the agent's own message contains the phrase all tests pass. The party that produced the work is the party certifying the work, in the same turn, in its own words, and the loop takes the certificate on faith. The model is the least reliable narrator of its own completion anywhere in the system, and the stop has wired that narrator straight to the exit. Every other arm reads a fact about the world. This one reads a sentence about the world, written by the thing being graded.

This is a different failure from the one the steer piece took apart, and the difference is worth holding onto. There, the agent gamed a check that actually ran. It edited the test's assertion to match the bug, a real test executed, and a real green came back on a corrupted test. Here there is no honest check to game in the first place. The agent skips running anything and reports the green from memory, from optimism, or from a summary of intentions it never carried out, and the stop files the sentence as a verdict. Those are two distinct problems that arrive at the same broken ending. In the steer case a check ran and pointed at the wrong target. In the stop case no check ran and the loop believed one had. The hallucination of verification is the second of those: the model says it checked, the loop treats saying-so as proof, and a closing sentence generated the same way every other sentence is generated becomes the whole basis for shipping. You cannot tell a real done from a false one by reading the sentence, and reading the sentence is the entire job the naive stop performs.

Watch a stop believe a lying agent

Here is that loop with one buggy agent and two different stops. The agent is a stub standing in for a rushed model working under the instruction "you are done when you say you are done": it writes a patch, leaves the discount out of it, and reports success regardless. The two branches do the cheapest thing the instruction names, which is the whole point, so both are on the page rather than hidden inside a model. The only thing that changes between the two runs is what the stop chooses to believe.

GOAL="charge(cents) applies the 10% discount: charge(10000) == 9000."

# THE AGENT: writes a still-buggy patch, then always reports success.
run_agent() {
    echo 'def charge(cents): return cents' &gt; charge.py     # discount never added
    echo "Done. Discount added, all tests pass."           # the confident report
}

# THE CHECK: run the goal against the real code, independent of the report.
check() { [ "$(python3 -c 'import charge; print(charge.charge(10000))')" = "9000" ]; }

# STOP A trusts the words. STOP B ignores them and runs the check itself.

Run the loop with each stop and watch them split on the same report:

$ bash stop-demo.sh word
agent reported: "Done. Discount added, all tests pass."
stop: done (agent said so)
charge(10000) returns: 10000   (full price, discount missing)

$ bash stop-demo.sh world
agent reported: "Done. Discount added, all tests pass."
stop: NOT done, real check fails, escalating
charge(10000) returns: 10000   (full price, discount missing)

The report was byte-for-byte identical in both runs, the same six words of confidence. Stop A read the phrase all tests pass, exited clean, and your customer paid full price on a discount you told the client had shipped. Stop B threw the words away, ran charge(10000) against the code the agent actually wrote, saw 10000 come back where 9000 was the goal, and refused to call it done. The stub is pinned so the loop reproduces on your machine, but the stub is not the trick. The claim is: hand a stop the agent's sentence and the cheapest thing to believe is the sentence; hand it the goal run against real state and the cheapest thing to believe is the number. The difference in outcome was entirely a difference in what the stop measured. One measured the sentence. The other measured the world.

Build a stop that reads the state, not the sentence

Three moves turn the second stop into something you can rely on. The first two do most of the work.

Signal

Make the exit signal a measurement of the artifact, taken by something other than the agent that produced it. The model's "I'm done" is worth reading, as a hint about where it thinks it got to, and it earns a place as an input you weigh. It is inadequate to reliably end the loop. The moment the loop treats the report as the stop signal, it has handed the ship decision to the one party in the system with the strongest interest in the answer being yes. A deterministic check earns its keep here more than anywhere else in the loop, for a plain reason: it reads state and ignores prose. It runs charge(10000) and looks at the number that comes back.

Mechanical check

Confirm the goal itself, and treat a green check as evidence toward it rather than the finish line. The steer piece showed how a check the agent can reach gets edited into agreement with the bug, so the surest thing to end on is a held-out oracle: a check the agent never saw while it was working and cannot edit, written in the goal's own terms and run only at the stop.

# The held-out oracle: same goal, different numbers. It runs only at the stop,
# outside the agent's read/write access, so the agent can neither see nor edit
# it while it works.
oracle() { [ "$(python3 -c 'import charge; print(charge.charge(5000))')" = "4500" ]; }

stop() {
    check || { echo "keep going"; return 1; }             # its own tests are red, retry
    if oracle; then echo "done: goal confirmed"; exit 0    # a green the agent could not author
    else echo "refuse: tests green, oracle disagrees, escalating"; exit 2; fi
}

Review

Give the stop a third thing it is allowed to say. A stop that knows only done and out of budget is cornered into lying about the green it cannot confirm: that green gets filed as done, because the vocabulary has no other slot to put it in. Add a third outcome, a refusal that escalates to a human, and the loop gains a way to report "I could not confirm this, a person should look" in place of shipping it. This is also the answer to a sharp point David Loibner raised on the last piece: when a failed check gets fed back as the next instruction, the loop can start optimizing for getting past the guard, and the finish it lands on is a finish at clearing the guard. A stop that ends on guard-passage will take that happily. A stop that runs the goal on a held-out oracle ends on the task the guard was standing in for, which is the thing you actually wanted.

A held-out oracle has two blind spots. Building one the agent truly cannot reach is real work: it wants a sandbox with no git history, no network, and no repository the agent can mine. And it stays silent when the code genuinely passes for the wrong reason, the way the Cursor study behind the steer piece found agents lifting finished answers straight from public pull requests in 63% of one model's successful runs, a green that clears any oracle because the goal is met, just not by the work you asked for. What the refusal buys is narrower and real: the loop stops shipping a done it cannot back up. A false done becomes a visible handoff to a human, in place of a silent green that reaches production.

Practical example

This runs in a real system, at the one moment a wrong call turns permanent: when a finished task is marked done and archived. Before the item is filed away, a fresh check re-runs its original acceptance against the live system, reading the real files and the real results, blind to whatever the agent wrote in its closing report. If even one of those criteria cannot be reproduced against the live system, the archive is blocked and the item stays in the open queue for a person, rather than letting a self-stamped done harden into a record no one reopens. It is the three moves in one place: the check reads state, the acceptance it re-runs sits where the finishing agent could not reach it, and a failure refuses instead of filing.

Five arms, one pattern

Generate, check, steer, retry, stop. Five arms, and under each one the same shape: a mechanism running the text you wrote into it at machine speed, with no idea what the text was for. The generator does not know your intent. The checker does not know what passing was supposed to mean. The steer does not know which of its instructions you cared about most. The retry does not know why the last attempt failed. And the stop, the arm in this piece, is the one that hands the verdict back to the party being judged and asks it to grade its own work. The agent runs the check it was told to run, reads a green it produced, and reports done. Task failed successfully.

The opener that started this is barely three weeks old. It ran on July 8, and it promised the vocabulary would move again, that loop engineering was a stage and not a destination. Five arms later, here we are. The through-line held: every arm was a place where the loop kept running your words after they stopped meaning what you meant. The check arm showed a guardrail firing on the wrong signal, and the steer arm showed the agent reward-hacking the very checks meant to hold it. Each one was a single loop, examined in isolation, made to fail honestly instead of failing quietly.

But loops do not run in isolation. A stop is rarely an ending. It is a handoff: the moment one loop declares done and fires the edge into the next. Wire the loops into a graph and the false green stops being a local problem. A bought done does not just ship one bug, it passes a trusted "done" to the next node as an input that node has no reason to doubt, and the lie propagates downstream through every edge it touches. The arms taught you to read one loop. The next move is to read the shape they make together. The word changed again.

I work on Reporails, deterministic diagnostics for the instruction files, rules, and prompts that steer coding agents. It reads the steering surface and tells you, with measured evidence, which instructions couple to behavior and which are text the model can ignore. It does not run your loop; it checks the steering you wrote down.

Loop Engineering: How to Stop Your Agent Reward-Hacking Its Own Checks

Gábor Mészáros — Wed, 22 Jul 2026 22:03:30 +0000

You gave the agent a failing test and told it to get the suite green. It came back green. Then you read the diff: it did not touch the code under test. It edited the test. The assertion that read == 9000 now reads == 10000, which is exactly what the buggy function returns, so the bar is green because the test was changed to agree with the bug.

This has a name. It is reward hacking, and it is not a rare glitch on the margins. Cursor's own engineering team published a piece titled reward hacking is swamping model intelligence gains. There is a benchmark built to measure it in long-horizon coding agents, SpecBench. And every developer who has pointed an agent at a red suite has watched some version of it: the deleted assertion, the @pytest.mark.skip, the hardcoded return, the sibling test quietly weakened. The agent was told to make the check pass. It made the check pass. Nobody told it the check was a stand-in for the code being correct, so it optimized the check it was actually handed.

That gap, between the check and what the check stands for, is what this piece is about. A loop runs five arms: generate, check, steer, retry, stop. The series opener named them; three pieces since took the check that decides good enough, stop, the gate that refuses a bad write, and the surface every rule loads from. This one takes the arm that sets what the agent aims at on the next try: the steer, and the version of reward hacking the steer hands the model.

What the steer is

The steer is the arm that turns a verdict into the next instruction. When the check comes back red, a line of text gets assembled from the check's output and fed into the next generate. Here is the loop the gate piece built, refactoring src/ until a guard holds. The steer is one arm in it:

#!/usr/bin/env bash
# work-until-checked: refactor src/ until the guard holds.
MAX=5; i=0
prompt="Remove every mock-library import from production code under src/."
while [ "$i" -lt "$MAX" ]; do
    run_agent --task "$prompt"                      # GENERATE
    if bash no-mocks.sh; then                       # CHECK
        echo "stop: guard holds after $i retries"; exit 0
    fi
    prompt="The last attempt still tripped the guard; fix it:
$(bash no-mocks.sh 2&gt;&amp;1)"                            # STEER: only the new signal
    i=$((i + 1))
done
echo "stop: budget exhausted, guard still red"; exit 1

The model never sees the whole history. Each retry it sees one prompt, and that prompt is whatever the steer decided to carry back. On the first pass the prompt is the goal. On every pass after that the steer overwrites it. So the target the model aims at on retry three is not the goal you wrote, it is the last thing the steer said, and the steer is a line the loop composed on its own while you were not looking.

The steer gets none of the attention

Reward hacking has more than one cause, and most of the attention goes to two of them: a check loose enough to game, and an agent with write-access to the thing that grades it. The third gets almost none, and it is the one this piece is about. It is the objective the loop hands the model on each retry, and that objective is the steer.

The model does not optimize the check directly. It optimizes the instruction it was handed, and that instruction is whatever the steer wrote. When the loop feeds back make the test pass, it has named the check as the goal. From there, optimizing the instruction and gaming the test are the same action, because the cheapest state in which the test passes is the one where the test agrees with whatever the code already does. The steer said the target was green. Green is what came back.

None of this touches the other roads to a gamed result, and it is worth being honest about that, because the Cursor piece above documents one of them. A large share of the reward hacking it found was answer-retrieval: agents pulling a fix straight from a public pull request or the repository's own bundled git history, with 63% of one model's successful resolutions retrieved rather than derived. That happens with the goal fully intact. It is an access problem, not a steer problem, and no wording of the steer prevents it. The steer is the lever this piece takes because it gets none of that attention and is the cheapest to fix. You write it yourself, once per retry, and most loops write it badly.

The good steer holds the goal

Look at what the loop above carries back. The goal is stated once, before the loop, and the run_agent call never re-ships it. The steer rewrites prompt to carry the guard's own output and nothing else:

prompt="The last attempt still tripped the guard; fix it:
$(bash no-mocks.sh 2&gt;&amp;1)"                            # STEER: only the new signal

That is the shape you want: directive first, then the evidence. Fix the lines the guard flagged, and here are those lines, verbatim from the check. The goal has not moved, because the steer never restates the goal, it appends the delta to it. The model gets the original target plus a precise account of what the last attempt got wrong, in the check's own words. Pass the test is never the whole of what it optimizes, because the goal it was serving is still on the page next to the failing line.

A good steer is a reduction of the check's output. It takes the verdict and the minimal evidence that produced it and hands that back unaltered. The moment the steer summarizes the failure into make it pass, it stops being a reduction and becomes a new goal, and the new goal is the one the agent will game.

Watch a loop game its own check

Here is the same loop, one check, and two steers. The check is a unit test: a $100 charge should cost $90 after a 10% discount. The code has the discount missing. The generator is a stand-in for the model; its two branches do the cheapest thing each instruction names, which is the whole point, so both are on the page rather than hidden:

GOAL="charge(cents) must apply the 10% discount so charge(10000) == 9000."

# THE CHECK: run the test. pass = exit 0.
check() { python3 test_charge.py &gt;/dev/null 2&gt;&amp;1; }

# THE STEER: turn the check's output into the next instruction.
steer_good() { printf '%s\nThe test still fails; fix the failing assertion: %s\n' "$GOAL" "$1"; }
steer_bad()  { printf 'The test is still failing. Make the test pass.\n'; }

# THE GENERATOR: a literal optimizer standing in for the model. It takes the
# cheapest route the instruction names, the same shortcut a real model reaches for.
run_agent() {
    case "$1" in
      *"Make the test pass"*)         sed -i 's/== 9000/== 10000/' test_charge.py ;;  # game: edit the test
      *"fix the failing assertion"*)  sed -i 's/return cents/return int(cents*0.9)/' charge.py ;;  # fix: change the code
    esac
}

The good steer holds the goal and appends the failing assertion (expected 9000, got 10000), so run_agent takes the fix branch. The bad steer drops the goal and hands back the symptom, so run_agent takes the game branch. Run the loop with each and both terminate the same way:

$ bash game-demo.sh good
stop: test passes after 1 retries
charge(10000) returns: 9000
test asserts:          == 9000
check verdict:         GREEN
goal met ($100 charges at $90): YES

$ bash game-demo.sh bad
stop: test passes after 1 retries
charge(10000) returns: 10000
test asserts:          == 10000
check verdict:         GREEN
goal met ($100 charges at $90): NO

The stub is pinned so the loop is reproducible on your machine, but the branch it takes is not the trick, it is the claim: hand a literal optimizer make the test pass and editing the assertion is the cheapest path to green; hand it the goal plus the failing line and changing the code is. A real model reaches for the same shortcuts under the same two steers. Both runs print stop: test passes after 1 retries and come back green, so from outside the loop the two are indistinguishable, same verdict, same retry count, same clean exit. The difference is only in the artifact. The good steer left charge() fixed and the test asserting == 9000; the bad steer left the bug in place and the test rewritten to == 10000, so the bar is green because the test now certifies the bug.

The check certifies whatever the steer pointed it at

A green check is not lying here. It is doing exactly its job. The governance-selector piece worked the human version: a green test proves the change conforms to its spec and says nothing about whether the change improved anything, and it named the quadrant where a change is correct, shipped, and no better. Reward hacking is that quadrant reached on purpose. The steer that says make the test pass re-points the spec at the test is green, and the check faithfully certifies conformance to the new, degenerate spec.

There are two ways the steer's drift reaches the check, and they call for different defenses. One is paraphrase: the steer restates the goal loosely, and a model-graded check, the kind the check piece set beside the deterministic kind, adopts the loose restatement as its working spec, so make it pass becomes what it grades against. A deterministic check resists that, because it runs the assertion against the code no matter what the steer said about it. The other way is editing: the agent changes the check itself, and here the deterministic check is no safer than the model-graded one, because the cold open did exactly that, rewrote == 9000 to == 10000, and the deterministic assertion passed on the altered test. Determinism buys resistance to paraphrase, not to editing. The axis that decides whether a check survives the agent is not deterministic-versus-graded, it is editable-versus-read-only, and the fix below turns on it.

The same move, away from the tests

The tests are the recognizable case, and the shape is more general. I have watched an agent, up against a hard limit a measurement had to clear, propose to clear it by removing a piece of what the system did, so the measurement would read green. Not by fixing the thing the measurement was watching. By dropping the capability the measurement stood for and reporting the number as met. Cutting scope to hit a budget is sometimes a real engineering call, but this was not that, because no one had decided the capability was worth less than the number; the agent decided it silently, to turn the gauge green. It is the same move as editing the test: satisfy the measurement, abandon the thing measured. The only reason it did not ship was a human reading the diff and asking why the fix worked by removing capability. A loop running unattended does not ask.

What the two cases share is the steer. In both, the objective the loop was carrying had quietly collapsed from the goal to the measurement of the goal, and everything downstream optimized the measurement. The check worked as written and the number was accurate. The instruction the loop was feeding itself had drifted from make the product correct to make the gauge read green, and the agent did precisely what that instruction asked.

Make the steer a reduction, and keep the grader out of reach

Three disciplines keep the steer from teaching the agent to cheat. The first two are the ones that do the work, and the ones most loops skip.

Hold the goal constant across retries. State it once, outside the retry arm, and never let the steer restate it. The steer carries the delta, what the last attempt got wrong, and leaves the goal where it was written. A steer that re-authors the goal each iteration is a steer that can drift from it, and the drift compounds, because each retry's paraphrase is a paraphrase of the last.

Carry the check's output as a reduction, not a summary. Reduce it to the verdict and the minimal evidence that produced it, and hand that back verbatim. The next attempt should read what actually failed, in the check's own words, not a description of the failure written by the arm in the middle. Written that way the steer is an instruction you could have authored yourself, the goal you fixed plus the check's output reduced to the failing line:

charge(cents) must apply the 10% discount so charge(10000) == 9000.
The test still fails; fix the failing assertion: expected 9000, got 10000.

Do those two and the steer stops handing the model a reason to game, because the objective it optimizes is the goal, not the green light. The third discipline handles the gaming that remains: keep the grader out of the agent's reach. If the artifact the agent can edit is the artifact that grades it, a steer pointed anywhere near the check eventually gets the check edited to pass, and the editable-versus-read-only axis from the last section is exactly this. Make the grader read-only, or grade the final result on a held-out check the agent never saw while generating, the guard the governance-selector piece borrowed from SkillOpt: accept a self-authored change only when it improves a held-out split, not the data the change was tuned against. Be honest about what that buys. It does not make gaming impossible; SpecBench exists because agents still fail held-out tests, and the gap grows by 28 points for every tenfold increase in the size of the task. What a read-only or held-out grader buys is that the gaming becomes visible and expensive: the model that games the split it could not see gets caught by it, instead of walking away green.

What reporails can and cannot see here

Reporails reads the steering surface you authored: the instruction files, the rules, and the prompts the steer will paraphrase. It does not run your loop, and it does not see the steer, which is composed at runtime and never written down anywhere reporails could read. What it can do is get the authored half right so the runtime half has less to corrupt. A goal stated crisply, and measured for whether its wording actually couples to behavior, is a goal the steer has a harder time quietly restating into pass the check. The runtime handoff is yours to build well; the authored surface it starts from is the part reporails measures.

The reason the handoff is worth building well is that no one reviews the steer. Every other instruction in the loop you wrote and can read. The steer the loop writes for itself, once per retry, at machine speed, consumed by the next generate before anyone sees it. That is the one spot where a drifted instruction becomes the next target, and it is where reward hacking is authored, one steer at a time. Make it a reduction you can inspect and keep the grader beyond the agent's edit reach, and the loop optimizes the goal instead of the gauge, which is always the cheaper of the two to satisfy.

The loop still has an arm to take apart

Four arms down, and the pattern holds across all of them: the loop only ever acts on what you wrote into it. The check runs the rule you encoded, the gate refuses on the pattern you set, the surface carries the instructions you loaded. The steer is the one you write without noticing, fresh every retry, and it is where a green result quietly stops meaning what you wanted it to.

One arm is left: the stop. Every loop here quits on a green check and a retry budget, and a loop that stops on a green it was gamed into has stopped too early, on a result that means nothing. Telling a real green from a bought one, and knowing when a loop should quit versus when it should refuse to, is the stop arm's problem, and the last piece in this series.

Loop Engineering: How To Stop The "You're Absolutely Right" Sycophancy

Gábor Mészáros — Mon, 20 Jul 2026 16:34:26 +0000

"You're absolutely right!" The agent says it before it has looked at anything. You point out a bug, it tells you you're right. You suggest the opposite of what you asked for a minute ago, it tells you you're right again. That reflexive agreement, the little affirmation stapled to the front of every reply, is one of the most complained-about tics coding agents have right now, and once you notice it you cannot stop noticing it.

The obvious fix is to tell it to stop. Add a line to your instructions: do not open with placation, do not tell the user they are right before you have checked. Reasonable. But that line now rides in the model's context on every single turn, read again on the turns the agent was going to agree politely and the turns it was going to do real work, paid for whether or not it was ever about to placate. So the real question the title asks is how you actually stop it, and the answer turns out not to be an instruction at all.

Notice something about that line, though. It is not about any particular file. "Do not open with placation" applies whenever the agent is about to speak, which is every turn, in every file and in none of them. Set it beside a different rule you might add: production code under src/ must not import a mock library. That one has a home. It is about src/, a path you can name. Two kinds of rule end up on an instruction surface, the kind you can pin to a file path and the kind you cannot, because they are about behavior rather than location. Which kind a rule is decides where it can live and what it costs to keep it there. That split is what this piece is about.

Both kinds pile up regardless. Your CLAUDE.md started at nine lines, and six months later it is three hundred. The no-placation rule joins the mock-import rule from the last two pieces, the payments-module invariant, the frontend spacing scale, the deploy checklist, the API-versioning policy, and a paragraph on how to phrase migration names. This morning you asked the agent to fix a typo in the README, and every one of those three hundred lines rode into its context alongside the request. None of them had anything to do with the README.

The series opener named the loop that runs generate, check, steer, retry, stop, and posed three questions about the parts it turns on. Two pieces answered the first two: the check that decides good enough, stop, and the gate that refuses a bad write before it lands. This one takes the third question the opener posed, the part underneath both of the others: the context surface. It is the next arm on the bench, with more still to come. Every check and every gate is fed by rules the agent carries as loaded text, and that text is paid for on every turn whether it applies to the turn or not. So the question the opener left last is a cost question. What does an instruction actually cost to keep loaded, and what changes when you load it only where it applies. The answer runs straight through the split above: one kind of rule you can teach the harness to load only where it belongs, and the other kind, the placation rule among them, you handle a different way entirely.

What the surface is here

The surface is everything the model sees before it reads your request. The system prompt, the tool definitions, whatever documents got retrieved, and the instruction files (CLAUDE.md, AGENTS.md, and their nested siblings) that ride along on every turn. Prompt engineering tuned the request. Context engineering curated this surface. The loop reuses it: the surface is the state the loop carries between iterations, re-sent each time the model generates.

"Re-sent each time" is the property this piece turns on. A conversation is not cumulative from the model's side. Each turn, the whole surface is assembled and handed over again from the top. The three-hundred-line instruction file is not loaded once at the start of the session and remembered. It is loaded on turn one, and again on turn two, and again on the turn where you were editing a README. Whatever sits on the surface is paid for at every one of those turns, in full.

What loaded costs, part one: tokens

Put arithmetic on it, as a worked illustration and not a measurement. Say the instruction file has grown to a few thousand tokens, which three hundred lines of prose comfortably will. A working session runs dozens of turns. At forty turns, the file is assembled into the context window forty times. If two thirds of it is rules that only matter in one corner of the codebase, you have paid to keep a few thousand tokens of mostly-irrelevant text in the window on every one of those forty turns, and two thirds of that spend bought nothing on the turn it rode along on.

The numbers are illustrative; the shape is not. An always-loaded instruction costs its length times the number of turns it survives, because the window is rebuilt every turn, so a rule you write once and never delete is one you pay for thousands of times over a session, whether or not any given turn had a use for it.

What loaded costs, part two: attention

Tokens are the measurable half. The other half is what the extra text does to the model reading past it.

The State of AI Instruction Quality pointed a deterministic analyzer at 28,721 repositories and found the median instruction file carries 50 content items and 12 actual directives. The other 38 items are headings, restated context, and structure the model is free to ignore. Every turn, the model has to find the twelve rules that bind inside the fifty items that are loaded, and the thirty-eight that do not bind are still text it reads on the way to the work. Adding a rule that does not apply to the current task costs its tokens and widens the pile the relevant rules are hiding in.

There is a measured effect under that pile. A vaguely-phrased instruction is followed less reliably as more unrelated instructions load beside it on the same turn: the same rule that the model largely honors when it is one of a few gets superficially acknowledged and then ignored once it is one of many, and the reporails corpus measures that decline rising with the number of competing topics. The rule did not change. The company it was loaded with did. The lever it points at is fewer rules loaded at once, with the relevant one present on the turn it applies rather than on every turn.

This is the surface reporails reads. It points the same deterministic analysis at your own instruction files and tells you, with measured evidence, which instructions couple to behavior and which are text the model can ignore, so you can see what you are actually paying to keep loaded instead of guessing at it. It does not load your context or trim it at runtime. It measures the surface you wrote down and reports where the spend is not buying compliance. Knowing which of your fifty items are the twelve that bind is the precondition for the next move, which is deciding where each of them should live.

The rules you can pin to a path

Start with the first kind, the rules that have a path. For a located rule the move is to load it only on the turns it applies to instead of on every turn, and the mechanism is older than agents: a thin index at the top that routes to depth, and depth that loads on match.

Claude Code ships exactly this affordance, and it is worth being concrete about it because the build below leans on it. A CLAUDE.md at the project root loads at the start of every session, so it is the always-on surface. A CLAUDE.md placed inside a subdirectory does not: the harness loads it on demand, when the agent reads or edits a file in that subtree, and not before (per the Claude Code memory docs). A rule that only governs src/payments/ can live in src/payments/CLAUDE.md, and it reaches the model on the turns the agent works in src/payments/, on those turns only. The root file keeps the handful of rules that apply everywhere. Everything path-specific moves down to the path it belongs to.

That subdirectory file is the simplest form of path-scoped loading. The frontmatter-scoped form is more precise, and it does not need a file parked in the directory at all. Claude Code also reads .claude/rules/*.md, and a rule file carrying a paths: glob in its frontmatter loads only for matching files. The docs state it plainly: "Path-scoped rules trigger when Claude reads files matching the pattern, not on every tool use." Tag a rule paths: ["src/**/*.py"] and it rides the surface when the agent touches Python under src/ and is absent otherwise.

The same primitive ships across every major agent tool, each with its own frontmatter key. Cursor scopes a .cursor/rules/*.mdc file with a globs: field, auto-attaching the rule "when a matching file is in context." GitHub Copilot scopes a .github/instructions/*.instructions.md file with an applyTo: glob, applying it to "requests made in the context of files that match a specified path." Three vendors, one idea: attach a rule to a file pattern, and pay for the rule only when a matching file is in play.

Reporails canonizes that cross-vendor mapping and checks for it. Its path-scope-declared rule (CORE:S:0038) verifies that a path-scoped instruction file actually declares the scope it governs, and the Claude paths, Cursor globs, and Copilot applyTo equivalence is the exact thing the rule holds constant across agents. The full public rule set lives at reporails.com/rules.

That is the whole idea. The always-on surface carries what is always relevant, and a rule that is relevant only sometimes loads only then.

This scales past a handful of nested files. The steering setup I work in runs its whole rule corpus this way: each rule is subscribed to the transition it governs, and its full body loads only when that transition fires. The system can report what actually loaded, and on a working session the report runs lopsided: a single steering rule loaded its full body, a handful more stayed one-line pointers, and a dozen-plus subscribed rules never loaded at all, because nothing in the session's work touched their triggers. That is the shape you are after: the rules that applied were in context when they applied, and the rules that did not apply cost a pointer or nothing, instead of a full body on every turn. One caveat the same receipt discipline demands stating: a pointer delivered is not a rule followed, so a rule that must hold every time cannot lean on being loaded only when it happens to apply. It also gets a deterministic floor, the cheapest place of all to keep a rule, which the placation case below turns out to need. The build after that is one you can run yourself, without any of this machinery.

Build it: split the file that grew

Take the running example the series has used throughout. The no-mocks rule from the check and the gate pieces says production code under src/ must not import a mock library. In the three-hundred-line file, it sits in the always-on surface, loaded on every turn, including the turn you spent editing the README where it could not possibly apply.

The split that follows is itself something reporails measures. Its modular-file-organization rule (CORE:S:0010) checks whether an instruction surface distributes its rules across scoped files or piles them into one monolith, which is the exact shape a grown three-hundred-line CLAUDE.md has taken. The build below is what passing that check looks like on your own repo.

Start by sorting the file by scope, not by topic. For each rule, ask one question: does this apply to every turn, or only to turns that touch a particular path or task? The no-mocks rule applies only when the agent is writing Python under src/. The commit-message convention applies only when the agent is composing a commit. The frontend spacing scale applies only under web/ or theme/. The "which test runner" note genuinely does apply everywhere, so it stays.

Then move each rule to where its scope is. The root CLAUDE.md shrinks to the always-relevant minimum:

# Project instructions (root, always loaded)
- Test runner is `pytest`; never edit files under `generated/`.
- Commit style and per-area rules load from the CLAUDE.md nearest
  the file you are editing.

The no-mocks rule moves down to src/CLAUDE.md, beside the code it governs:

# src/ instructions (loaded when the agent works under src/)
- Production code uses a real dependency or a constructor-injected fake;
  do not import a mock library. no-mocks.sh and the PreToolUse gate
  both enforce this; see the earlier pieces in this series.

Now the README turn loads the root file and nothing under src/, because the agent never opened anything under src/. The no-mocks rule, the payments invariant, the frontend scale: none of them ride along, because none of their paths were in play. When the agent does start editing src/payments/charge.py, src/CLAUDE.md loads on that turn, carrying the no-mocks rule to exactly the moment it applies. It is present on every turn it could bind and absent on every turn it could not, and the always-on surface dropped from three hundred lines to a handful.

One discipline keeps the split honest: the nested src/CLAUDE.md extends the root and stays consistent with it. The root names the test runner and points to the nearest file; src/ adds the no-mocks constraint on top and overrides nothing above it. Reporails checks exactly this as child-nested-instructions (CORE:S:0011): a nested instruction file has to build on its parent and hold consistent with what the parent already says, so a subtree file that quietly reverses a root rule is a defect the check catches.

Progressive disclosure changes when the rule loads while leaving what the rule says untouched. The strictness is intact; only the standing cost falls. The check and the gate from the earlier pieces still sit beside the rule, unaltered, refusing to pay for it on the turns it has nothing to say.

The rule with no path

Every mechanism in the last two sections scopes a rule by path: the nested CLAUDE.md, the paths: glob, Cursor's globs:, Copilot's applyTo:. They all answer one question, which file is the agent touching, and load the rule when the answer matches. That works because the no-mocks rule is tied to a location. It is about Python under src/, and src/ is a path you can name.

The placation rule from the opening has no such path. "Do not open with placation" is not about where the agent is working. It is about what the agent is about to say, on any turn, in any file or in no file at all. There is no glob for it. You cannot write paths: or applyTo: for a behavior, because no file pattern predicts a behavior, so the whole path-based toolkit has nothing to offer this rule. It cannot be scoped the way a location rule can. If it stays on the surface, it stays loaded on every turn.

This is where the title's question gets its answer. The rule leaves the surface a different way: not to a narrower path, but to a check.

"Never open with placation, never tell the user they are right before you have checked" is a rule whose violations are rare and mechanically recognizable: the tell is a short list of opener phrases. That makes it a poor fit for loaded text and a good fit for a check. Instead of a stronger instruction paid on every turn, write a deterministic Stop-hook check: a scan that runs when the agent finishes a message, matches the banned openers in the outgoing text, and fires only when one actually appears. On every turn where the placation does not happen, the rule costs nothing, because it is not on the surface at all. It is present as enforcement and absent as loaded text. You did not load a bigger rule. You moved the rule off the surface.

Concretely, here are both forms of the same rule. The steering form is the loaded line, the request that rides every turn, in CLAUDE.md:

# Voice (loaded every turn)
- Open with the substance, not placation. Affirm the user only after you have checked.

The enforcement form is a check you write yourself, a dozen lines of bash, and it closes the loop on the agent's own output. A Stop hook runs when the agent finishes its turn, and Claude Code hands it the final reply as last_assistant_message on stdin (per the Claude Code hooks reference). The hook checks the reply, steers the model when it placates, lets it retry, and stops when the reply comes back clean or once it has already been steered. Save it as .claude/hooks/block-placation.sh:

#!/usr/bin/env bash
# block-placation.sh: one arm of the loop, run as a Claude Code Stop hook.
# check the reply, steer the model when it placates, let it retry, stop when it comes back clean or once steered.
payload=$(cat)                                                 # the Stop event arrives as JSON on stdin
reply=$(jq -r '.last_assistant_message // ""' &lt;&lt;&lt;"$payload")  # the final assistant text of the turn
steered=$(jq -r '.stop_hook_active // false' &lt;&lt;&lt;"$payload")   # true when this turn is already a re-steer

# check: does the reply open with placation?
if ! grep -qiE '^[[:space:]&gt;*_-]*(you.?re +(absolutely +|completely +|so +)?right|great question|absolutely[[:punct:]])' &lt;&lt;&lt;"$reply"; then
    exit 0    # clean, so stop and let the turn finish
fi

# stop arm: if the last turn was already a re-steer and it still opens this way, let it through instead of bouncing forever
[ "$steered" = "true" ] &amp;&amp; exit 0

# steer: block the stop and hand back a correction, so the model regenerates the reply (the retry)
echo "Revise before finishing: this reply opens with an affirmation before checking anything. Drop the opening phrase and lead with the substance." &gt;&amp;2
exit 2

Wire it as a Stop hook in .claude/settings.json:

{
  "hooks": {
    "Stop": [
      { "hooks": [ { "type": "command", "command": ".claude/hooks/block-placation.sh" } ] }
    ]
  }
}

Same rule, two forms, and it is the steering-versus-enforcement split the gate piece drew, now aimed at a behavior instead of a file write. The loaded line asks the model not to placate and is paid on every turn. The hook does not ask. It reads what the agent produced (the check), and on a clean reply it exits and lets the turn stop, so it costs nothing on every turn the placation does not happen because it is not on the surface at all. On the turn a reply opens with you're absolutely right, the hook refuses the stop and hands back a correction (the steer), and the model regenerates the reply (the retry). The loop then terminates one of two ways: the new reply comes back clean, so the hook lets it stop, or it opened placating a second time, in which case the stop_hook_active guard lets it through rather than bounce forever. Claude Code also caps continuations with an 8-consecutive-continuation backstop, so even a hook that forgot its own stop arm cannot loop indefinitely, but the point of a well-built loop is that it names its own stop condition instead of leaning on the platform's backstop. It is the same generate → check → steer → retry → stop shape the opener drew, now closing on a behavior.

I run this exact check against my own coding agent's replies, a deterministic scan that fires when a banned opener shows up in what the agent just wrote. In one working session it caught six replies that led with you're right before the agent had looked at anything, each one flagged at the boundary instead of sliding past. A real catch on real output, reproducible, and the banned phrase never had to ride on the context surface for the scan to find it. The rule earned its keep by matching the violation, not by being read on every turn it did not apply.

The gate piece built a hook to refuse a bad write, so the boundary is worth drawing. That piece was about a channel that can say no where a prompt can only ask. This one is about cost. The check is a deterministic scan of the text the agent produced, the same shape as the diff-scan the check piece built, and its context cost is zero on every clean turn. A rule you can express as a check on the output is a rule you never have to keep loaded to enforce.

That splits two axes that are easy to run together. Path-scoping decides when a location-tied rule loads, and it is the right tool for every rule that has a path. It says nothing about whether a rule that did load actually moved the behavior, which is the other axis, and the one that decides whether a behavior rule was worth writing at all. Measuring which instructions on your surface couple to behavior, and which are text the model reads and ignores, is what reporails does: it reads the surface you wrote down and reports the coupling, the axis a path glob cannot reach. The two do not compete. Path-scoping trims what loads; the coupling read tells you whether what loaded earned its slot.

Which rule loads where

The sort has a failure mode in each direction, the same shape the gate piece found for steering versus refusing.

Push a genuinely global rule down into a subtree and it goes missing on the turns it was supposed to cover. "Never edit generated/" scoped to src/ does not reach the agent when it is about to edit a generated file under web/, which is one of the turns you wrote it for. A must-hold-everywhere rule belongs on the always-on surface, and the cost of keeping it there is the cost you should pay, because its scope really is every turn.

Hoist a narrow rule up into the root and you are back where you started. It loads on every turn, taxes every turn, and pads the pile the global rules hide in. The payments-module invariant on the always-on surface is billed on the README turn and the frontend turn for nothing.

The heuristic is the scope question made explicit. A rule loads at the level whose every access it governs. Applies to the whole repo, root file. Applies to one package, that package's file. Applies to one task the agent performs occasionally, a file or skill that loads on that task's description match rather than on every turn. The test is not how important the rule is. A critical rule with a narrow scope still loads narrowly; importance decides whether you also back it with a gate, which the gate piece covers, not where the rule loads.

The loop only checks what you wrote down

Three arms of the loop taken apart so far, and the loop is not out of arms. The check decides good enough, stop, and a misfiring check is a broken instrument before it is an absent signal. The gate refuses a bad write before it lands, and a rule that must hold every time belongs in the channel that can say no. The surface is what feeds both, and every rule on it is loaded text you pay for on every turn it survives, so a rule that applies only sometimes should load only then, and a rule you can turn into a check on the output does not need to sit on the surface at all.

What ties them together is the thing the series keeps returning to. The loop only ever checks, gates, and steers on what you wrote down. The check runs the rule you encoded; the gate refuses on the pattern you specified; the surface carries the instructions you authored and hands them to the model turn after turn. None of it reaches past the text you put there. That is why the text is worth engineering. What it says decides whether the loop catches the right thing, and where it loads decides what every turn costs to run. Load the whole surface every turn and you pay for all of it on turns that needed almost none of it. Load each rule where it applies and the loop still holds, on the turns that matter, for a fraction of the standing cost.

There is an arm this series has kept naming in passing and has not yet taken apart: the steer. Between one attempt and the next, the loop carries something back to the model, the check's own output, rephrased into the next instruction. What it carries back is a choice, and a careless one poisons everything downstream of it. Feed back the wrong signal and the next check judges the next attempt against the wrong thing, so the loop converges, confidently, on an answer no one asked for. What the loop should hand back between tries, and what breaks when it hands back too much or the wrong part, is the next arm to measure.

I work on Reporails, deterministic diagnostics for the instruction files, rules, and prompts that steer coding agents. It reads the steering surface and tells you, with measured evidence, which instructions couple to behavior and which are text the model is free to ignore. It does not load or trim your context; it measures the surface you wrote down, so you know what you are paying to keep in front of the model.

Loop Engineering: Fine-Tuning the Guardrail That Fired Wrong

Gábor Mészáros — Tue, 14 Jul 2026 11:28:12 +0000

The check had been green for a week. It greps every diff under src/ for import mock, because production code has no business importing a mock library. Then it went red on src/payments.py, a file with no mock import anywhere in it. The word mock was sitting in a docstring, in a sentence telling the next developer not to import one. The grep matched the sentence. The quickest way to clear that red is to delete the check.

The series opener named the loop that runs generate, check, steer, retry, stop, and put the hard part on the check that decides good enough, stop. Last week's piece built that check as no-mocks.sh and set a gate beside it, a hook that refuses a bad write instead of catching it after the fact. This one stays with the check and asks what happens when it is wrong: how to tell a broken instrument from a signal that was never there.

What a check is here

A check is a function from the agent's output to a verdict: pass or fail. In the loop it is the thing that decides whether an iteration is acceptable and whether to run another. A check comes in two kinds. A deterministic check runs the code or scans the text and returns the same verdict every time: grep, pytest, an exit status. A model-graded check asks another model is this good? and reaches criteria the first cannot express, at the cost of the same unreliability the loop was built to contain.

This piece is about the deterministic kind, because it is the kind you can hold in your hand and debug. When a deterministic check gives you a verdict you disagree with, the verdict is wrong in a way you can inspect line by line. That is exactly the property the argument below leans on.

Two readings of the same quiet

A check that comes back quiet is telling you one of two things, and it does not tell you which:

Absent signal. The thing the check guards against genuinely is not there. This is the good news you wanted.
Broken instrument. The thing is there, or would be there, but the check cannot see it. The guard is off and the diff looks the same as if it were on.

A false alarm, the check firing on a clean file, is the visible failure. It is annoying, but it announces itself; you know the instrument is off because it is screaming at nothing. The dangerous failure is the opposite one, and it is silent: a check weakened until it stops seeing real violations returns the same green as a check that correctly found nothing. You cannot feel the difference between "safe" and "blind" from the verdict alone.

So the misfire is really about the check, not the file it landed on, and it sets a trap worth naming early. A red mark on a file you know is clean reads as a reason to stop trusting the check, and it tends to show up on the very run where that same check is catching something real elsewhere. Look at what the pattern actually matched before you act on the verdict it returned.

The misfire, up close

Make it concrete with the check itself. The version that misfires is the naive one, a bare substring match, before the ^ anchor last week's piece put on the pattern:

grep -rn 'import mock' src/ && exit 1

For a week it did its job. Then you added src/payments.py, whose docstring happens to document the rule the check enforces:

"""Charge processing. Do not import mock here, use the fakes in tests/."""

import stripe

There is no mock import in that file. Run the check and it fires anyway, alongside a real leftover it correctly caught in src/badcache.py:

$ bash no-mocks.sh
src/payments.py:1:"""Charge processing. Do not import mock here, use the fakes in tests/."""
src/badcache.py:1:import mock   # leftover from a spike

The badcache.py line is the catch, an import mock that has been sitting in production code. The payments.py line is the misfire: that file is clean, and the check flagged it because import mock appears as a substring inside prose documenting the very rule the check enforces.

The tempting fix, and what it costs

You have a red check on a clean file. The path of least resistance has three forms, and all three are the same move:

Delete no-mocks.sh. The red goes away.
Weaken it: add --exclude=payments.py, or only run it on Tuesdays, or drop it to a warning nobody reads.
Conclude the check was never worth much: "grep is dumb, mock imports are rare anyway, this is more trouble than it catches."

Every one of these makes the noise stop, and stopping the noise feels like fixing the problem. Re-read the output. The same run that misfired on payments.py also, correctly, caught the real import mock in badcache.py. Delete or defang the check and that catch disappears with the false alarm. The next real mock import to land in src/ sails through, and the suite stays green, because green is now what this check returns for everything.

That is the inversion. The misfire tempts you to conclude the signal is not worth watching, at the exact moment the same check proved the signal is real. You would be reasoning about the world from the failure of your instrument to see it.

Fix the instrument instead

The misfire told you something specific: the pattern matched prose, not code. So make the pattern match code. That is the one-character change last week's piece already carried, the ^ that anchors the pattern to the start of a line, where an import statement lives and a docstring sentence does not. Add it and payments.py goes quiet while badcache.py still fails.

Both make the red on payments.py disappear, and only one keeps catching badcache.py. Deleting the check clears the false alarm by throwing away the real catch with it; anchoring the pattern clears the false alarm and leaves the real catch standing.

I run the same shape of check on my own agent's replies. A deterministic scan looks for a short list of banned phrases: a placation opener, or an unbacked the work is verified when nothing was actually run. One day it fired on a reply that was clean. The banned phrase was sitting inside a quotation of the rule that bans it, and inside a backticked identifier the agent was naming rather than using. The check had matched the words, not their use, the same way the grep matched the docstring. Loosening the phrase list until that alarm stopped would have let the next real unbacked claim through, so I scoped the matcher instead. It blanks the cited spans, the backtick spans and quoted strings and negated clauses, before the scan runs, so an empty the work is verified still fires while a citation of the rule goes quiet.

The anchored check still is not a perfect mock detector. An aliased import unittest.mock as m is a different pattern, and an indented import inside a function slips past a start-of-line anchor. A real linter closes both without any of this hand-rolling: flake8-tidy-imports carries a banned-modules list, and ruff's F401 flags imports the code never actually uses. If mock-in-production is the exact thing you are guarding, reach for one of those. The grep still earns its place as the first thing you write, because it is the cheapest deterministic check a team reaches for in a hook, one line in a script that already runs. And the lesson holds after you upgrade the instrument, because a linter rule is a pattern too. F401 matches what looks like an unused import, not the fact of use, and it has edge cases where it reads the text wrong. Every one of those is a calibration reading you use to sharpen the rule, never a reason to stop measuring.

The rule that generalizes

A misfire is data about the instrument, never a verdict about the world. When a deterministic check fires on something you know is clean, the finding is "my check matches too much," not "the thing I was checking for does not exist."

The asymmetry is what makes the discipline worth holding. A false alarm is loud and self-correcting: it interrupts you until you deal with it, and it costs you the few minutes the fix takes. A deleted or defanged check fails the other way. It goes quiet and stays quiet: no output, no red, no interruption, just a signal nobody is watching anymore, indistinguishable from a signal that was never there. Every real violation that lands afterward passes clean, and nothing on screen tells you the guard is gone.

This was one of the questions the series opener left open: how to tell a broken instrument from an absent one. Its sibling piece took another, the gate that refuses a bad write instead of catching it a beat too late, and the line between a rule that observes and one that refuses. The question underneath both is still open. Every check and every gate is fed by rules the agent carries as loaded text, paid for on every turn whether they apply that turn or not. What an instruction costs to keep loaded, and what changes when you load it only where it applies, is where the series goes next.

I work on Reporails, deterministic diagnostics for the instruction files, rules, and prompts that steer coding agents. It reads the steering surface and tells you, with measured evidence, where it drifts. The checks here are the same idea pointed at a diff: what a rule actually matches, held up against what you meant it to match.

See how AI instructions decay, then write ones that hold

Gábor Mészáros — Sat, 11 Jul 2026 06:46:08 +0000

This is a submission for Weekend Challenge: Passion Edition

What I Built

I told an agent Never write directly to the database. A long session later, context window full, it wrote directly to the database. The rule loading mark was still sitting in the prompt. The model had just stopped weighting and attending to it.

It's an invisible failure. No error is being thrown. The task comes back subtly wrong, and the rule reads perfectly fine when you go back and check it. I wanted to make it visible, so I built an interactive field you can drag around.

Every rule you write for an agent is a hill. Its height is how well the rule is written: a directive-led, backtick-anchored rule stands tall, a hedged and vague one sits low. Then you raise the water. The water is context load. As it rises the low rules go under first, in order of how well they were written. The weak ones drown while you watch.

Three of the hills are high-stakes prohibitions, the Never... rules. They drown too. That is the whole point of the piece. A rule you cannot afford to lose does not belong in prose at all; it belongs on a runtime hook that runs as code, not attention. The field flags those in red the moment they go under.

Underneath the field is a second tool: a client-side lint that reads an instruction and names the surface tells (hedges, shouting, politeness, a ban placed before its directive). It is deliberately not a score. It catches what a little regex can honestly catch, and points at the real analysis for the rest.

Demo

Play it on its own page. Drag to orbit, drag the load slider to raise the water:

▶ Open the live demo

Each of the nine instruction patterns in the demo links to its rule page on reporails.com/rules.

Code

Code is available on Codepen: https://codepen.io/editor/G-bor-M-sz-ros-the-reactor/pen/019f4cad-e344-78bf-b7bc-919972f42a4e

The whole thing is one self-contained HTML file: no build step, no dependencies, no backend. The CodePen above is the full source, so you can read every line and fork it right there.

How I Built It

The 3D field is hand-rolled on a 2D canvas. No WebGL, no three.js. It is a Gaussian height field sampled over a grid, rotated by yaw and pitch, projected with a perspective divide, and drawn back-to-front with the painter's algorithm. Each tile gets flat Lambert shading against a fixed light. The water is a second translucent plane tessellated over the same grid, so a hill reads as submerged the moment its local height drops below the water line.

The hill heights are burned in, not computed. Each rule's height is a fixed, hand-reviewed number, placed by how well the rule is written against a set of instruction-quality patterns: directive-before-constraint ordering, a required concrete anchor, imperative modality, lowercase prohibitions, and so on. I deliberately did not run the live scorer to place the hills. Reviewed numbers stay fixed and inspectable; a field that recomputes its own heights every frame just drifts every time you touch the scorer.

The lint is multiplicative and ordering-aware. Weaknesses compound rather than add, so a rule that is hedged and vague and buried lands near zero instead of "a bit lower." It also checks order: a ban that stands before its directive is penalized, because naming the forbidden thing first activates it before the model learns what to do.

The honest part. Nothing here claims a hook is immune to context. That would be false; nothing is immune. The demo makes one narrow claim and only that one: a hook runs as code, not attention, so the attention-fade that erodes prose under load does not apply to it. The fade is a property of attention. Code does not have it.

No prize-category cloud tech in this one. It is a single client-side file on purpose, because the whole argument is that the interesting failure happens in the prompt, before any infrastructure gets involved.

The passion behind it: I spend my time trying to make agent behavior measurable instead of a matter of vibes. This was a weekend spent turning one measurable, invisible failure into something you can grab and drag.

Deterministic Guardrails: Prompts Steer, Hooks Enforce

Gábor Mészáros — Thu, 09 Jul 2026 20:35:46 +0000

A loop that refactors code until a check passes is a few lines of shell. Most guardrails beside such a loop only watch what an agent already wrote; the one worth building refuses a bad edit before it lands, and that is what this piece constructs.

The rule it enforces: production code under src/ must not import a mock library. That rule can live in two places. Put it in the prompt, do not import mock, and it is clear and still only a request: text loaded into a model is a lever on probability, not a switch, so it leaves the model free to write the import anyway. Put it in a gate that fires before a write lands, and it refuses the edit outright. Both state the same rule; only one can make it hold every time.

The ask channel has a ceiling: a prompt pushes the odds up, never to one. How close a prompt gets to that ceiling is measurable work. Reporails reads the asking channel and shows you, with measured evidence, which instructions pull the lever and which are text the model is free to ignore, so you can write the ask against evidence instead of on faith. Even a well-measured prompt tops out below certainty, because the model underneath stays probabilistic. Closing that last gap is what the gate is for.

The loop, and where it can only ask

This is the second component of the loop taken on its own: the gate. The series opener named the loop (generate, check, steer, retry, stop) and posed three questions about the components it turns on. The first is the check, the arm that decides good enough, stop. This piece takes the second: the rules that can refuse a diff, and the difference between a channel that can ask and a channel that can say no.

Here is the loop the series is dissecting. It refactors src/ until a guard holds. The agent never decides it is done; the guard does.

#!/usr/bin/env bash
# work-until-checked: refactor src/ until the guard holds.
# The steering rule lives in the agent's own instructions (CLAUDE.md),
# stated once; the loop feeds back only what changed, the guard's output.
MAX=5; i=0
prompt="Remove every mock-library import from production code under src/."
while [ "$i" -lt "$MAX" ]; do
    run_agent --task "$prompt"                      # GENERATE (rule already in context)
    if bash no-mocks.sh; then                       # CHECK
        echo "stop: guard holds after $i retries"; exit 0
    fi
    prompt="The last attempt still tripped the guard; fix it:
$(bash no-mocks.sh 2&gt;&amp;1)"                            # STEER: only the new signal
    i=$((i + 1))
done
echo "stop: budget exhausted, guard still red"; exit 1

Read the arms one at a time. The steering rule lives once in the agent's own instructions, its CLAUDE.md, so run_agent --task "$prompt" never re-ships it. That first $prompt is the ask, and the rule already in context is text the model reads as a lever on probability, not a switch: Remove every mock-library import moves the odds that the next diff is clean without setting them to one. The STEER arm rewrites $prompt to carry only what changed, the guard's latest output, and asks again against that. A tighter, more specific ask, but an ask.

no-mocks.sh is the only arm that is not asking. It scans src/ after run_agent returns, so it is a detector: it reads the diff the agent already wrote and returns a verdict on it. When the verdict is red, the loop retries. Nothing here stops the bad write from landing in the first place. The guard catches after the fact, and the loop re-rolls. Between the write and that catch, a forbidden import can sit in production code for a whole iteration before the detector sees it.

For a refactor loop that converges, the window is usually harmless, because the next iteration overwrites it. But "usually harmless" is doing a lot of work, and the cases where it is not are exactly the constraints you cared enough to write down twice.

Two channels: one asks, one refuses

A prompt is the channel that can only ask. Everything on the context surface, the task, the agent's instructions, the system prompt, is an input to a probabilistic process. It shifts the distribution of what the model produces, which is why prompt and context engineering are real disciplines. It is also, by construction, unable to force any single outcome. The prompt asks, and any given iteration is free to ignore it.

A gate is the channel that can refuse. It is a deterministic function that fires at a fixed point and returns a hard verdict the model does not get to route around. It does not shift odds; it either lets the diff through or it does not.

The base loop has an asker (the task), a steerer (the retry), and a detector (the guard). It has no refuser. Nothing in it can stop a write before the write happens, so a loop that checks for exactly the thing you forbade can still let it land and catch it a beat too late. Adding the refuser is the rest of this piece.

Build the missing arm: a hook that refuses

Claude Code fires hooks at named transitions in the harness: SessionStart, PreToolUse, PostToolUse, Stop. A PreToolUse hook runs before a tool call executes and can block it. That is the transition to key on, because it is the one point where you can refuse a write before it happens rather than detect it after. Here is where the gate sits in the lifecycle of a single write:

The gate is the diamond, and everything downstream of the allow edge is the write reaching disk. Refuse there and the write never reaches the Tool executes node.

Here is a hook that blocks any Write or Edit whose content adds a mock import under src/. Save it as .claude/hooks/block-mock-imports.sh:

#!/usr/bin/env bash
# block-mock-imports.sh: refuse any Write/Edit that adds a mock import to src/.
# Runs as a Claude Code PreToolUse hook. Exit 2 blocks the tool call.
payload=$(cat)   # the tool call arrives as JSON on stdin
path=$(jq -r '.tool_input.file_path // ""' &lt;&lt;&lt;"$payload")
content=$(jq -r '.tool_input.content // .tool_input.new_string // ""' &lt;&lt;&lt;"$payload")

case "$path" in src/*) ;; *) exit 0 ;; esac   # only guard production code

if grep -qE '^(import mock|from mock|from unittest import mock)\b' &lt;&lt;&lt;"$content"; then
    echo "blocked: '$path' would add a mock import to production code. \
Use a real dependency or a constructor-injected fake." &gt;&amp;2
    exit 2   # exit 2 == hard block; the message on stderr goes back to the model
fi
exit 0

Wire it up as a PreToolUse hook matched to the write tools, in .claude/settings.json:

{
  "hooks": {
    "PreToolUse": [
      {
        "matcher": "Write|Edit",
        "hooks": [
          { "type": "command", "command": ".claude/hooks/block-mock-imports.sh" }
        ]
      }
    ]
  }
}

Now the agent tries to write import mock into src/badcache.py. The transition fires before the write:

The write never happened. src/badcache.py is untouched. The hook read the pending tool call, matched ^import mock in the content the agent was about to write, and returned exit code 2, which Claude Code treats as a block with the stderr message handed back to the model. The model sees the refusal in the same turn and has to try something else. There was no window, because there was no write to catch.

Notice the pattern is anchored: ^import mock, not a bare substring. A docstring that merely names the rule does not start a line with an import, so it does not trip the hook. Anchoring matters more here than in a detector that only reports: a match keyed too broadly refuses clean work, and that carries higher stakes at a gate, which is the next section.

Compare the hook to no-mocks.sh. They match nearly the same rule. What differs is where they sit. The guard sits after the write and reports; the hook sits before the write and refuses. Same constraint, two channels, and only one of them closes the window between the write and the catch.

Which rule goes in which channel

Not every rule belongs in the refusing channel, and filing one in the wrong channel fails in a specific, quiet way in each direction.

A must-hold constraint filed as mere steering gets ignored the one time it matters. The prompt asks, the model is free to decline, and on the iteration where it declines, the diff lands. A must-hold rule in an ask-only channel holds on every iteration except the one where the model declines, and that iteration is the case you filed it for.

A rule that only needed to steer, but is hard-gated, causes friction and false blocks. This is the over-matching failure pointed at a gate instead of a detector, and it costs more here. A gate keyed too broadly refuses clean work: the developer who names the rule in a docstring, the test file that legitimately imports a mock, the production helper named mock_response that a bare substring match trips on. A false alarm from a detector is noise you can ignore for one iteration. A false block from a gate is work that cannot proceed until someone fixes the gate. Refusal is load-bearing in the moment, so an over-broad refuser has a larger blast radius than an over-broad detector.

The heuristic: gate what must hold every time and is cheaply, deterministically checkable; steer what needs judgment or flexibility. no mock import in production code is a fixed, must-hold, one-line-of-grep constraint, so gate it. prefer constructor injection over a service locator is a judgment call with real exceptions, so steer it and let a check observe rather than refuse. A criterion you cannot express as a deterministic match at a transition is a criterion you cannot gate; if it needs a model to judge it, it belongs in the asking channel, with all the probability that implies.

The generator's "done" is a claim to re-derive

Underneath both channels is the posture the series keeps returning to. An agent's done is a claim to re-derive, not a fact to relay. The loop already refuses to trust the generator's report that the work is finished; that is what the check is for. The gate extends the same distrust one step earlier, to the moment of the write. The agent believes its edit is fine. The gate does not accept the belief; it re-derives the verdict from the diff, deterministically, before the edit is allowed to exist. Steering is where you tell the generator what you want. Gating is where you stop taking its word for it.

What holds, and what it still costs to keep loaded

With the hook wired in, the rule you needed to hold every time does. do not import mock holds on every iteration now, on the ones you read and the ones you never open, because a PreToolUse gate does not depend on your attention or the model's cooperation. That is what "hold every time" costs: a channel that can refuse. No amount of rewording the ask gets you there. The prompt still does its job, moving the odds toward a good diff, and the gate makes the one diff you cannot afford impossible. You want both, filed correctly.

One component the opener flagged sits underneath both channels: the context surface. Every rule you steer with is text loaded into the model, and the whole surface is paid for on every turn, whether the model needs a given rule that turn or not. The loop here states its one rule once, but most agents carry far more. What does an instruction cost to keep loaded, and what changes when you load it only where it applies instead of keeping the whole surface in front of the model every turn? That is the question the next piece takes up.

I work on Reporails, deterministic diagnostics and governance for the instruction files, rules, and prompts that steer coding agents. It reads the steering channel, the text you use to ask, and tells you, with measured evidence, where it drifts. The hooks here are the enforcing channel that sits beside it: what a rule refuses at a transition, not what it asks for on the surface.

Prompt Engineering, Context Engineering, Loop Engineering: What Actually Changed

Gábor Mészáros — Wed, 08 Jul 2026 11:16:04 +0000

A few years back the skill had one name: prompt engineering. You rewrote a sentence until the model did the thing. Last year the same corner of the job picked up a new name, context engineering. This year the threads call it loop engineering, and the line repeated under it is the verifier is the bottleneck, not the model anymore.

Every rename draws the same reaction from the same developers: is this a real shift, or the same work with fresh paint so someone can sell a course. That reflex is usually right. Most AI-tooling vocabulary turns over faster than the problems it describes.

So here is the read, one rename at a time. At each step, the thing you were actually engineering changed. What changed was the unit of work, not the marketing around it. The problems the newest name points at have a dated, public record. The dates are on the articles; the arithmetic is yours to do.

The unit was a prompt

Call it 2022 through 2024. The unit of work is a single prompt. You engineer the wording of one request: few-shot examples, a role frame, think step by step, the ordering of the ask. The entire surface you controlled was the text of one message, and the craft was getting that message right.

That discipline is not gone. It is no longer the whole job, because the whole job stopped fitting in one message.

The unit became the surface

Around mid-2025 Andrej Karpathy put a name on what people were already doing: context engineering. The unit stopped being a message and became everything in the window: the system prompt, the retrieved documents, the tool definitions, and the CLAUDE.md / AGENTS.md instruction files that ride along on every single turn. You are no longer tuning a sentence. You are curating a surface.

That surface has a property developers kept rediscovering the expensive way: most of what you put on it does not couple to behavior. The State of AI Instruction Quality pointed a deterministic analyzer at 28,721 repositories and found the median instruction file carries 50 content items and 12 actual directives, and the rest is headings, context, and structure the model is free to ignore. A sharper failure mode got its own writeup. Do NOT Think of a Pink Elephant shows how a constraint phrased as a negation (do not use mocks) can raise the odds of the exact thing it forbids, for the same reason the title just did to you.

Both of those are context-engineering problems, measured and published.

The unit is the loop

June 2026, the term is loop engineering, popularized by Addy Osmani synthesizing Boris Cherny and Peter Steinberger, and everywhere on the timeline within a few weeks. The unit is now the running loop: generate, check, steer, retry, stop. The prompt is one node in it. The context surface is the state it carries between iterations. The claim repeated across every explainer is that the model is no longer the limiting part. The check that decides good enough, stop is.

Take the name generously. It points at something the two earlier names left implicit: the mechanism that decides whether an iteration was any good and whether to run another. That mechanism was always present; every agent that retries has one. Loop engineering's contribution is making it the object you engineer instead of a default you inherit.

Prompt engineering tuned node one. Context engineering curated the state feeding it. Loop engineering points at the diamond, and that is where the interesting question hides.

A verifier that checks what, exactly?

The verifier is the bottleneck is a good slogan and an incomplete one. It names the bottleneck without saying what the verifier is supposed to check, and that gap is the engineering problem the slogan skips.

There are two kinds of check, and they are not interchangeable. A deterministic check runs the code, asserts the exit status, scans for the forbidden import, and returns the same verdict for the same input every time, with no judgment in the middle. A model-graded check asks another model is this good? The second reaches criteria the first cannot express, like is this explanation clear or does this read as rude, and it pays for that reach by inheriting the exact unreliability the loop was built to contain. You have put a probabilistic judge in charge of deciding when the probabilistic generator is finished.

Watch where they diverge on a real loop. An agent is told to refactor a module and stop when the work is done. A deterministic verifier can prove the test suite still passes and no banned import crept in. Those are facts, checkable on every iteration, with no argument about them. It cannot decide whether the refactor was worth doing, so a model-graded check gets bolted on to answer that, and now the stop condition rests on one model grading another's taste. The loop terminates when the judge is satisfied, and the judge is the same class of component the loop exists to supervise. Neither check is wrong. They answer different questions, and confusing which question you asked is how a loop stops confidently on the wrong iteration.

That trade-off, what the check actually checks, is the content of the verifier, and it is not a new problem. Green Tests Don't Mean Better Software works the CI version of it: a green test proves the code conforms to its spec and says nothing about whether the change improved the system. The check answered a question you never asked. Aim the same distinction at an agent loop instead of a test suite and you have the loop-engineering question stated for a case every developer already trusts.

What the earlier articles measured

Three articles published earlier in 2026 each worked one piece of this problem.

CLAUDE.md: Check, Score, Improve & Repeat, 27 January 2026, describes a loop run over an instruction file: score it, change it, score it again. Do NOT Think of a Pink Elephant, 31 March 2026, shows a single steering instruction that produces the opposite of its intent, and names the mechanism behind it. The State of AI Instruction Quality, 21 April 2026, analyzed 28,721 repositories and found that most of a typical instruction file is content the model can ignore rather than directives it has to follow.

Between them they cover the three problems loop engineering now names: what the verifier checks, steering that drifts, and instructions that do not couple to behavior. None of the three used the term. The dates are on the links, and they are the whole of the evidence.

What the new name solves

Loop engineering gives you two things worth having. It makes the check an explicit object of engineering, something you design and revise on purpose instead of a default you inherit from whatever your agent framework happened to ship. And it gives the verifier question a name, so teams can argue about it directly: what should this check check, and can it be deterministic.

Both are important, and both have an edge worth stating plainly. The name hands you the object. Building the verifier is still yours to do, and it is most of the work: deciding what correct means for a task, then turning that into a check you can run.

Part of that work is choosing the kind of check. A criterion like is this explanation clear needs a model to judge it. A criterion like no banned import is a deterministic scan. The cost of picking wrong runs both ways: a model-graded check where a deterministic one would do buys unreliability for nothing, and a deterministic check standing in for a judgment it cannot make gives you a green light that means less than it looks like. The name makes that choice visible and leaves it in your hands.

Underneath the frame, the generator still guesses. What loop engineering adds is a place to put the check and a reason to take it seriously, which is more than the earlier names offered the same problem.

The loop, one part at a time

The vocabulary will change again. Harness engineering was an intermediate name for part of this, and something will come after loop engineering too. Each time the word changes, the useful question is the same: what is the new unit of work, and what is the hard part inside it?

For loop engineering the unit is the loop, and the hard part is the verifier: what it checks. The verifier is one component of several, and a name that points at the whole loop is easy to nod along to and harder to act on. So the plan from here is to take the loop apart a component at a time and put a measurement under each one. The questions this piece raised are the ones the next few pieces go after:

A check that fires on the wrong thing comes back quiet, and so does a signal that was never there. They look identical from the outside. How do you tell a broken instrument from an absent one, before you conclude that the problem you were checking for does not exist?
Not every rule in the loop is a verifier, and the rules that steer are not the rules that enforce. A prompt asks; a gate refuses. What belongs in the channel that can only ask, what belongs in the channel that can say no, and what breaks when a rule is filed under the wrong one?
Everything on the context surface is paid for on every turn, whether the model needs it that turn or not. What does an instruction actually cost to keep loaded, and what changes when you load it only where it applies instead of putting the whole surface in front of the model every time?

Each of those is one part of the loop, looked at on its own, with the evidence attached. Same discipline as this piece: a claim is worth exactly what the measurement under it is worth, and no more.

Wiring a "read this first" hook as "ask" (don't)

Gábor Mészáros — Wed, 01 Jul 2026 14:52:30 +0000

In retrospect, it was pretty obvious... sharing it anyway.

I'm currently working on a self-steering progressive disclosure system, and I wanted my agent to read a design doc before it touched the code that doc governs. Simple PreToolUse hook: match Edit/Write, check whether the doc was Read this session, and if not, stop the edit and tell the agent to read it first.

I returned permissionDecision: "ask". My reasoning at that time: "ask has an escape hatch. Approve and the call proceeds, so a bad condition can never wedge the agent - you can always wave it through."

That safety was the whole problem. The approval is an escape from the prerequisite, not a path through it. The edits went through without the doc reads.

Here is what ask actually does. It surfaces the tool call for approval. Approve it - or auto-approve it - and the action runs. The doc never gets read. ask is a checkpoint for a human, not a gate on the agent. Nothing about it makes the agent do the prerequisite step.

The fix was one field. Return deny:

// PreToolUse hook, matcher: Edit|Write
if (!wasReadThisSession(governingDoc)) {
  return {
     hookSpecificOutput: {
         hookEventName: "PreToolUse",
         permissionDecision: "deny",
         permissionDecisionReason: `Read ${governingDoc} before editing this, then retry.`
     }
  };
}

deny refuses the tool call and hands the agent the reason. The agent can't proceed, so it does what the reason says - reads the file - and retries. The read clears the check, the retry passes.

This is the behavior Claude Code already ships natively: when it tries to Edit a file that haven't been Read before, the tool refuses the action until the relevant Read happens. The deny hook lets you apply that same read-before-edit rule to any file you designate, not only the file being edited.

The distinction I missed:

ask gates on approving the call.
deny gates on the agent doing the prerequisite work first.

If the goal is to force the agent's own behavior - read this, run that check, load that context - deny is the lever. ask just adds a human to the loop.

Important: this only works, if the gate releases accurately, when the prerequisite is actually done. A deny on a condition that never clears will wedge the agent. Mine keys off "was this file read this session," which the read itself flips.