DEV Community: Lex

Agents Get Replaced. The Doctrine Is the Product.

Lex — Mon, 05 Oct 2026 09:10:20 +0000

Earlier today a session in my own system tried to write a file outside its
territory. A hook denied it before the write happened, and the session
couldn't override itself: the bypass is an environment flag the hook process
reads, not something the agent can set. It had to hand the work back.

That hook is less than a week old. The rule behind it is much older than the
system it's running in.

From multiagent-system to agentic-os

I moved from multiagent-system — a 19-agent orchestration setup I'd been
running for months — to agentic-os, a rebuild, for three reasons: so the
system had memory and a record of everything, so I could work faster, and to
get closer to loops between agents. For now I still review every loop
myself. A human is in the loop.

After migrating I reviewed the initial agents, modified them, and reused a
lot of doctrine from the old system. The reason is simple: the AI was making
mistakes again that the old design had already had to correct.

So I audited it. On 2026-09-27 I took 100 rules from multiagent-system and
checked each one against agentic-os. 29 were already there, 35 were partial,
36 were missing. The 36 defined the work, and they were mostly enforcement:
rules the old system had turned into hooks, and this one still carried as
text. The roster was replaced; the doctrine was ported by subtraction,
keeping what corrects a known error and dropping the rest.

A rule is the record of an error the system already paid for once. Agents
change with each model generation. The list of mistakes a model tends to
repeat changes much more slowly.

What transferred

Four rows of that inventory:

Row	Rule
A01	Hard-block > advisory (advisory = 0% enforcement)
B01	The orchestrator is the only invoker
C01	The auditor is independent, anti-self-grading
D24	Deterministic signal before an LLM judge

A01 and B01 are marked present in agentic-os: three fail-closed hooks, and
the invoke tool blocked in the frontmatter of every subagent. A01 comes from
What Advisory Rules Actually Do in an Agent
Loop.
The measured result in the earlier system was "all 9 blocked tool-calls
across 7 runs, 4 via explicit logged override, 0 unauthorized writes". Those
numbers belong to multiagent-system. What carried over is the rule they
justify: a prompt that politely asks is not a boundary.

The other two are the same instinct. The author never grades its own work.
If a script can answer the question, the script answers before a model does.

None of this came from an AI course. It came from running live
products,
where one person owns the schedule, builders don't sign off their own
builds, and a metric beats an opinion. Only the workers changed.

The rules arrived with their tests

Each of the three guards was built the same way, and the order is in the
commit log. The test went in first, failing, with the hook absent — 3 of 12
red for territory, 2 of 13 for protected paths, 2 of 7 for the stop gate.
Then the hook, and the same suites green: 12 of 12, 13 of 13, 7 of 7.

Committing a red test is the point. A gate that has never been seen denying
anything is a gate nobody has tested — it is indistinguishable, from the
outside, from a gate that always says yes.

The deny path is the one that has to be boring. Empty input, malformed
input, an agent not in the config, an unreadable config file, any unhandled
exception: all of them deny. The failure mode of a verification layer should
be refusal, not silence.

The session that got denied this morning was mine, doing research for this
article. The rule that stopped it was ported from a system that no longer
runs.

Limits

Three of my five hard rules are still only prose. One is enforced by a hook, one partially, and three — default billing mode, nothing gets deleted from the vault, one script per file — exist as text and nothing else. The port was real and it was incomplete.
One operator, a handful of runs, one audit of 100 rules, and nobody outside has reviewed any of it.
A human is still in the loop. Loops between agents without my review are the goal, not the current state.
The inventory measures whether a rule exists, not its effect. A "yes" means present, not proven.

Agents get replaced. The doctrine is the product.

A Whole Lane Was Losing on Geometry, Not on Merit

Lex — Fri, 25 Sep 2026 15:43:38 +0000

I had three categories of document coming in, and a semantic classifier that
scored every one against all three. It worked. Except one of the three never
showed up near the top — not rarely, never.

The obvious explanations were that there were fewer of them, or that they
were simply worse matches. Both were wrong.

They matched just as well. They were losing on geometry.

How a whole lane can lose without being worse

The setup is the ordinary one. Each category is described in a paragraph;
that paragraph becomes a vector. Each document becomes another vector.
Cosine against all three, keep the highest: that's its lane, and that number
is its fit.

The bug is in the last step. Cosines get compared across lanes as if they
were the same unit. They are not.

A lane paragraph written in common vocabulary, heavily covered by the
model's training data, lands in a dense region of the space. Everything near
it scores high — 0.55, 0.60. A lane paragraph written in industry-specific
vocabulary lands somewhere sparser, where the same degree of real fit
produces 0.30 or 0.35. That gap isn't measuring fit. It's measuring how much
text the model has seen that talks like this.

So when you sort by raw cosine, the sparse lane loses every time. Not
usually — every time. Its best possible document scores below a dense lane's
mediocre one.

The fix is five lines

Don't compare across lanes. Normalise within each one.

Route every document to its lane by max cosine, same as before. Then, per
lane, take the mean and standard deviation of those cosines, and turn each
raw cosine into a z-score: how many deviations above its own lane's average
this document sits.

by_lane = {ln: [] for ln in lane_names}
for r in recs:
    by_lane[r["lane"]].append(r["cos"])

lane_stats = {}
for ln, v in by_lane.items():
    arr = np.array(v) if v else np.array([0.0])
    lane_stats[ln] = {"mu": float(arr.mean()),
                      "sd": float(arr.std() + 1e-9),
                      "n": len(v)}

for r in recs:
    s = lane_stats[r["lane"]]
    z = (r["cos"] - s["mu"]) / s["sd"]

A z of 2.0 means the same thing in all three lanes: strong, for whatever
this is. The lane's absolute scale drops out, and with it the advantage it
never earned.

The epsilon on the denominator isn't superstition. A lane with one document
in it has zero deviation, and without it the z comes back infinite. That
lane existed.

Making it count

The z-score isn't the ranking. It's one signal feeding a larger score, and
how it feeds in is its own decision. Symbol names changed for privacy,
structure as it runs:

const cfg = rules.semantic;                    // { w_sem: 0.4, z_cap: 2.0 }
const map = rules._semantic && rules._semantic.map;

if (cfg && cfg.w_sem && map && item.id) {
  const rec = map[item.id];
  if (rec && Number.isFinite(rec.z)) {
    const zCap  = cfg.z_cap != null ? cfg.z_cap : 2.0;
    const bonus = cfg.w_sem * Math.max(0, Math.min(rec.z, zCap));
    if (bonus > 0) { b += bonus; trace.push(`sem:${rec.lane}`); }
  }
}

Three things there are worth the space.

The clamp starts at zero, so a negative z never subtracts: the semantic
signal can only lift a document, never sink one. A weak semantic match can
mean "bad fit", but it can also mean "this text is written strangely", and I
didn't want the second one burying anything.

The upper cap is duller and more important. A lane with few documents has a
tiny standard deviation, and a tiny deviation produces enormous z-scores.
Without the cap, the emptiest lane wins the entire list — which is the
original bug again, running backwards.

And the whole guard fails open. No cache, no config, or an item with no
embedding, and the branch is skipped and the score comes out identical to
what it was before any of this existed. I was adding a signal to something I
already used daily. The one thing that couldn't happen was a missing file
quietly changing an ordering I'd already come to trust.

The general shape

None of this is about semantic search. It's arithmetic that applies the
moment a score ranks across groups that don't share a scale.

Grades from different teachers. Latency percentiles from services with
different load profiles. Product reviews in categories where people rate
with different harshness. Anything classified by a model that has seen far
more of one kind than another. The number looks comparable because it has
the same name and the same range, and it isn't.

The tell is specific: a whole category that never reaches the top. Poor
performance mixes in. Total absence is structural. When an entire group is
missing from your best-of list, the first thing to suspect isn't the data —
it's the comparison.

The fix is always the same. Normalise within the group, then compare across
groups. Cap the output, because small groups produce extreme values. And
decide deliberately whether the signal is allowed to subtract, or only to
add.

Limits

The mechanism is correct by construction: normalising within a group before
comparing across groups is arithmetic, not a bet. What I don't have is a
measured before-and-after. I didn't freeze a pre-change ranking to compare
against, so what I can claim is that the sparse lane stopped being invisible
— not how much the ordering improved in quality.

Recording the baseline is what I'd do differently. The change took an
afternoon; the measurement that would have proved it needed starting before
I touched anything.

One corpus, one embedding model, three lanes I defined by hand. The lanes
are prose paragraphs, so rewriting one moves its vector and shifts its mean
— the z-score is robust to lane size, not to me editing the description. I
haven't measured how much.

The slowest part wasn't the fix. It was seeing that the sparse lane wasn't
failing.

I'd spent weeks reading that list and assuming a third of the corpus simply
didn't match anything well. The data had been saying otherwise the whole
time. I was sorting by a number that didn't mean the same thing in every
row.

It's no accident that it took so long. A whole category missing reads like a
verdict, not a bug. And as long as it reads like a verdict, you don't go
looking.

git blame Told Me I Wrote 767 Lines I Didn't Write

Lex — Fri, 18 Sep 2026 15:33:18 +0000

git blame Told Me I Wrote 767 Lines I Didn't Write

Before writing about a gate, I checked who wrote it.

The gate is 767 lines of JavaScript that refuses to ship a generated document when the model asserts a number or a tool that isn't backed by the source files. It sits in a document-generation pipeline I run locally — the same source material, retargeted per audience, which is exactly the setup where a model starts filling gaps to fit. I wired the gate into that pipeline four days ago, watched it abort real output the same afternoon, and decided it was worth a write-up.

So I ran git blame. Seven hundred and sixty-seven lines out of seven hundred and sixty-seven: me. Two commits in the log, both mine. Clean history, single author, no ambiguity.

I didn't write a single line of it.

Why the blame lied

The repository is a fork. origin points at an upstream project — MIT licensed, someone else's — that ships an updater for its own system layer. I run a command; it fetches the canonical files and writes them to disk.

That's how this file arrived. When I committed it, git recorded the only thing it can record: who moved the bytes. The commit message says as much — it's an automated system-files update, and nothing else — and the blame still puts my name on all 767 lines.

git blame does not answer "who wrote this". It answers "which commit touched this line last, and who signed that commit". Those are usually the same answer. In a fork with an auto-updater they stop being the same answer, and the blame has no way to tell you that.

Which is a shape I've written about before. A prose note asserting that a setting was off. A blame asserting that I wrote a file. Both authoritative, both mechanical, both wrong — except the note was prose and the blame is a tool, and the tool is far more convincing.

Who did write it

The authorship lives in the upstream history, not mine — named there, not here. Blame the file against origin/main and three contributors come back:

Contributor	Surviving lines
A	301
B	246
C	66

A fourth touched the file early on; none of those lines survive in the current blame.

The shape of that history matters more than the split. The module started as a validator for invented numbers, and was extended weeks later to cover asserted names as well as numbers — the things a document says exist, not just how many. Then came the corrections, and the corrections are the interesting part. One trigger was case-sensitive in a way nobody intended, so a whole class of claims walked past untouched. In several languages the checker silently found nothing at all. One internal heuristic had drifted into answering a different question than the one it was built for. And a pattern was dropping claims on the floor without reporting it. Four people, roughly a month, five separate ways for a fact-checker to be quietly wrong.

I would not have built that. I'd have built the version that works against my own text and fails silently against someone else's — which is the same version, right up until it isn't. The quality in this module isn't in the original design; it's in the corrections. None of the corrections are mine.

What is mine

Here's the part that surprised me more than the blame did: upstream never calls the gate.

The module ships as a library and a CLI, and nothing in the generation path invokes it. The engine existed; the barrier did not. My commit wires it into three call sites along that path — 197 lines, on my branch only, not an ancestor of upstream's main. Before it, the gate was something you could run. After it, the gate is something the output has to get past.

The call site is nine lines. Symbol names changed for privacy, structure untouched:

const result = checkClaims(text, { sources, config, cwd });

if (result.verdict === 'block') {
  throw new Error(`blocked: ${describe(result)}`);
}

No retry at this point, no warning mode, no flag to skip it. The verdict comes back and the document either gets written or it doesn't. Nine lines is the whole of my contribution to the enforcement itself — the other 188 are plumbing: finding the source files, threading the config path, deciding which of three generation steps each check belongs to.

The second thing that's mine is a config file: the allow and deny lists the engine reads. Verified figures the generator is permitted to state, phrases I never want appearing in generated text. The engine is deliberately generic — it knows how to ask whether a claim is supported, and nothing about what it's checking. That file is where it learns.

The honest name for that work is integration, not authorship, and the distinction is worth keeping because the failure modes differ. The authors get it wrong when the gate mis-detects. I get it wrong when the gate doesn't fire. A perfect gate nobody wired is a file.

The first thing it blocked was true

The document was a technical write-up about my own agent system. The gate aborted it, and the reason it gave was a list of four asserted tools it could find no backing for:

planner-worker
orchestrator-executor
artifact-based handoffs
role enforcement

My first read was the obvious one: the model invented four tools. It does that. I've watched it do that. I went and checked anyway, because the blame had just taught me what assuming costs.

It invented nothing. git grep across my agent system returns zero literal matches for all four strings — and every one of the four concepts is real and documented there. Planner and implementer are roles in the roster. Root-driven orchestration is what the architecture became after the orchestrator agent was absorbed into root. Artifact-based handoffs are how deliverables cross between agents: plans, reports, checkpoint files on disk. And role enforcement is a deny-by-default hook — the one I wrote my first article about.

So the model described my architecture correctly, in vocabulary that doesn't appear in the source files it's allowed to draw from. It didn't invent facts. It invented synonyms. The gate cannot tell the difference, because from where the gate stands there is no difference: an assertion with no backing text behind it is an assertion with no backing text behind it.

And the gate was right. A gate that accepts "true, but not in the sources" is a gate that negotiates, and a gate that negotiates isn't a gate — it's a suggestion with extra steps. The cost of blocking fabrications is blocking unreceipted truths. That's the trade, and I'd make it again.

Three days later, it lied properly

A different abort, different class. This time the gate reported an invented figure — a count with no counterpart anywhere in the sources — and added the part that matters: the model had produced the same drift once already, been corrected, and produced it again on the next pass.

No synonym this time. The number came from nothing.

Which sent me looking at how that kind of drift had been handled before the gate existed. I found it in the generation code: a correction written by hand into the prompt, naming one specific figure and the wrong value the model kept reaching for. One fact, pinned in prose, inside the instructions.

It's the same move I wrote about in August — a value living in a prompt string instead of coming from a source — and it's the reason this gate needs to exist at all. That line only ever protected one number. Every other figure the model might drift on had nothing standing behind it until something started checking all of them.

That's a pattern I've hit before. In an earlier piece I described a claim the model kept inverting; fixing the wording at the source reduced the problem and did not remove it, and it took a hard stop downstream to close. Same shape here. A correction is a suggestion the model weighs against the task. A gate is a fact about the world.

Two aborts, three days apart, and the useful thing is that the gate treated them identically. It does not distinguish true from false. It distinguishes supported from unsupported. That makes it blunt in the first case and exactly right in the second — and since it cannot know which case it's in, treating them the same way is not a flaw in the design. It is the design.

Both documents stayed on my machine. Neither had to be caught by a human reading carefully at the wrong hour, which is the review process the gate replaced.

Both stories are the same story

A tool that cannot verify what it asserts will give you the answer with exactly as much confidence as one that can. The blame said I wrote 767 lines. The gate said four real tools weren't real. Neither of them lied. Both of them answered a question slightly different from the one I asked, and neither had any way to flag the gap.

The difference between them is that the gate declares its method. It blocks for want of backing and says so: unsupported, not false. The blame hands you a name and a date and never tells you what it is accountable for. A tool that documents its own limit is usable. A tool that doesn't will mislead you without ever being wrong.

Three pieces in, this keeps being the same lesson at different altitudes. Prose rules don't survive contact with a loop, so enforcement has to live one layer down. Prose that describes configuration is configuration, just without a type checker. And now: tools that summarise the past are making claims about the world, and those claims go stale in exactly the way prose does. git blame is a derived artifact with a fossilised assumption inside it — that whoever committed a line is whoever wrote it.

What I actually do with this: before asserting anything a tool handed me, I ask what question that tool answers, not what question it appears to answer. That's the same move the gate makes on every document it sees. The difference is the gate does it every time, and I have to remember.

Limits

The gate has been wired for four days, and everything above comes out of that window.

Three aborts carry a verbatim message in the logs. Counting the narrative entries in my own daily notes it's around six, but those overlap with the logs and I can't cleanly separate them, so six is a bound and three is the count I'd defend. There is no aggregate counter anywhere. I tallied these by hand, which is its own small irony in a piece about trusting derived artifacts.

The split: two aborts for an unsupported tool, three for an unsupported number. Zero for the blocked-phrase list, which is configured, loaded, and has never once fired. I mention it because it's the part of the gate that has demonstrated nothing, and a reader looking for the weak joint should be handed it rather than have to find it.

One operator, one pipeline, one month of upstream history I didn't write. No control group, and no idea whether any of this survives contact with a setup that isn't mine.

What I'd do differently

The config file — the one that teaches a generic engine which figures are verified — is gitignored. No history. No blame. Not even the wrong blame. If a figure in that allow-list turns out to be stale, nothing on disk can tell me when it went in or what it was checked against.

Which is the thing I wrote about in August: configuration living somewhere no tool would ever look. I published that piece, and roughly a month later populated this file outside version control without noticing. Knowing the failure mode is not immunity from it, and I'd like that to be less funny than it is.

So: version the file, local branch if nothing else, and write down where each allowed figure comes from. Right now an entry asserts that a number is fine and doesn't say why — a frozen value with no provenance, which is precisely the shape that bit me last time.

The open problem is bigger than the file. The gate distinguishes supported from unsupported, which ties it to whatever is already written in the sources. A genuine tool that isn't documented there is indistinguishable, from where the gate stands, from one the model made up. The answer isn't to loosen the gate — it's that the sources are where that gets fixed, which means someone has to keep them current. I haven't solved who, or how often, and four days of data doesn't tell me.

I started by checking who wrote the gate because I was about to assert it in public and I wanted the assertion to be true.

The check took twenty minutes and cost me the article I meant to write.

The gate demands a source before it lets a claim through. I ran the same check on my own. That's the whole of it.

Ten years directing live products before I knew it was called product management

Lex — Fri, 21 Aug 2026 18:29:50 +0000

In 2016 I didn't know what a server was.

I was a kid uploading ARK: Survival Evolved videos to YouTube, and the game — as shipped — was unrecordable. Taming took days. Walking took hours. So I started digging into config files: player speed, taming multipliers, breeding timers. Trial and error, on my own machine, until it was recordable. I didn't know it yet, but that was my first product decision: the game had to fit the format, so I changed the game.

The channel grew. I got invited to multiplayer series with bigger creators. And one day, mid-series, the guy running the server had a falling-out with the owner and walked. Nobody else knew how to keep it alive. I said: I'll run it. I had never run a real server. Three days later the series ran on it. I've said that sentence for every job I've taken since.

The conversation that changed how I quote work

Word spread that I could handle ARK servers. Then Willyrex, a top Spanish-speaking creator, joined a series I was running and asked if I could set up Conan Exiles and GTA V servers too. I had no idea how. I said yes, learned, and delivered.

But that's not why he kept coming back. He told me why: every technical person he'd worked with said yes to everything and then blew the deadline. I did the opposite. When he asked for something enormous, I'd say: "What you actually want for the video is THIS, right? Building it exactly as you describe is madness — weeks of work. But framed this way, you get almost the same result on screen, and you'll have it tomorrow."

He and Vegetta777 loved it. I didn't have a word for it then. Now I do.

That was the start of ten years of commissions: an ARK series with Pokémon-style gyms and creatures (with Staxx in it), servers for Rubius, xFarganx, and others. They came to me with a game and a theme; I owned everything else — research, team, budget, game systems, and enough fresh content that each creator had material every single episode. They kept commissioning me across different games, because what traveled was the execution, not the game.

Shipping on a platform nobody had documented yet

When Epic launched Fortnite Creative, Vegetta had a sponsored campaign with them — he and Fargan needed maps to showcase the new mode. The mode was less than a month old. No Verse, no documentation, no best practices, nothing to copy. I called three friends — Arsilex, Jesusseron, Hernybreak — and we built our first two maps under constraints nobody had mapped yet. The maps shipped and the campaign ran.

Some time later, Vegetta asked me how I'd approach a series around creator-made maps. I told him two things were missing: more people on screen, for dynamics — and maps built expressly for the group, not generic ones. I took care of both. By mid-2019 that group of friends had a name the audience gave us — Los Noobs — and a series on Vegetta's channel, Minecraft con Noobs, alongside Fargan, Elyas360 and Alexby. Vanilla Minecraft, me on camera as a participant, at a time when nobody was making Spanish-speaking Minecraft series.

Running my own platform, on my own payroll

Then I built Tomatecraft, my own Minecraft network. The product decision that made it work: Java (PC) and mobile players could finally play together, when almost no server offered crossplay. I funded it myself, out of a channel that by then had passed half a million subscribers, with programmers, designers and builders on payroll. Later I merged networks with Arsilex and we ran large branded events — up to 20 contributors per event, with Vegetta, Willyrex, Fargan and Rubius joining in.

Every feature I cut, I cut knowing what the payroll cost that month.

Then I stopped for a long while — health comes first. When I came back, I formalized what I'd been doing by instinct and earned a Java certification.

The comeback: 65,000 concurrent players

Years earlier, when Epic ran the Fortnite World Cup and Rubius needed an official map for his qualifier — RubiusTrials — I had pushed hard for two builders I believed in: Hooshen and Iscariote. I didn't build a single brick of it; I just insisted, to Vegetta and Rubius, that they were the right ones. They were.

So when those same two needed someone to lead projects and bring in clients, they called me. I directed the Fall Guys collaboration island (with Vegetta777 and Willyrex): 65,000 concurrent players at peak, the #1 most-played island globally across all of Fortnite for several hours on day 2, and #1 in its category for two weeks — ahead of Epic's own official maps for the event.

Afterwards, with Los Noobs, we shipped island after island — I directed five-plus of them end to end, with a core team of five, and contributed to twenty-plus more in specialist roles: gameplay programming, configuration, 3D modeling, mechanics design, and pricing. We monetized through Epic's engagement payouts, and when in-island micro-purchases arrived, pricing became my most common role across those twenty-plus islands. My last project with the studio was a battle-pass system: VIP pass, level purchases, score multipliers — built as a reusable framework, adopted across 4 maps by the time I left. What I learned pricing them: the hardest purchase is the first one. After that, buying again feels natural — so I priced the first one to be bought.

Today: AI agent systems

Today I build AI agent systems. My 19-agent orchestration went from advisory rules — followed 0 times out of 3 — to deny-by-default enforcement: across 7 runs, 9 blocked tool-calls, 4 allowed only via explicit logged override, 0 unauthorized writes — published with reproducible numbers, extracted into an open-source package (loopward). Different stack. Same rule: the deadline is part of the product.

I never held the PM title. I have the proof.

A Line of Documentation Was Acting as a Global Config Flag

Lex — Fri, 14 Aug 2026 15:16:50 +0000

I spent a morning hunting for a setting that did not exist.

A while back I turned off Claude's co-authorship trailer in my commits — a deliberate choice at the time. Last week I decided I wanted it back. So I went looking for the switch I'd flipped. ~/.claude/settings.json: no key. settings.local.json: no key. ~/.claude.json, parsed as JSON, top-level plus all 38 project entries: no key. The 27 .claude/settings*.json files scattered across my two working drives: no key. Every CLAUDE.md and AGENTS.md I own: no mentions. Environment variables: nothing. A final sweep of my entire user directory — every *.json and *.md — returned three raw hits: a changelog and two copies of an editor extension's JSON schema. The note I wrote when that finished was two words: "Cero hits reales." Zero real hits.

There was exactly one thing anywhere on disk that turned attribution off, and it was a sentence:

~/.claude/rules/ecc/common/git-workflow.md:12
  Note: Attribution disabled globally via ~/.claude/settings.json.

It had a Chinese twin, same file path with zh/ instead of common/, same line 12. Both files are rules files. Rules files get loaded into every session. So every session opened with a line of documentation asserting, flatly and falsely, that a global setting was off — and the model behaved accordingly. The switch I remembered flipping never existed as a switch. The prose was the switch.

The key that sentence gestured at, includeCoAuthoredBy, is deprecated and replaced by attribution. Neither is present in any of my configs, which means the default was active the whole time. The feature was on. Only the description of the world said otherwise, and the description won.

The thesis

Here's what I take from that, and from two more receipts this week: in an agent system, prose and constants are not documentation about the control plane. They are the control plane.

I run a personal multi-agent system on top of Claude Code — 19 specialized agents, root-driven, single-writer, coordinating through on-disk artifacts; a sanitized snapshot is public at multiagent-system-lex. Around it sits a set of always-loaded rules files, and a separate document-generation script I run locally that feeds an LLM a system prompt full of hand-written constraints. Both of those are, in the strict sense, configuration. Neither is stored anywhere a linter, a schema, or a test would ever look at it.

That's the mechanism. A rules file is loaded unconditionally into every session, so a claim inside it has the reach of a global setting with none of the accountability: no schema, no type, no default, no git blame if the directory isn't a repo, nothing downstream that notices it disagrees with reality. The model doesn't verify the claim against the system, because from inside the loop the claim is the system.

The constant that outlived its own source

Second receipt, same week, different shape. In that document-generation script, the system prompt carried a block of figures marked FROZEN — meaning: reproduce exactly, never paraphrase, never invent. One of them was a subscriber count, hardcoded as a literal in the prompt string. It was wrong: 617K, when the real figure is 577K.

The interesting part is the ordering:

When	Commit	What
2026-08-10 14:04:31	`fc580b2`	removes the frozen `617K` from the system prompt
2026-08-10 21:28:35	`69e0499`	corrects one remaining profile document, `617.000` → `577.000`
2026-08-14 11:02:19	`102f63e`	a generated artifact is still emitting `617K`, in three separate places

The structured sources were already right. The commit message for 69e0499 says so in as many words: "la cifra canonica es ~577.000" — the canonical figure is ~577,000 — and names the two source files that already held it. The only thing carrying 617K was a literal sitting inside a prose instruction, wearing the word FROZEN.

And FROZEN is exactly why it won. That annotation exists to stop the model from softening or improvising a number. It did its job perfectly — on the wrong value. A freeze marker is authority, and I had handed authority to a copy instead of a source. The fix was one line in the doc comment and one in the prompt rule: the count is no longer hardcoded there at all, it comes from the profile facts or it doesn't appear. Four days later I found the stale value still sitting in a generated artifact downstream, because generated output doesn't retroactively fix itself when you fix its generator.

The mirror: a prose rule that held

I want to be fair to prose, because one of my prose rules worked, and why it worked is the whole point.

When I wrote the first article, the brief asked for a specific external citation — a multi-agent failure taxonomy, with two percentages attached. It also carried a standing rule: use only data present in a specific on-disk notes file, with its source; anything unsourced doesn't go in. The figures weren't in that file. The plan recorded the refusal explicitly — "NO está en el fichero → NO se usa", not in the file, so it isn't used — and moved the citation to a forbidden list. The published piece contains zero references to it.

That rule held where the attribution note failed, and the difference isn't discipline. It's that "is this string present in that file?" is a question with a mechanical answer. The rule wasn't asking anyone to be careful; it was asking a question I could go check, and the check was cheap enough to actually run. Prose that resolves to a disk lookup behaves like code. Prose that asserts a fact about the world behaves like a lie waiting to be believed.

Fixing the prose is not enough

Fourth receipt, and it's the one that connects back.

In that same document-generation script, the model kept inverting a claim from the first article — writing that "nine unauthorized writes were blocked", when what was blocked were nine tool-calls and unauthorized writes were zero. Same numbers, reversed meaning, and the reversed version reads better, which is precisely the problem. I fixed the wording at the source. The commit records what happened next, with one noun swapped for anonymity: "Fixing the source cut this from 4 generated documents to 1; the model still paraphrased its way back, so it needed a hard stop." The hard stop was a denylist on the inverted phrasing, with tests in both directions so the correct claim stays legal.

Four to one, not four to zero. A sibling change the same day says it from another angle: a gold example meant to fix the output's register was already in the prompt and didn't prevent the failure — "the lever is explicit rules plus a mechanical constraint, not more example."

The first article argued that if a boundary matters, it has to live in the substrate the agent can't reason its way around — the tool-call layer, not prose the model weighs against its objective. That enforcement layer is packaged as loopward. This piece is the other half: the prose is substrate. Just substrate with no type checker, no default, and no test. It needs the same treatment — single-sourced values, mechanical checks, provenance — and when it can't get it, a hard stop downstream.

Limits

Four incidents, one operator, one system, one week. That's an anecdote with receipts, not evidence of a law. Everything above is reconstructed from commits and session transcripts I can point at, and quoted rather than paraphrased, but there's no control group here and no idea how any of it generalizes to a system that isn't mine.

The sharpest limit is in the first story. I can't tell you when that note was introduced, because ~/.claude is not a git repository — git rev-parse there returns "fatal: not a git repository", and the trail ends there. A line of text stood in for a config flag and I can't date it, so the audit stops at "undated". I'd rather say that plainly than round it to a date.

What I'd do differently

Put ~/.claude under version control, first thing — the rules directory is configuration with production effects and it deserves a history. Single-source every constant that appears in a prompt; if a figure lives in a YAML file, the prompt gets a reference, never a copy, and FROZEN marks the rule, never the value. Write prose rules that resolve to a disk lookup wherever the choice exists, because those are the ones that survive contact with a loop. And treat a fix at the source as step one of two: generated artifacts don't self-heal, so something has to go find the stale copies.

The uncomfortable version: I've been writing configuration in English and grading it as documentation. It was never documentation. It ran.

What Advisory Rules Actually Do in an Agent Loop

Lex — Thu, 06 Aug 2026 10:05:43 +0000

Advisory rules in my 19-agent system produced 0/3 compliance in the three runs right after the rule was canonized — the hook warned, the agent proceeded anyway. I replaced the warning with a deny-by-default gate plus a one-shot, logged override: over two months, 9 blocked tool-calls, 4 authorized overrides, 0 unauthorized writes. Small, well-delimited evidence, reproducible by script — but the direction is clear: enforcement has to live in the substrate, not in prose the model weighs against its objective.

The hook fired. My root agent read the warning telling it that plan.md belongs to the planner subagent, acknowledged it, and wrote the file anyway. Then it did it again on the very next step. The audit report for that run records it drily:

"plan.md root direct ×2 (step 1 + 2), planner NO invocado. hook PreToolUse advisory flagged 2x consecutivos; contenido aprobado verbatim inline → root procede."

Root authored the planning file directly, twice; the advisory hook flagged it twice in a row; root proceeded anyway, logging "aprobado verbatim inline" — approved verbatim inline — as its justification. Two runs later, a third instance with exactly the same shape: "plan.md root direct 3ª instancia consecutiva. Hook PreToolUse advisory disparó + root ack con razón 'aprobado verbatim inline'."

That third one is what stopped me. I was reading the audit reports as they came in, expecting the new rule to have changed something — instead I was watching my own system politely log the same violation, run after run.

Those were the three runs immediately after I canonized the rule. The hook fired every single time. Compliance: 0/3.

The system

Some context for that scene. I run a personal multi-agent system on top of Claude Code: 19 specialized agents — planners, implementers, reviewers, domain specialists, independent auditors — under a root-driven, single-writer architecture. One main thread is the only physical invoker of subagents, and coordination happens through on-disk artifacts rather than shared mutable state: plans, hypothesis files, reports. The system has accumulated 171 documented runs, each with its own directory, plan, and audit trail. That number is scale and nothing more — raw transcripts from the early era are gone to retention, so I cannot measure rule compliance across all of it, and I won't pretend otherwise. The piece of the division of labor that matters for this story: the planner subagent owns plan.md. Root coordinates, but it is not supposed to author the plan itself. A sanitized snapshot of the whole thing is public at github.com/lexosi/multiagent-system-lex.

Why advisory rules fail

Here's the mistake I made first, and I don't think it's an exotic one: I wrote the rule down and assumed that was enforcement. The advisory hook was technically real — a PreToolUse hook, introduced 2026-05-21, that fired on every root write into planner territory. But it always exited 0. It warned; it never blocked. Five days later, on 2026-05-26, I canonized the rule in the system's constitution: root must not author plan.md; the planner owns it. The three runs I opened with are the three runs immediately after that canonization.

A warning is just more context. The model isn't being malicious when it walks past one; it's optimizing for the task in front of it. When a soft rule competes with the immediate objective — produce the plan, keep the run moving, the content is already approved anyway — the rule loses. In my data, it lost every time. The failure mode isn't disobedience; it's incentives. The path of least resistance runs straight through the warning, and "approved verbatim inline" isn't a rebellion — it's a locally reasonable justification generated by a system that was never actually stopped. Anthropomorphizing this as the agent "ignoring the rules" misses the mechanism and, worse, suggests the wrong fix: a sterner warning is still a warning.

Once I saw the pattern for what it was, I stopped tweaking the wording and went looking for where enforcement could actually live — somewhere the model couldn't weigh it against the task. The answer was one layer down.

The cutover

Six days after canonizing the rule, the warning was gone. On 2026-06-01 (commit b3efe00) the advisory hook was deleted and replaced by a deny-by-default PreToolUse hook: any write into a guarded territory returns permissionDecision=deny unless the caller is the owning subagent — the hook discriminates callers by agent_type — or a valid one-shot sentinel file authorizes exactly one override, with a short TTL. The previous design's session-level environment-variable escape hatch died in the same commit: an override that lives for a whole session is an override that outlives its justification.

The numbers, verbatim from the eval report — which is regenerated by script, not written by hand:

Advisory compliance after the rule was canonized (2026-05-26, plan.md territory): 0/3 runs.
Hard-block era (2026-06-02..2026-08-04): 9 blocked tool-calls (attempted root writes to planner-owned files), across 7 runs.
4 writes authorized via the one-shot sentinel — each logged with a non-empty reason.
Unauthorized writes to guarded territories: 0.

I want to label this evidence carefully, because the labeling is half the point. The advisory side is a natural experiment, n=3: three consecutive runs immediately after a single dated rule change, one territory, with frozen citations into on-disk audit reports that the eval script re-verifies. The hard-block side is a mechanical log: an append-only deny-ledger covering two months, parsed by script, no human or agent judgment in the count. And the honest limit: the hook logs denials and overrides, not allows — so "0 unauthorized writes" is an invariant by construction (the hook denies guarded territories unless the caller is the owner or consumes a valid sentinel), corroborated by the logs, not an exhaustive census of every write that ever happened. This is not a large-scale study and I'm not dressing it up as one. It is a small, well-delimited experiment that anyone can re-run against the repository with python eval/run_eval.py.

Deny, deny, override — as designed

The pattern I care most about in the ledger is deny → deny → override. Root tries to write a guarded file: denied. Tries again: denied. Then one of two things happens. Either root routes the work to the owning subagent — which is the intended outcome, and the common one — or I look at the situation, decide the exception is legitimate, and drop a sentinel file that authorizes exactly one write, which the hook consumes and logs. Four of those in two months. All deliberate, all recorded, all with a stated reason.

That flow is the design, not a failure of it. A hard block with no exit teaches the system — and its operator — to fight the wall, and fighting the wall is where the really creative workarounds come from. What I actually want is a gate: violations impossible by default, exceptions possible, authorized, and logged. The one-shot property is what keeps it honest. A sentinel authorizes one write, not a mood.

Where this fits

The reason I think this matters beyond my setup: everyone is building loops right now. Boris Cherny, head of Claude Code at Anthropic, said it on stage in June 2026: "I don't prompt Claude anymore. […] My job is to write loops" (Acquired Unplugged) — outer programs that invoke the agent loop as a subroutine and decide what happens next. And the recent evidence says those loops need harder gates than most of us are giving them. SlopCodeBench (UW-Madison/MIT, March 2026) had agents extend their own code under changing specs and found structural quality eroding iteration after iteration even while core tests kept passing — best strict rate 17.2%, agent code 2.2× more verbose than comparable human repositories. LoopsBench (Microsoft, August 2026) found the best outer-loop configuration solved 25.00% of its tasks, with visible regression events across every loop profile it measured — though it's days old and unreplicated, so hold that one loosely.

The shared reading of both: objective verifiers like tests are necessary but insufficient — what passes the gate can still rot. My result is a small datapoint in the same direction, one layer down. Even the process rules — who is allowed to write what — don't hold as good-faith conventions inside a loop. If a boundary matters, it has to live in the substrate the agent can't reason its way around: the tool-call layer, before the write happens, not prose the model weighs against its objective and discounts.

Touch it

I extracted the enforcement layer into a standalone package, loopward: a reliability layer for multi-agent systems — anti-loop limits, human stop-gates, per-run audit trails — enforced structurally rather than by convention. pip install git+https://github.com/lexosi/loopward.git, then loopward-demo runs a deterministic offline demo with zero API keys; this isn't an announcement, it's the touchable version of this post.

Enforcement by construction, not good faith

That's the one-line version of two months of logs. Rules the agent can weigh get weighed away; gates the agent cannot pass produce routing instead of violations, and the exceptions become visible, deliberate acts instead of silent drift.

What I'd do differently, and what's still open — because a write-up that ends in triumph is usually hiding the interesting part. I'd log allow decisions from day one, so "0 unauthorized" could be a census instead of an invariant plus corroboration. I'd keep raw transcripts — retention ate the advisory era, and n=3 is what survived, not what I designed. I wouldn't canonize a rule five days before I could enforce it. And open questions remain: n=3 is small even if it's clean; plan.md is one territory, and the same claim for the others is only partially measured; and I don't yet know how the gate scales as guarded territories multiply, because every new wall adds override friction and friction is a budget you spend from. What I do know is that the direction has stopped feeling arguable: if I need an agent to respect a boundary, I don't ask. I make the boundary the default, and the exception a logged, deliberate act.