DEV Community: John

Stop Hooks as Hard Constraints: Enforcing Claude Code Behavior Outside the Model

John — Mon, 13 Jul 2026 09:00:06 +0000

Originally published on hexisteme notes.

I run a fleet of Claude Code agents, and for a long time I tried to enforce behavioral rules the obvious way: write them into the system prompt and hope they hold. They mostly did — until pressure showed up. A plausible-sounding challenge from me, a long task, a shortcut that saves a turn, and the model would quietly drop a rule it had followed all session. Prompt instructions are soft constraints. They shape behavior probabilistically, and when they conflict with RLHF-trained patterns — deference to the user, hedged language, listing options instead of committing to one — the RLHF training usually wins, especially under pressure.

Stop hooks are what I actually rely on now. They're hard constraints: shell scripts that run after Claude Code generates a response and before I see it, and they can force a revision the model can't talk its way around.

A stop hook is a shell script registered in settings.json under a Stop event. It runs after every response, receives the full session as JSON on stdin, and either exits 0 (allow it through) or exits 2 with a message (block it and force a revision Claude must address). Because it runs outside the model, it's enforced, not suggested.

How stop hooks actually work

The registration itself is small:

// ~/.claude/settings.json
{
  "hooks": {
    "Stop": [
      {
        "matcher": "",
        "hooks": [{
          "type": "command",
          "command": "~/.claude/hooks/stop_combined.sh"
        }]
      }
    ]
  }
}

The script receives the full session context as JSON on stdin and can:

Exit 0 — allow the response through
Exit 2 — block it and inject a correction message Claude must address

The correction message comes back as feedback, and Claude generates a new response. I only ever see the final, corrected output — the hook's intervention is invisible unless I go looking for it in the transcript.

The AND-gate pattern

A hook that fires on a regex match alone has an obvious failure mode: false positives. A rule like "block responses that ask the user to decide" will also block a legitimate clarifying question like "what's your timeline for this?" That's not a failure mode I can tolerate in something that blocks output on every turn.

The fix is to require two independent conditions before firing:

CONDITION_1: Response text matches behavioral anti-pattern regex
     AND
CONDITION_2: Session transcript shows prior data-collection tool calls

# Only fire if BOTH are true
if [[ $RESPONSE_MATCHES == "1" && $HAD_TOOL_CALLS == "1" ]]; then
    block_with_message
fi

The tool-call evidence half is what makes this work. If Claude asks a clarifying question without having done any analysis first, that's a legitimate question — let it through. If it's deferring back to me after running searches and reading files, it already has what it needs and is offloading work it should be finishing itself.

Production hook 1: decision-ownership enforcement

The failure mode here is specific: RLHF trains models to hand judgment back to the user. "Which approach would you prefer?" after already analyzing both options. "It depends on your priorities" after having every piece of information needed to just recommend one. That isn't politeness — it's transferring cognitive labor that was the whole point of asking in the first place.

The hook, stop_decision_ownership_check.sh, works like this (condensed from the production script):

#!/bin/bash
# Read session JSON from stdin
SESSION=$(cat)

# Extract last response text
RESPONSE=$(echo "$SESSION" | python3 -c "
import json,sys
d=json.load(sys.stdin)
msgs = d.get('messages', [])
for m in reversed(msgs):
    if m.get('role') == 'assistant':
        content = m.get('content', '')
        if isinstance(content, list):
            print(' '.join(b.get('text','') for b in content if b.get('type')=='text'))
        else:
            print(content)
        break
")

# Check for deference patterns
DEFER_REGEX='(어느.*(좋아|나을|선택)|당신이.*(정하|결정)|뭐가.*좋을|what.*prefer|which.*would you|up to you|your call|depends on your)'
if ! echo "$RESPONSE" | grep -qiE "$DEFER_REGEX"; then
    exit 0  # No match, allow
fi

# Check for prior tool evidence (AND gate)
TOOL_CALLS=$(echo "$SESSION" | python3 -c "
import json,sys
d=json.load(sys.stdin)
count = sum(1 for m in d.get('messages',[])
            for b in (m.get('content',[]) if isinstance(m.get('content'), list) else [])
            if isinstance(b,dict) and b.get('type') in ('tool_use','tool_result'))
print(count)
" 2>/dev/null || echo "0")

if [[ "$TOOL_CALLS" -lt 2 ]]; then
    exit 0  # No tool evidence, likely a legitimate question
fi

# Nag-once: compute hash, check if already nag'd
HASH=$(echo "$RESPONSE" | shasum -a 256 | cut -c1-16)
NAG_FILE="/tmp/.hook_nag_ownership_$HASH"
if [[ -f "$NAG_FILE" ]]; then
    exit 0  # Already nag'd this exact response
fi
touch "$NAG_FILE"

echo "DECISION_OWNERSHIP: You collected data then deferred the decision back. Commit to a single recommendation with one-line rationale and one falsifiable condition under which you'd be wrong." >&2
exit 2

The regex mixes Korean and English deference markers on purpose — I work across both languages, and the pattern has to catch either.

Production hook 2: sycophancy prevention

The second failure mode is worse, because it's invisible in the output. When challenged, Claude reverses its position at a 98% rate — that's from Anthropic's own published sycophancy research (arXiv 2310.13548). Not because the challenge supplied new evidence, but because RLHF rewarded agreement. Claude doesn't announce the capitulation ("you're right, I was wrong") — it just quietly shifts.

stop_challenge_reverify.sh catches the pattern with a three-part gate: challenge, capitulation, and the absence of any actual verification.

# Gate: (challenge in user message) AND (capitulation in response) AND (no verification evidence)
# USER_MSG and RESPONSE are pulled from the session transcript the same way hook 1 extracts RESPONSE

CHALLENGE_REGEX='(아닌데|그건 아니|틀렸|잘못|다시 생각|actually|that'\''s wrong|not quite|disagree|incorrect)'
CAPITULATE_REGEX='(맞네요|맞습니다|수정|맞는 말씀|사실|좋은 지적|you'\''re right|good point|I was wrong|actually yes|I misread)'
VERIFY_REGEX='(검증|계산|확인|출처|논문|측정|verified|calculated|source|measured|according to)'

if echo "$USER_MSG" | grep -qiE "$CHALLENGE_REGEX"; then
    if echo "$RESPONSE" | grep -qiE "$CAPITULATE_REGEX"; then
        if ! echo "$RESPONSE" | grep -qiE "$VERIFY_REGEX"; then
            echo "CHALLENGE_REVERIFY: Position changed without new evidence. Choose:
HOLD — restate original position with grounds
CHANGE — cite specific new evidence or calculation that overrides original" >&2
            exit 2
        fi
    fi
fi

This hook has a deliberate blind spot: it can't catch silent capitulation, where the tone shifts without any explicit reversal language. I've accepted that trade-off — the priority is keeping false positives low on legitimate corrections, not catching every possible way a model can cave.

The nag-once pattern

Blocking indefinitely just creates a new failure: Claude keeps generating blocked responses and I get no output at all. Nag-once breaks that loop:

First violation — compute a sha256 of the response, write it to /tmp/.hook_nag_{hash}, block with a message
Second attempt with the identical response — the hash file already exists, so exit 0 and let it through
A genuinely different response — new hash, new nag file, block again if it's still violating

This surfaces the issue exactly once and lets the conversation keep moving. I can still push back on the hook's correction if I disagree with it; that's fine — the hook's job is to force the question onto the table, not to win it.

What hooks can't do

Hooks operate on the final response text. They can't see the model's reasoning before it outputs anything, can't inject into the middle of a generation, and can't stop the model from starting down a bad path — only from completing it. Deeper behavioral constraints still have to live at the system-prompt and instruction level; hooks are the last line of enforcement, not the only one.

Combining hooks

I run several gates through one stop_combined.sh that calls each in sequence and returns on the first block — I don't want two hooks piling correction messages on top of each other. Ordering matters: the highest-priority gates run first.

#!/bin/bash
# stop_combined.sh — ordered gate sequence
SESSION=$(cat)

# Gate 1: Decision ownership (highest priority)
RESULT=$(echo "$SESSION" | ~/.claude/hooks/stop_decision_ownership_check.sh 2>&1)
if [[ $? -eq 2 ]]; then echo "$RESULT" >&2; exit 2; fi

# Gate 2: Sycophancy / challenge reverify
RESULT=$(echo "$SESSION" | ~/.claude/hooks/stop_challenge_reverify.sh 2>&1)
if [[ $? -eq 2 ]]; then echo "$RESULT" >&2; exit 2; fi

exit 0

None of these hooks make the model smarter. What they do is close the gap between the behavior I actually want and the behavior RLHF defaults to under pressure, by moving enforcement outside the place where that pressure operates.

More notes at hexisteme.github.io/notes.

Why I Rejected an Event Bus for My Solo Agent Fleet: State Is Truth, Events Are Rumors

John — Mon, 13 Jul 2026 00:00:05 +0000

Originally published on hexisteme notes.

I run a personal fleet on one machine: a handful of small agent CLIs, a pile of cron jobs and LaunchAgents, and several MCP servers, some of them third-party tools I didn't write and can't modify. At some point I wanted a single inbox that could answer one question — what needs my attention right now? New outputs I haven't read yet. Decisions only a human can close. Dependencies that broke overnight without telling anyone.

The obvious architecture for that is an event bus. I sketched it, then rejected it on purpose and built a pull-based design instead. The reasoning generalizes past my particular setup, so here it is.

For a small, high-churn fleet you don't fully control, pull (scan on-disk state) beats push (an event bus). An event bus needs every producer — including third-party tools you can't modify — to emit; anything that doesn't is silently invisible. A poller that reads on-disk state instead — file mtimes, health checks, process liveness — is self-healing: it reflects reality even for components that never report anything, and a new tool shows up the moment it writes a file, with zero instrumentation.

The architecture I almost built

The event-bus version looks clean on a whiteboard. Every component emits events — job.finished, output.created, decision.pending — onto an append-only log. The inbox is just a reader of that log. Closing an item writes a close-event that advances whatever comes next. It's the textbook "single source of truth" pattern, and it's the first thing most people reach for.

I killed it for four reasons.

Reason 1: the instrumentation tax is a project killer

Push means every producer has to emit. In a fleet that grows most weeks, that's a standing tax on every new agent, every new cron line, and — the one that actually killed the design — every third-party MCP server I didn't write. You cannot add an emit() call to a server someone else built. So the moment I add a component and forget to instrument it, or simply can't, it goes invisible in the inbox.

That's the trap: the components you least control — third-party tools, things that crash early — are exactly the ones an event bus can't see. Push optimizes for the easy case (code you own) and silently fails the hard case (code you don't). An observability layer whose blind spots grow with the system is worse than no observability layer at all, because it still looks authoritative while quietly lying.

Reason 2: state is truth, events are rumors

An event is a claim that depends on the claimant surviving long enough to make it. If an agent crashes before it emits done, an event-based monitor shows nothing — the failure is invisible. State is the evidence left behind regardless of whether anyone remembered to report it: a stale output file, a log that stopped growing, a health check that fails, a process that just isn't there anymore.

# pull: derive status from evidence the component already leaves behind
status = {
    "alive":   process_is_running(job),               # ps / launchctl print
    "fresh":   newest_output_mtime(job) > expected,    # <project>/report/ glob
    "healthy": health_check(dependency),               # poll endpoint
}
# no emit() anywhere — a component that never reports is still seen

State is truth, events are rumors. An event only exists if something remembered to send it; state is evidence left behind regardless. A pull monitor reads that evidence and reflects a crash without needing the agent's cooperation — which is what makes it self-healing, converging on reality every cycle, while push is only ever as honest as its least-reliable emitter.

Reason 3: orchestration schizophrenia

A closed-loop bus, where "close this item" emits an event that advances the next step, quietly turns the inbox into a workflow engine. I already run an orchestration hub that decomposes goals into steps. Building a second one inside the monitor duplicates that responsibility and doubles the debugging surface — now a stuck task could be the hub's fault or the inbox's, and I have to check both. A monitor should report state, not drive it. Keeping those two jobs in two separate systems is what keeps either one debuggable.

Reason 4: the bus itself becomes an unreviewed dump

An append-only event log isn't free infrastructure. It accrues schema drift — the shape of output.created changes and old readers break — plus duplicate events, events nobody ever closes, and unbounded growth that eventually demands compaction. I'd be adding a database with none of a database's guarantees, and it would need its own monitoring just to know if it was healthy. Pull doesn't produce that artifact: there's nothing to compact, because there's nothing stored beyond the state that already exists on disk.

What pull looks like instead

The inbox ends up as a computed view over state that already exists — no new store, no emit() calls anywhere:

Attention type	Derived from (pull)
New / unread output	per-job output glob mtime vs. a read-timestamp record
Pending decision	existing on-disk sources — an attention scan's output, an undecided ledger entry, an expired evaluation date
Broken dependency	health-check failure, propagated to anything that declares a dependency on it

Priority is deterministic, not a model's guess: BLOCKED → STALE → NEW, where each is a hard fact — a failed health check, a file newer than its read-timestamp, a schedule past due. An LLM-generated priority number would be unfalsifiable and would drift over time; a deterministic trigger is reproducible and debuggable. I keep the model out of the ranking entirely.

Freshness is "unread," not "old"

Pull also fixed a metric I had wrong. My first instinct for "freshness" was elapsed time — this ran three days ago, so it's stale. But age isn't actually the problem; unread output is. A report that finished an hour ago and that I haven't opened yet demands more attention than one from last week that I already read.

So freshness is computed as a join: does an output exist whose mtime is newer than the last time I opened it? Clicking a card records a read-timestamp; unread items rise to the top, read ones sink. This join is cheap only because the design already scans state for everything else — freshness falls out of the same glob. In a push system it would have been yet another event to emit and reconcile.

The boundary that keeps it honest

Choosing pull forces a discipline I've come to think is the actual point: the monitor must never mutate fleet state. "Closing" an inbox item means acknowledge, and deep-link to the real place the work gets closed — it does not reach in and change a job, a ledger, or an agent's state directly. The moment a monitor starts writing back, it's an orchestrator again, and reasons 1 through 3 all come back. Read the world, link to the controls, never become the controls.

When push is actually right

None of this means event buses are a bad idea in general — it means they fit a different shape of problem. If you own every producer, can instrument all of them, and need high-throughput, low-latency fan-out, push is the right tool for that job. The pull argument wins specifically for a small, heterogeneous, high-churn fleet with components you don't fully control, where the dominant cost is instrumentation and the failure that hurts most is the silent one. The question isn't "push or pull" in the abstract — it's which cost is fatal for your system: throughput, or blind spots.

More notes at hexisteme.github.io/notes.

When a Site Blocks Your Scraper, Read the Browser Tab You Already Have Open

John — Sun, 12 Jul 2026 00:00:05 +0000

Originally published on hexisteme notes.

I needed live listing prices from a property portal for a personal buy-decision tool. The portal does everything a site does to stop a scraper: it disallows the paths in robots.txt, it fingerprints headless clients, it hides behind a login, and the prices don't even exist in the fetched HTML — they're painted in by JavaScript after the page loads. I spent an afternoon losing to all four defenses. Then I stopped fighting them, because the data was already sitting on my screen: I had the page open in Chrome.

For a personal tool, the move past robots.txt, bot detection, login walls, and JS-rendered single-page apps is not a better headless scraper — it's a ~40-line AppleScript bridge that reads the DOM out of the authenticated Chrome tab you already have open. You inherit the human's session and the fully-rendered content, and you add zero HTTP requests, so there is no scraper for the site to detect. It sidesteps all four walls at once because a human already solved the hard parts. It is also strictly a solo, read-only, human-scale technique — and that boundary is the point, not a limitation to route around.

The wall: the data is behind everything that stops a scraper

Every defense a site can raise was stacked on this one. A plain HTTP fetch of the listing URL got redirected to a login page. Sending cookies got past that, until the anti-bot layer noticed a client with no browser fingerprint and started serving challenges. And even a "successful" fetch was useless, because the response was an almost-empty HTML shell — the actual numbers are rendered client-side by the app's JavaScript, so the price I wanted simply wasn't in the bytes I'd downloaded. Each of these is solvable on its own with enough effort (headless browser, session replay, a rendering engine), and stacking all the solutions together is how a weekend project becomes an unmaintained pile of anti-detection hacks that breaks every time the site ships.

The tell that I was solving the wrong problem: I was reconstructing, badly, a thing that already existed and worked perfectly. A logged-in browser session that renders the page and shows me the price. I had one open in front of me.

The move: you already have the page open

The reframe is small and it changes everything. Don't fetch the page — read the page you already fetched by opening it. When the portal is open in a Chrome tab, that tab has already passed the login, already survived the bot check (it's a real browser doing real navigation), and already run the JavaScript that paints the prices into the DOM. The numbers are right there in document.body.innerText. The only thing missing is a way to reach into that tab from a script.

On macOS, that way is Apple Events. Chrome can evaluate JavaScript inside any open tab on request (a one-time setting, View → Developer → Allow JavaScript from Apple Events), and a JXA script — JavaScript for Automation, run via osascript — can drive it. The whole bridge is about forty lines: enumerate the windows and tabs, find the first tab whose URL contains a substring you pass in, and evaluate an expression inside it.

// read-tab.js  —  osascript -l JavaScript read-tab.js "new.land.naver.com" "document.body.innerText"
function run(argv) {
  const [pattern, jsExpr = "document.body.innerText"] = argv;
  const Chrome = Application("Google Chrome");
  let tab = null;
  for (const w of Chrome.windows())
    for (const t of w.tabs())
      if (t.url().indexOf(pattern) !== -1) { tab = t; break; }
  if (!tab) throw new Error("no open tab matching: " + pattern);

  // Evaluate inside the live, authenticated, already-rendered tab.
  const wrapped =
    "(function(){try{const v=(" + jsExpr + ");" +
    "return typeof v===\"string\"?v:JSON.stringify(v);}" +
    "catch(e){return \"JS_ERROR: \"+(e&&e.message||e);}})()";
  return tab.execute({ javascript: wrapped });
}

That's the entire mechanism. Point it at a URL substring, hand it any expression, and it returns the result as text. Pass document.body.innerText for the rendered page, a specific querySelectorAll(...) to pull just the fields you want, or document.documentElement.outerHTML to get the post-render HTML for a downstream parser. From a Python collector it's one subprocess call; the deterministic pipeline downstream never knows the data came from a browser tab instead of an API.

Why it sidesteps every wall at once

The reason this feels like cheating is that it doesn't defeat any of the defenses — it makes all of them irrelevant, because the human already dealt with each one before the script ever runs.

Wall	Why it's already gone
Login / session	You read the tab under the user's authenticated session — no credentials in your code, nothing to store or refresh.
JS-rendered SPA	You read the DOM after the app's JavaScript ran, so you see the painted numbers, not the empty shell a fetch returns.
Bot detection	You issue no HTTP request at all. There is no headless client to fingerprint; the traffic already happened, made by a real browser a human drove.
`robots.txt` / rate limits	You aren't crawling. You read one page the user chose to open, at the rate a person clicks — the politest possible traffic profile.

Every one of those problems is hard to solve in a scraper and free to inherit from a browser. That's the whole trade: you give up automation-at-scale and get correctness-without-a-fight.

The honest edges

This is a sharp tool with a narrow blade, and pretending otherwise is how you get burned.

Reading is reliable; writing is a different, harder problem. It's tempting to also fill search boxes or click through pagination the same way. Don't assume it'll work: Apple Events JavaScript runs in an isolated world, and modern frameworks override the native value setter, so a plain el.value = "..." is silently ignored and no input/change event fires — the app never sees your input. Writing has to go through the native value setter plus manually dispatched events, and it's fragile enough that I keep the reader strictly read-only and treat interaction as a separate concern.
It is single-user, single-machine, by design. This works because a person has the page open on their desktop. It does not run in the cloud, does not scale, and is not a commercial scraper. If you need server-side or high-volume collection, this isn't your tool — and you should be honoring the site's terms and rate limits, not looking for a bypass.
It's coupled to the browser being open and permissioned. The one-time Apple Events permission, the specific browser, the tab actually being open — these are real dependencies. The script fails loudly ("no open tab matching…") when they're not met, which is the right failure: it tells you to go open the page, not silently return stale or empty data.

## When to reach for it

The fit is narrow and clear: a personal, low-volume tool that needs data from a site that fights scrapers, where you are already an authenticated human looking at the page. Live prices for your own buy decision, a number off a dashboard you have access to, a value from a portal you log into anyway. In that slot it's unbeatable, because you're not extracting anything you couldn't already see — you're just saving yourself the copy-paste. Outside that slot — scale, headless, other people's data, evading a wall you're not entitled to be past — it's the wrong tool, and the boundary isn't a limitation to engineer around. It's what keeps the technique honest: it only ever reads a page a person chose to open.

More notes at hexisteme.github.io/notes.

The Silent 10 Tax: How a Nondeterministic System Prompt Voids Your LLM Prompt Cache

John — Sat, 11 Jul 2026 00:00:06 +0000

Originally published on hexisteme notes.

Prompt caching is the largest single token lever for any long-lived LLM session — a cached token is served at roughly a tenth of its normal price. It is also one of the easiest things to break by accident, because when it breaks there is no error. One timestamp in the system prompt, one unsorted JSON key, and your cache-read rate quietly goes to zero while the bill goes up tenfold. This is how the failure works, and how to make it loud.

A prompt cache is a prefix match: the cache key is the exact bytes of the rendered prompt up to a breakpoint, and a single byte change anywhere in the prefix invalidates everything after it. So anything volatile near the front of the prompt — a datetime.now() in the system header, a per-request UUID, a dict serialized without sorted keys, a tool list that reorders — silently defeats caching for the entire request. The cost is real (a miss is ~10× a hit on the cached span) and invisible (no exception, no warning). The only signal is one field in the usage object, and the only durable fix is to assert on it.

How prefix caching actually works

The mental model that prevents every bug below is one sentence: the cache key is the bytes of the rendered prompt, matched as a prefix. The provider renders your request in a fixed order — tools, then the system prompt, then the messages — hashes the bytes up to each cache breakpoint, and looks for a prior entry with the same prefix. If it finds one, the shared span is a cache read; the rest is processed fresh. If the very first byte differs, nothing matches and the whole thing is processed at full price.

Three properties follow, and all three are load-bearing:

Prefix, not fuzzy match. There is no "close enough." Byte N changing invalidates the cache for every breakpoint at position ≥ N. Stable content has to physically come before volatile content in the rendered prompt, or the volatile content poisons everything after it.
Model-scoped. Caches are keyed per model. Switching the model mid-session — even to a cheaper one for a sub-task — starts from a cold cache.
Short-lived. On Anthropic's API the default cache entry lives ~5 minutes (a 1-hour tier exists at a higher write premium). A gap longer than the TTL means the next request pays to write the cache again.

The economics are what make this worth caring about. A cache read costs about 0.1× the base input price; a cache write costs about 1.25× for the 5-minute TTL. So the break-even is fast — two requests sharing a prefix already come out ahead — and the downside of a silent miss is steep: every token you could have read at one-tenth price is instead reprocessed at full price. That is the 10× in the title, and on a long agent loop that resends a large fixed preamble every turn, it is most of the bill.

The silent failure: `cache_read` stays zero and nothing tells you

Here is what makes this class of bug nasty. A broken cache is not an error. Your requests succeed, your outputs are correct, your latency is a little worse and your cost is a lot worse, and nothing in the response says "you just paid full price for 40,000 tokens you had cached thirty seconds ago." The system is behaving exactly as designed; it simply never found a matching prefix.

The only signal is the usage object. Every response reports three numbers you have to actually look at:

Field	Meaning
`cache_read_input_tokens`	Served from cache — you paid ~0.1×
`cache_creation_input_tokens`	Written to cache — you paid the ~1.25× write premium
`input_tokens`	Reprocessed at full price — not cached

If cache_read_input_tokens is zero across repeated requests that should share a prefix, a silent invalidator is at work. That single field is the whole diagnosis. When it's zero and you expected a hit, the move is mechanical: capture the fully rendered prompt bytes from two consecutive requests and diff them. The byte that changed is your invalidator, and it is almost always one of a small, known set.

The invalidators, in order of how often they bite

Every one of these changes the prefix bytes without changing the logical prompt — which is exactly why they're easy to write and hard to notice.

A timestamp or UUID in the system prompt. The classic. f"Current time: {datetime.now()}" or a per-request trace ID interpolated into the system header changes the prefix on every single request, so the cache is never read even once. Anything that must be dynamic belongs after the last breakpoint, in the message list — a fact injected at turn 5 invalidates nothing before turn 5.
Non-deterministic serialization. Building any part of the prefix with json.dumps(d) without sort_keys=True, or by iterating a set, can emit different bytes run to run for identical content. Same logical prompt, different key order, cache miss. Sort your keys and order your collections.
Tool-set churn. Tools render at position zero, ahead of everything. Adding, removing, or reordering a tool changes the very front of the prefix and invalidates the entire cache below it — across the whole session, and across users if the tool set is built per-user. Freeze the tool list and sort it deterministically; for genuinely dynamic tools, use a discovery mechanism that appends schemas rather than swapping the base set.
Per-user or per-session interpolation into the system prompt. Splicing a user ID, name, or session key into the system header gives every user a distinct prefix and kills all cross-request and cross-user sharing. Move it into the messages.
Conditional system sections. if flag: system += "..." means every combination of flags is a different prefix. Each variant caches separately and shares nothing.
Mid-session edits and model switches. Editing the top-level system prompt partway through a conversation re-processes the whole history uncached; switching models throws the cache away entirely (it's model-scoped). If you need to inject an operator instruction mid-run, append it as a message rather than editing the frozen system prompt.

## Why it hurts most exactly where you'd want caching

The cruel part is that the failure concentrates in precisely the workloads caching is for. A one-shot classification call barely cares. A long-running agent — a fleet of workers, a multi-step DAG, an orchestration loop that resends a large fixed system prompt and tool set on every one of hundreds of turns — is entirely built on the assumption that the big fixed prefix is nearly free after the first write. When a worker reassembles its system prompt on each call and one field in that assembly is nondeterministic, that assumption silently inverts: the largest, most-reused span of the prompt becomes the most-repaid. The bigger and more disciplined your prompt architecture, the more a single stray byte costs you.

The audit: make the cache assert itself

Because the failure is silent, the only real defense is to stop trusting and start measuring. The audit is short:

Classify every input by how often it changes. Never → put it early, before any breakpoint. Per-session → after the global prefix. Per-turn → at the very end. Per-request (timestamps, IDs) → eliminate it from the prefix or move it dead last.
Check that rendered order matches stability order. Stable bytes must physically precede volatile bytes. A timestamp in the system header defeats every breakpoint after it no matter how many cache_control markers you sprinkle downstream.
Freeze the load-bearing pieces. System prompt frozen, tools deterministically ordered, model fixed for the session, JSON serialized with sorted keys.
Then assert. After the first request warms the cache, the code should check usage.cache_read_input_tokens > 0 on the second — and fail loudly if it isn't. A silent cache miss caught by an assertion in development is a non-event; the same miss shipped to a long-running fleet is a standing tax you won't see until you read the bill.

None of this is exotic. It's the same discipline as any other cache: know your key, keep it stable, and verify your hit rate instead of assuming it. The only twist with prompt caching is that the key is the entire rendered prefix, so "keep it stable" reaches all the way back into how you build the system prompt — and the penalty for a stray byte is measured in tens of percent of your token bill, paid quietly, forever, until someone looks at the one field that would have told them.

More notes at hexisteme.github.io/notes.

A Fair Coin Isn't Enough: When a Perfectly Randomized Experiment Is Impossible to Analyze

John — Fri, 10 Jul 2026 00:00:06 +0000

Originally published on hexisteme notes.

We built a randomized experiment to earn the right to say one word — because. The random assignment was cryptographically fair; we verified it. Weeks in, an audit found that the experiment could not be analyzed at all, and never could have been. The coin was flawless. The join from the coin to the thing it decided did not exist in the data.

To make a randomized experiment analyzable you need three things that have nothing to do with the randomizer: a join that links each draw to the unit it was applied to and that unit's outcome, a guard that makes selective re-drawing impossible or at least detectable, and a power calculation done before you start so you know the sample can answer the question. We had a provably fair coin and none of the three. This is what each one is, why its absence is fatal, and what you actually have to record at the instant of assignment.

Why randomize at all: to earn one causal word

Most of what you can measure about a running system is observational — patterns under whatever policy happens to be in force. Observational data can generate hypotheses, but it cannot, by construction, tell you that a change caused an outcome, because the thing you changed is tangled up with everything else that moved at the same time. There is exactly one clean escape: assign the treatment at random. Randomization is what severs the treatment from all the confounders and lets you say the forbidden word — because — about the difference between arms.

So for one specific class of decisions we did the disciplined thing. Instead of always routing a task to the default handler, a dispatcher flipped a fair coin and sent it to arm A or arm B. That single randomized track was meant to be the one place in the whole system where a causal claim would be defensible. Everything hinged on it being done right.

The coin was provably fair — and that was the easy 10%

The randomizer used a cryptographically seeded source (SystemRandom().choice, backed by the OS entropy pool), so the draw was an honest, independent p=0.5 flip. When the observed split came in lopsided — roughly four to one — the instinct was to suspect a broken coin. That instinct was wrong: with that few draws, a two-sided binomial test puts a 4:1 split at P ≈ 0.375, nowhere near anything you would call biased. The coin was fine.

Here is the trap. Verifying the coin feels like verifying the experiment. It is vivid, it is mathematical, it produces a satisfying green check. And it is maybe ten percent of what makes a randomized experiment work. The other ninety percent is bookkeeping so boring nobody writes a test for it — and that is exactly where this one had already failed.

The fatal bug: nothing joined the draw to the task it dispatched

To analyze a randomized experiment you compute an intention-to-treat comparison: for every unit, what arm was it assigned, and what outcome did that unit get. That requires a join key — a stable identifier stamped on the unit at the moment of assignment and carried through to wherever the outcome is recorded. It is the spine of the entire analysis.

Our dispatcher was a function that took no arguments, returned an arm, and appended the draw to a log. The log recorded that a draw happened and which arm won. It did not record which task the draw was for. Nothing downstream ever read the dispatch log to tag the resulting work. In other words, the assignment and the outcome lived in two separate piles with no key joining them.

You cannot backfill this. The pairing between a draw and the task it dispatched is only knowable at the instant the dispatcher hands the task off. Once that moment passes without a shared id being written down, the link is gone — not hard to reconstruct, gone. Afterward you have N draws and M outcomes and no way to say which draw produced which outcome. You can count each pile. You cannot pair them. The intention-to-treat comparison — the only reason the experiment existed — is uncomputable by any code you could ever write against that data.

Three ways the ledger couldn't defend itself

The missing join was the headline, but the same audit found three smaller failures, each of which independently prevents an experiment from being trustworthy even by an honest operator.

No guard against selective re-randomization. Nothing in the pipeline made "one unit, one assignment, recorded once" an enforced invariant, and nothing would have detected a second draw for the same unit. The danger here is not that anyone cheated — it is that the experiment has no mechanism to prove it couldn't have. An experiment that cannot rule out selective re-draws cannot defend its own result against that exact objection.
Test draws share a pile with real draws. Draws made while building and debugging the harness landed in the same log as production draws, separated only by a note in a design doc — not a flag in the data. A cluster of near-simultaneous draws (42 milliseconds apart, machine-fast, obviously not real dispatches) sat unclassified in the log, occupying exactly the position where genuine data would be counted. The rule "developer draws don't count" is worthless if it lives in prose instead of an is_test column.
The falsifier that can't fire. The experiment was pre-registered with a kill switch: "if assignment/outcome mismatch exceeds 20%, declare the design a failure and stop." But mismatch is a quantity you compute from the join — and there is no join. So the falsifier references a number that cannot be measured. It can never fire. A stopping rule you cannot compute is not a stopping rule; it is a comforting sentence in a spec.

## Even fixed, it was underpowered — and nobody had checked

Suppose we fix all of the above tomorrow. There is a separate, earlier failure hiding underneath: the experiment was never powered. No minimum detectable effect was computed before it started. When we projected the effective sample size forward to the pre-registered deadline, it came to roughly 14–19 usable units against a target of 40 — under half. The experiment was on track to end inconclusive no matter how clean the bookkeeping was.

This is the quietest and most common way experiments waste themselves. Without a pre-planned MDE, an inconclusive result is ambiguous: you cannot distinguish "the two arms are genuinely equivalent" from "we never had enough units to see a difference." Those are opposite conclusions that produce identical-looking null results. The power calculation is what forces you to find out, before you spend the sample, whether the question is even answerable at the scale you can afford.

What instrumenting a randomized experiment actually requires

The transferable lesson is that randomization is the part that comes free from a library, and everything that makes the randomization usable is manual work you have to design in from the first line. Concretely, before the first real draw:

Stamp a joined record at assignment time. One row per draw carrying a stable unit_id, the assigned arm, an is_test flag, and a timestamp. The unit_id is non-negotiable — it is the key the entire analysis hangs from.
Carry the unit id to the outcome. The place where you measure the result must write down the same unit_id. If the outcome side doesn't know the key, the assignment side stamping it was pointless.
Make re-draws impossible or loud. Enforce one-assignment-per-unit, or at minimum emit a warning when the same unit is drawn twice within a short window, so selective re-randomization can't happen silently.
Separate test from production in the data, not in prose. The is_test flag, set at draw time, is the only thing that reliably keeps development noise out of your effect estimate.
Compute the minimum detectable effect first. Decide, before you begin, the smallest effect worth detecting and whether your reachable sample can detect it. If it can't, you don't have an experiment yet — you have a plan to collect an ambiguous null.

Randomizing an assignment is a one-line call to a good RNG. Turning that into an experiment you can actually analyze is a join, a guard, a flag, and an arithmetic check you do up front. We had the one-liner and skipped the four boring things, and the result was a beautifully fair coin deciding an experiment that, on the day we looked, could not answer its own question. The coin was never the hard part.

More notes at hexisteme.github.io/notes.

Model or Method? Building a Deterministic Lab to Measure an AI Agent Fleet's Own Behavior

John — Sat, 04 Jul 2026 00:00:05 +0000

Originally published on hexisteme notes.

A one-time forensic on our agent fleet concluded that "flagship-grade" quality lived in the method — orthogonal verification, ground-truth-first, closing the feedback loop — not in any one model's weights. That's a satisfying conclusion, and completely unfalsifiable as stated. A hand-audit of a few dozen sessions doesn't scale, doesn't update, and quietly launders the auditor's priors into the result. So we built the thing that would let the claim be wrong: a small, deterministic lab that measures the fleet's own behavior, session after session, with no LLM in the scoring path.

The forensic said method beats model. To test that ongoingly you need two columns most transcript tooling never captures: which method a session used, and what outcome it got. We built a stdlib-only SQLite lab that mines the fleet's own Claude Code transcripts, reassembles logical sessions (94% of the raw files are sidechain fragments — a file is not a session), and attributes every behavior to a (model, epoch) tuple instead of a thread-level majority. It refuses causal language everywhere except one randomized-dispatch track. The newest piece is a method fingerprint: per-epoch counts of which disciplines a session actually invoked — a council, a cross-family verifier, symbolic-math and theorem-prover engines, skills. And the first real lesson wasn't in the data. It was that the "obvious" field for skills — a top-level slug appearing 21,000+ times — turned out to be a session codename, not a skill. Trusting our own field catalog instead of the raw values would have ingested pure noise as signal.

The trap: "quality is the method" is unfalsifiable until you measure both axes

Here is the uncomfortable thing about concluding that quality comes from method rather than model. It is almost certainly true, and it is also exactly the shape of a claim that survives because nobody can check it. If a session goes well, you can always point at some discipline it followed. If it goes badly, you can always point at a discipline it skipped. The claim is a magnet for confirmation bias because the "method" is observed after you already know the outcome.

To make it falsifiable you need to record the method independently of the outcome, and record the outcome independently of your feelings about the session. That is two columns: a method label and an outcome measure, captured deterministically, per session, before anyone forms an opinion. Almost no transcript tooling captures either well — it captures tokens, latency, and cost, which are the things that are easy to log and the things that least distinguish a careful session from a sloppy one.

Why measuring an agent fleet is harder than it looks

Three structural facts make this genuinely hard, and each one silently corrupts the naive version of the analysis.

A file is not a session. In our corpus, 94% of the transcript files are sidechain fragments — subagent and workflow spawns, not top-level sessions. If you treat one file as one unit of analysis, your "sessions" are mostly fragments of sessions, and your per-session statistics are measuring delegation shape, not work. The unit has to be a reassembled logical session: a main thread plus every fragment that hangs off its UUID directory.
One session is not one model. A single thread frequently spans two or three models (a main model delegating to cheaper subagents, or a mid-session model switch). If you attribute the whole thread's behavior to its majority model, you mis-file a large fraction of turns onto the wrong model — in our data, when a thread was mixed, a majority-vote attribution pooled the minority model's turns straight into the wrong bucket. The fix is to attribute every metric to a (family, version, epoch) tuple at the level of the emitting assistant turn, and to keep a model_purity column so you can gate cross-model comparisons to near-pure threads.
The corpus evaporates. The raw transcripts auto-delete after about 30 days. That makes the least glamorous component — an incremental, delete-free snapshot job on a cron — the single most important pipeline in the whole system. If it dies, the lab silently loses its ability to ever look further back than a month, and you don't notice until you need the history.

## The discipline that makes it a lab and not a dashboard

The temptation with behavioral data is to immediately declare that model A "is better than" model B because its numbers look nicer. That is not what observational transcript data can support, and pretending otherwise is how you get confident, wrong routing changes. So the lab is built around one rule: association-only, everywhere, except one place.

Concretely: every comparison table reports descriptive medians, IQRs, and cell counts, plus a bootstrap 95% CI — and no p-values at this stage, because with this many implicit comparisons a p-value would be a license to fool yourself. Every table is forced to carry an overlap diagnostic (how many sessions actually populate each model × stratum cell), and any cell that doesn't overlap its comparison is stamped NOT_COMPARABLE rather than quietly compared anyway. Causal words — "because," "improves," "outperforms" — are banned from the report generator's vocabulary. The one exception is a randomized-dispatch track: for a specific class of delegations, the model is assigned at random, which is the only condition under which the data can actually identify a causal effect. Everything else is honestly labelled as what it is: a pattern under a fixed policy, Grade C evidence at best.

The method fingerprint: what discipline did this session actually use?

This is the piece that connects the lab back to the original "method beats model" claim. To ever correlate a discipline with an outcome, you have to record, per model-epoch, which disciplines a session reached for. Not what it should have done — what it did, as a deterministic count.

The disciplines that matter for our "flagship-grade" hypothesis all happen to leave the same kind of trace: they are tool calls to specific verification instruments. A multi-persona council for hard trade-offs. A cross-family verifier that routes a claim to a different model lineage so its failure modes are uncorrelated with the author's. Symbolic-math and theorem-prover engines that turn a prose numerical claim into something that either checks or doesn't. Explicitly loaded skills. Each of these is a tool invocation the transcript already records — so the method fingerprint is just a per-epoch, per-instrument call count, stored in a normalized table so the open-ended set of instruments never becomes an ever-widening pile of columns.

Across 155 sessions the fingerprint is already legible: symbolic-math and cross-family-verification calls dominate the flagship model's epochs, while the current relaunched-flagship regime has only a handful of sessions and a correspondingly thin fingerprint. That thinness is itself the honest finding — you cannot yet say anything about the newest regime, and the table says so out loud.

The first lesson wasn't in the data — it was to distrust our own catalog

Before any of that could work, we had to answer a boring question: where, in the raw transcript, does "which skill was invoked" actually live? An earlier structural census had catalogued a top-level field called slug, appearing over 21,000 times, and labelled it the skill/command slug. Perfect — ingest that.

Except the raw values were snuggly-whistling-hamster, functional-toasting-bird, cozy-stargazing-meadow. Six distinct values across a large sample, each repeated hundreds of times. That is not a skill name. That is an auto-generated session codename. The census had counted the field correctly and described it wrongly, and the description was the part we were about to build on.

The actual skill signal was somewhere much less exotic: the ordinary tool-call stream the parser was already reading. Skills invoked through the tool interface carry their name in the call's input; verification instruments carry theirs in the tool name itself. Using those meant zero new field dependencies, zero parser changes, and — critically — no reliance on a summary that had already been wrong once. We also looked at a richer-seeming top-level attribution field, but it was stamped on roughly three times more records than there were actual tool calls, because it also tagged the follow-up turns that merely processed a tool result. Counting it would have inflated "method usage" threefold. We left it on the floor and counted the calls themselves.

The transferable lesson is not about one field. It is that a catalog of your data is itself data, and it can be wrong. The counts were right and the meaning was wrong, which is the most dangerous combination, because the counts look like verification. The only defense is to read the raw values with your own eyes before you build on the label — the same ground-truth-first discipline the lab exists to measure.

Honest limitations (the part that keeps it a science)

Four caveats, stated plainly because a measurement system that hides its weaknesses is just a dashboard with better PR.

Process proxies are not quality. The closest thing to an outcome the lab has today is a set of proxies — a same-file re-edit rate, a validation-after-edit rate, a completion flag. The re-edit rate in particular cannot distinguish careful iteration from thrashing rework, and it carries that caveat in the code, the schema, and the report. Real outcome labels are the hard, unsolved next axis; without them, "method vs outcome" is still half a sentence.
The interesting regime is too small. The current relaunched-flagship model has a single-digit session count in the comparable window. The system's response is not to shrink the error bars and pretend; it is to print NOT_COMPARABLE and register a dated falsifier: if that count doesn't cross a threshold by a specific date, the associated hypothesis stays suspended.
The lab is in its own corpus. The sessions that built this lab are transcripts too, and they're in the data — and transcript-processing work has an unusually spiky tool signature. Self-measurement contamination is noted in the report's limitations and slated for its own exclusion stratum.
Association is not causation, still. Everything outside the randomized track is a pattern under a fixed routing policy. It can generate hypotheses. It cannot, by construction, prove one.

## The lab holds itself to the same standard

The final piece of honesty is a falsifier pointed at the lab itself. It has a due date: if several weeks of these reports never once change an actual routing decision — logged in a ledger, not remembered — then the lab has failed its only job and gets cut down, regardless of how elegant the schema is. A measurement system whose measurements never change a decision is a hobby, and the way you keep it from becoming one is to write down, in advance, the observation that would prove it useless.

That is the whole posture, really. The claim that started this — quality is method, not model — only becomes worth anything the moment you build the apparatus that could show it's false, capture the two axes independently, and read your own raw data before you trust your own labels. The lab isn't the answer to "model or method." It's the thing that makes the question answerable.

More notes at hexisteme.github.io/notes.

How to Stop AI Agent Skills, Hooks, and Cron Jobs from Silently Conflicting Over Where They Run and What Data They Trust

John — Wed, 01 Jul 2026 00:00:08 +0000

Originally published on hexisteme notes.

Make every skill, hook, and scheduled job declare four invariants before it ships — Locality (where it can run), Source-of-truth (which facts it owns or borrows), Cross-ref (what depends on it and what it depends on), and Trigger-measurability (whether its trigger is observable at runtime or hidden in external state) — and refuse to hand off any component that leaves one undeclared, because an undeclared assumption is exactly the seam where two components silently disagree.

Two separate runtime leaks surfaced in a single audit session, and both traced back to the same root cause: a component that never declared its assumptions. One read configuration from a file that had stopped being the source of truth (so it always returned a stale default); the other was a scheduled job pointed at a remote sandbox while its prompt referenced local-only paths — caught minutes before registration, where any later and it would have billed compute and produced nothing. Neither was a coding bug. Both were missing declarations.

The failure mode: components that work alone but leak when combined

When you build an AI agent system out of small parts — skills the model loads on demand, hooks that fire on lifecycle events, cron jobs and scheduled routines that run unattended, helper scripts, config profiles — each part usually gets tested in isolation. It works. You move on. The trouble is that "it works" only proves single-shot correctness; it says nothing about whether the part's assumptions agree with the rest of the system.

Every component carries hidden assumptions: where it runs (local machine vs. a remote sandbox), which facts it treats as authoritative, what other components it silently depends on, and what its trigger actually measures. When those assumptions go undeclared, conflicts stay invisible until they surface to the user as a flaky, hard-to-trace symptom — the kind that feels like a vicious cycle because every fix in one place re-opens a gap somewhere else. The deeper problem isn't any single line of code; it's that each component is operating without knowing its own assumptions, so nothing can detect when two of them disagree.

The fix: four invariants every component must declare

The rule is simple and enforceable: every new skill, hook, cron job, script, or routine declares four invariants, or it cannot be handed off. An undeclared invariant means the component itself is incomplete — it is the literal root of conflicts, leaks, and automations that bill compute but return nothing.

Locality — Where can this run? Declare one of local-only, repo-only, or both. A job that runs in a remote sandbox but references local-only paths is broken before it starts.
Source-of-truth — Which facts does it own authoritatively, and when it borrows an external fact, where does that fact actually live? Declare a sources: [...] list of the files/components it reads.
Cross-ref — What depends on this component, and what does it depend on? Declare a consumers: [...] list (who applies or reads it) and a "See also" section (what it relies on).
Trigger-measurability — Can the model observe this trigger in real time within a session, or does it depend on external state (machine powered on, network reachable, a remote service up)? If it's external-state-dependent, say so explicitly.

These four answer the four questions that, left unanswered, become the four ways components leak into each other.

Two real leaks, one missing declaration

Leak 1 — stale source-of-truth. A small script reported which config profile was active by reading the mcpServers key from one settings file. But the authoritative definition of those servers had migrated to a different file. The script was written when the first file was the source of truth, and never updated after the move — so it always returned the default base, and the status it displayed was quietly wrong. Had a Source-of-truth invariant been declared ("this fact lives in file B, not file A"), the mismatch would have been visible the day the truth moved.

Leak 2 — locality mismatch. A scheduled routine was registered to run in a remote sandbox, but its prompt referenced local-only resources (paths that exist only on the developer's machine). This was caught in the final seconds before registration. Any later and the schedule would have fired on the remote, found nothing, and produced billed-but-empty runs on a recurring cadence. A declared Locality invariant ("this runs remote → it may only reference remote-visible resources") would have flagged it immediately.

The two leaks looked unrelated — one a config-reading bug, one a scheduling mistake — but they share a single cause: a component that never declared its assumptions. That they happened in the same session is the tell that this is a structural gap, not a one-off slip.

Where the declarations live, by component type

Keep the declaration cost to roughly one line per invariant — the weight of declaring is far below the cost of verifying a conflict after the fact. Concretely:

# For a reusable decision record / doc with frontmatter:
---
id: NOTE-XXX
locality: local-only        # or repo-only / both
sources: ["path/to/file-that-owns-the-fact"]
consumers: ["who-applies-this", "downstream-automation"]
---

For a skill or command, put the same four in the markdown header or description — minimally a one-liner like (locality: local-only — depends on ~/config) plus a "See also" listing the RDUs/skills/commands it relies on. For a hook defined in JSON (where comments aren't allowed), register the invariants in the hook's companion doc and list the settings file in sources. For a cron job or routine, gate registration behind five questions answered out loud first: (1) Locality — does it run local or remote, and do its referenced resources exist there? (2) Sources — what's the source-of-truth for every fact the prompt cites? (3) Cross-ref — where does the output flow (a user report, a file update, a record evolution)? (4) Trigger-measurability — does the cron time line up with resource availability (machine on, service reachable)? (5) Reversibility — how do you turn it off if it's wrong (launchctl unload / disable / delete)? If even one answer is fuzzy, hold registration until it's clear.

Backfilling existing components and keeping it honest

You don't have to retrofit everything at once. Prioritize by blast radius. System entry points (the scripts that classify profiles or route config) come first — they're the most dangerous because everything downstream trusts them. Next, hooks that fire on every prompt, since a wrong assumption there taxes every interaction. Then your highest-frequency skills. Everything else can be backfilled slowly on a periodic audit cadence.

Bake the check into that cadence: scan for decision records whose frontmatter is missing locality (backfill candidates), skills whose headers don't declare invariants (backfill candidates), and any routine/cron/hook unchanged for 30+ days (re-verify its invariants, because the world it assumed may have moved underneath it — exactly how Leak 1 happened). Declaring an invariant isn't bureaucracy; it makes each component self-document its assumptions so that the next person to touch the system — including a future version of you, or your own agent — can detect a conflict on sight instead of debugging it in production.

Honest limitations

This is a discipline, not a compiler. Nothing mechanically enforces the four invariants unless you build that enforcement — a frontmatter linter, a registration gate, a pre-commit check. Declarations can also drift: a component can declare sources: [file A] truthfully on Monday and become a lie on Friday when the source-of-truth moves, which is precisely why the 30-day re-verification step exists rather than a one-time declaration. The invariants also can't catch a conflict between two components that each declare correctly but make incompatible assumptions about each other's behavior — declaration surfaces the seam, it doesn't reconcile semantics. And there's a real failure mode of over-declaring: if every trivial helper grows a four-field header, people stop reading them and the signal drowns. Keep declarations to one line each, reserve the full five-question gate for unattended automation (where a silent failure is most expensive), and treat the lightweight inline form as enough for in-session skills and commands.

More notes at hexisteme.github.io/notes.

Do Multiple Personas on One LLM Give Real Diversity, or Do You Need Different Model Families?

John — Tue, 30 Jun 2026 00:00:05 +0000

Originally published on hexisteme notes.

Multiple personas on a single LLM do not give you real diversity — they are prompt variations of one set of weights, so they share the same blind spots, and the only durable diversity comes from different model families plus external tool verification plus an adversarial round.

In an 8-round self-audit of a persona-only council (all personas on the same model family), the measured ceiling was a track_record of 31% and internal_consistency of 65% — no amount of prompt-trailer tuning, persona-count tuning, or model-tier splitting pushed past it. Two patterns explain why and how to fix it: the Aggregator Bottleneck (a single aggregator model re-homogenizes whatever diversity its sub-agents produced) and a ΔEVD test (measure mean-pairwise-cosine-distance between answers; keep a reframing only if it adds more than +0.15, discard it as theatrical if it adds +0.05 or less).

The short answer: same weights, same blind spots

If you spin up eight "experts" — a skeptic, an optimist, a security reviewer, a contrarian — and they are all the same underlying model behind different system prompts, you have built a debate club whose members were all educated at the same school, from the same textbooks, by the same teachers. They will phrase disagreement differently, but they hallucinate the same nonexistent citations, anchor on the same wrong dates, and miss the same structural weakness in your design. Persona diversity is a presentation-layer trick; it is not an epistemic one.

This matters because a council's whole value proposition is catching the error your first answer missed. If every member shares the same training data, the same RLHF shaping, and the same tokenizer, then the error your base model is prone to is an error all members are prone to. You get a chorus, not a cross-check. The fix is not more personas — it is genuinely different sources of judgment.

The measured ceiling from an 8-round self-audit

This isn't a vibe. Over eight rounds of meta-auditing a persona-only council (every persona running on one model family), the aggregate numbers settled at a track_record of 31% and an internal_consistency ceiling of 65%. The audit deliberately exhausted the obvious levers: evolving the persona system-prompt trailers across five versions, fixing the data pipeline that scored outcomes, and splitting personas across a larger and a smaller model of the same family. None of it moved the ceiling. That is the signature of a limit that lives in the model weights, not in the prompt.

The interpretation is straightforward and a little humbling: when consensus is built from one model's weights, agreement measures conformity, not correctness. A 65% internal-consistency ceiling means the personas couldn't even reliably agree with themselves across runs, and a 31% track record means their confident consensus was usually not the thing that actually held up. Prompt engineering could change the flavor of the answers but not the underlying distribution they were drawn from.

Three axes of real diversity

If persona count is the wrong lever, what are the right ones? Three axes, applied in sequence, each contributing a kind of diversity that prompt variation cannot manufacture:

(A) Cross-model family. Route the same question to models with different training data, different RLHF, and different tokenizers — e.g. one from each of two or three independent providers. Because their failure modes are uncorrelated, a weakness that all of them independently flag is a real weakness; a claim only one of them makes is suspect. This is the load-bearing axis.
(B) Tool verification (one pass). After the models converge, run a single web/citation check against the claims they relied on. Models confidently cite papers that don't exist and dates that are wrong; one external fact-check pass severs the hallucinations before they reach your conclusion. Label anything you can't verify as unverified and drop it from the consensus.
(C) Adversarial perspective (separate round). Once a verified consensus exists, run a distinct round whose only job is to attack it: "assume this consensus is wrong — what single experiment would prove it?" This is not a compromise or a synthesis of the earlier rounds; it is a deliberate falsifier-generation step that produces an experiment + observable + timeline + null-result implication, not more prose.

The output shape is the point. A persona council hands you a paragraph of consensus. A three-axis council hands you a weakest link, a list of verified citations, and a falsifiable prediction — three things you can act on immediately (run the experiment, dig deeper, or discard the idea). In one head-to-head application the actionable ROI was roughly an order of magnitude higher, almost entirely because the deliverable changed kind, not just quality.

The Aggregator Bottleneck

Here is the trap that quietly destroys multi-family setups: you fan out to three different model families, collect three genuinely diverse answers — and then ask one model to summarize them. That aggregator's alignment acts as a funnel. Its RLHF rewards balanced, agreeable, smoothed-over prose, so it averages the dissent away and re-homogenizes exactly the diversity you paid for. You did the expensive cross-family work and then threw the result through a single-model bottleneck at the last step.

This was identified empirically: when the adversarial round was written by the same aggregator model that ran the rest of the pipeline, an outside-family model pointed out — correctly — that the aggregator had funneled the earlier diversity into its own house style. The remedy: do not let one model both diversify and conclude. Have the adversarial round (axis C) written by an external family, or by the outlier voice from round one — not by the aggregator. Preserve dissent as raw branches rather than collapsing it into a synthesized middle; the median of disagreeing experts is frequently the one position none of them would defend.

The ΔEVD test: is your reframing real or theatrical?

A subtler failure is Prompt Framing Lock-in: when all your families converge on the same weakest link in nearly identical words, the cause may not be that they agree — it may be that your single shared prompt framed the problem so narrowly that no model could escape it. The instinct is to add a "reframing layer" (a router that rewrites the prompt, a per-family persona generator, an outcome-first reformulation). But every reframing option merely moves the lock-in to a different layer; you can't assume it helped. So measure it.

The ΔEVD test (Embedding Variance Delta) makes the decision empirical instead of hopeful:

1. Send the raw prompt to N model families → embed the N answers
   with a neutral embedder → compute MPCD0
   (Mean Pairwise Cosine Distance).
2. Apply ONE reframing option → re-run the N families → MPCD1.
3. Decide on ΔMPCD = MPCD1 - MPCD0:
     > +0.15        keep the reframing (real diversity gain)
     +0.05..+0.15   marginal — hold; try another round/reframing
     <= +0.05       discard it as theatrical noise; keep raw prompt

The harsh-but-honest conclusion from running this test on itself: most reframings fall at or below the +0.05 floor, which means they were theater. Treat the raw prompt as the most honest baseline and only adopt a reframing that clears the +0.15 bar on actual embedding distance. This is the same discipline as not collapsing dissent into a median — you don't get to claim diversity you can't measure.

Honest limitations and when persona councils are still fine

Three caveats keep this honest. First, cross-family diversity is necessary but not sufficient: the academic consensus (e.g. work on multi-agent debate in 2025) is blunt that a debate cannot exceed the accuracy of its strongest participant — diversity surfaces and weights candidate answers, it does not conjure correctness that none of the participants possessed. If all your models are weak on a topic, a council of them is still weak. Second, naive iterative debate and majority voting can actively entrench an initial error through model conformity; the gains come from careful diversity, argument-quality weighting, and preserving dissent — not from more rounds. Third, the 31% / 65% numbers are from one specific persona-council implementation's self-audit; the ceiling will differ across setups, and the cross-family ROI multiplier is a single observed comparison, not a benchmark — treat it as directional.

And don't over-apply this. The three-axis pattern is for decisions where being wrong does real damage: new system designs, claims you're about to ship, intuitions you suspect touch an unsolved problem, a thesis your single-model review couldn't crack. For a quick gut-check on a low-stakes opinion — a color choice, a naming preference, a sanity skim — a single fast call (persona or otherwise) is entirely adequate and the full pipeline is overkill. The skill is matching the verification depth to the cost of being wrong, not running the heavy machinery on everything.

More notes at hexisteme.github.io/notes.

Claude Code '400: no low surrogate in string' on every turn: repairing a permanently broken session transcript

John — Mon, 29 Jun 2026 00:00:08 +0000

Originally published on hexisteme notes.

A Claude Code session that returns API Error: 400 ... not valid JSON: no low surrogate in string on every turn is poisoned by a lone UTF-16 surrogate (a code point in U+D800–U+DFFF) baked into its on-disk transcript, and the fix is to close that session, strip only those lone surrogates from the offending line of the .jsonl file while leaving real emoji untouched, re-serialize that one line, and then claude --resume.

The poison is already on disk and is precisely targetable: a normal emoji is a single Python code point (e.g. U+1F9ED) and can never fall inside the surrogate range U+D800–U+DFFF, so deleting only that range removes the broken half-character with zero collateral damage to valid text. A cheap C-level byte pre-filter (scan for the \ud escape or raw ED A0–BF bytes before doing any per-line json.loads) cut a 174-file transcript scan from 3.4s to 1.1s, making it cheap enough to run automatically on every session start.

Why every turn fails: the surrogate is in the transcript, not the network

Claude Code persists each session to a JSONL transcript on disk (one JSON object per line, under your projects directory). Every turn replays the accumulated history back to the API. So if a single byte sequence in that history is invalid, the API rejects every subsequent request with the same error, at the same byte offset — the session is permanently bricked, and reopening it doesn't help because the bad data is reloaded from the file.

The specific failure is 400 The request body is not valid JSON: no low surrogate in string: line 1 column N (the mirror-image variant is no high surrogate). Non-BMP characters — emoji like 🧭, some extended CJK ideographs — are encoded in UTF-16 as a pair of surrogate code units: a high surrogate (U+D800–U+DBFF) followed by a low surrogate (U+DC00–U+DFFF). When a large tool output is truncated by a length limit and the cut lands exactly between the two halves of a pair, one orphaned half survives. That lone surrogate gets written into the transcript, replayed on every turn, and the API's strict JSON parser refuses it.

The triggering pattern is mundane: dumping a big, emoji-heavy blob into the context — a worker log peppered with status emoji, a daily-report job's output, a verbose ingest run — right before a large body that gets truncated mid-pair. Content-heavy projects (anything generating a lot of natural-language or creative text with non-BMP characters) re-hit this per session, not once.

The safe repair: strip only U+D800–U+DFFF, leave real emoji intact

The key insight that makes the fix safe is a property of Python's str: a valid emoji is a single code point (🧭 is U+1F9ED), so it can never land in the surrogate range U+D800–U+DFFF. Anything you find in that range is, by definition, a broken half. So you can delete exactly those code points and every legitimate character — emoji included — is left byte-for-byte untouched. You are not "removing emoji"; you are removing the orphaned halves that should never have been on disk.

The manual version of the fix, when you don't have a script handy:

import json

# read the one offending line, parse it, walk every string,
# drop only lone surrogates, re-serialize compactly
obj = json.loads(line)

def strip(o):
    if isinstance(o, str):
        return "".join(c for c in o if not (0xD800 <= ord(c) <= 0xDFFF))
    if isinstance(o, list):
        return [strip(x) for x in o]
    if isinstance(o, dict):
        return {k: strip(v) for k, v in o.items()}
    return o

fixed = json.dumps(strip(obj), ensure_ascii=False, separators=(",", ":"))

Write fixed back as that single line, leaving every other line of the transcript exactly as it was. Always copy the file to a .bak first. Re-serializing only the broken line keeps the diff minimal and preserves the rest of the conversation verbatim. After the rewrite, claude --resume and the 400 is gone.

Close the session before you touch the file

This is the step people skip, and it silently undoes the repair. While a session is open, Claude Code holds the transcript and will overwrite your edited file from its in-memory state — your fix vanishes the moment the next turn flushes. You must close the target session first, or verify nothing holds the file:

lsof -- ~/.claude/projects//.jsonl

Empty output means it's safe to edit. A good repair tool checks this for you and refuses (skips) any transcript that is currently held open, so an automated pass can never corrupt a live session. The trade-off: the one session you most want to fix — the one throwing 400s right now — may be the one you have open, so a fully automatic pass can skip exactly that file. That's the case where you fall back to the manual close-then-fix once.

Make it cheap enough to auto-repair on every session start

Because a content-heavy project re-breaks per session, a one-time manual fix isn't durable — you want the repair to run automatically before you ever see the error. The natural place is a SessionStart hook that scans recently-modified transcripts and silently cleans the closed ones. The problem is that per-line json.loads over every transcript in your projects tree is too slow to run on every launch.

The fix is a cheap byte-level pre-filter that runs before any JSON parsing. A lone surrogate only persists to disk in two shapes, and both are detectable by scanning raw bytes:

The JSON escape \ud... (some serializers emit unpaired surrogates as a \uXXXX escape; valid UTF-8 text never produces a literal \ud on disk).
The raw three-byte UTF-8 surrogate encoding ED A0–BF (valid UTF-8 only allows ED followed by 80–9F, so ED + A0–BF is unambiguously a surrogate).

A file with neither signal is provably clean and skips the expensive parse entirely. In practice this matters: a 174-file scan dropped from 3.4s to 1.1s, cheap enough to run on every session start. Run it scoped to recent days only, quiet unless something was actually fixed, and skipping any open transcript via lsof. A representative one-shot invocation in a hook:

python3 fix-jsonl-surrogates.py --fix-all --recent 3 --quiet

## Why you can't prevent it upstream (honest limitation)

This is a repair pattern, not a prevention pattern, and that's a deliberate concession to where the truncation happens. The cut that orphans a surrogate occurs inside the harness's own truncation logic, between the model output and the transcript write. The user-facing hook lifecycle (PreToolUse, PostToolUse, Stop, SessionStart, and so on) fires around lifecycle events, not in the middle of serializing the request body — so there is no interception point that can stop the bad byte from being written in the first place. Making the recovery fast and automatic is the only lever you actually control.

The pre-filter approach also has a real gap: if you immediately --resume the very session that's broken, the auto-repair pass may find the file held open and skip it (correctly, to avoid clobbering live state), so coverage is not 100% — that's the one case needing a manual close-then-fix. And do not lean on /compact as a fix: if a lone surrogate survives into the summary, the 400 persists; if it works, it worked by luck, not by design. The only genuine frequency-reduction is behavioral: avoid dumping huge emoji-saturated blobs (verbose logs, status-emoji-heavy output) into the context wholesale, or ASCII-ify such logs at the source.

More notes at hexisteme.github.io/notes.

If an LLM Extracts the Inputs, Is Your Deterministic Score Really Deterministic? Stopping Provenance Laundering

John — Sat, 27 Jun 2026 00:00:06 +0000

Originally published on hexisteme notes.

No — a scoring function that consumes whatever values an LLM hands it is only deterministic in name; the LLM's judgment launders straight through the "deterministic" gate, and you close the hole with three rules (host-verified FACT sourcing, FACT-only scoring, and an asymmetric penalty where bad signals are penalized regardless of provenance while good signals only score when verified) plus multi-round adversarial testing.

The load-bearing trick is an asymmetric-penalty mechanism: an unverified input can only ever lower a score, never raise it. Bad signals are penalized regardless of where they came from (so you can't dodge a penalty by routing the bad news through a weak source), while good signals are credited only when they carry a FACT provenance. We hardened this through three rounds of adversarial review, with an attacker-satisfaction score climbing 58 → 71 → 96 as each round peeled back a deeper laundering channel — and the decisive case was a candidate that scored 94/ADOPT on 9 web-sourced (unverified) signals plus a single real FACT, which the fixed gate correctly flipped to AVOID.

The trap: "the LLM only extracts, Python decides" is not enough

A common and sensible architecture for trustworthy automation is to split labor: a language model reads messy sources (web pages, docs, API responses) and extracts structured inputs, then a plain deterministic function scores those inputs. The appeal is obvious — the model never gets to invent the verdict, so the verdict is reproducible and auditable. Teams describe this as "the LLM extracts, the code decides," and treat the output as deterministic.

The problem is that determinism of the function says nothing about the integrity of its inputs. If the scoring code consumes any value that is present, then the model's judgment re-enters through the input channel and the "deterministic gate" becomes a laundering channel for exactly the judgment you tried to remove. The function f(x) is perfectly deterministic; it is x that the model controls. Calling the result deterministic is a category error unless you also gate what x is allowed to be.

Concretely, imagine a generic scoring pipeline that rates a candidate (a library to adopt, a vendor to onboard, a data record to act on) on a 0–100 scale and emits ADOPT / HOLD / AVOID. An LLM fills the input fields by reading sources. The day you ship it, the score is reproducible — and also completely steerable by whatever the model decided to write into the fields.

Three laundering paths a deterministic score is exposed to

Even when each input carries a provenance tag — say a 4-level ladder FACT | INFERENCE | ASSUMPTION | UNKNOWN where FACT is supposed to mean "machine-fetched from a primary source" — there are three distinct ways arbitrary or model-generated input still poisons a deterministic output:

(a) Label forgery. The provenance tag is itself just another field the model writes. If the code trusts the tag, the model can stamp FACT on a value it guessed. Nothing was fetched; the label lied.
(b) Score laundering. The scoring function consumes non-FACT values as if they were FACT — "the field is filled, so use it." A floor like "require at least N FACTs" does not stop this, because counting FACTs never inspects which values the score actually consumed. You can satisfy the FACT count with one real fact and still drive the whole score off nine inferences.
(c) Penalty dodging. If a bad signal is only penalized when it arrives as a FACT, an adversary routes the bad news through a weak source (label it INFERENCE) and the penalty silently disappears. This is a perverse incentive baked into the gate: weak sourcing becomes a way to suppress red flags.

The measured failure that made this concrete: a candidate built from 9 web-sourced (unverified) signals plus a single genuine FACT scored 94 → ADOPT. A FACT-count floor passed it. The score was "deterministic" and wrong, because it had laundered nine pieces of model judgment into a confident green light.

The fix: three rules plus one asymmetry

You close all three paths with three rules and a single asymmetry. The rules establish what counts as trustworthy input; the asymmetry guarantees that anything failing those rules can only ever hurt a score, never help it.

1. FACT means machine-fetched AND host-verified — fail closed. Do not trust the source label. Verify that the value's URL host is on a whitelist for the claimed source type, and reject subdomain spoofing. An unregistered host means the value cannot be FACT: demote it and record a gap. This kills path (a).

def host_matches(source, url):
    allow = HOST_WHITELIST.get(source)   # canonical hosts per source type
    if not allow:
        return False                     # fail-closed: unknown source type
    h = host_of(url)
    # exact host or a true subdomain; blocks evil-example.com.attacker.net
    return any(h == a or h.endswith("." + a) for a in allow)

2. The score is driven by FACT only. The scoring function trusts a value only when provenance is FACT; every non-FACT value resolves to a conservative default, never to its raw model-supplied number. Ban "the field is known, so use it." This kills path (b).

def num(ev, default):
    if ev.provenance is not Provenance.FACT or ev.value is None:
        return default, True   # non-FACT -> conservative value, unverified=True
    return float(ev.value), False

3. Asymmetric penalty. A bad signal is penalized regardless of provenance — so you cannot dodge a penalty by laundering the bad news through a weak source. A good signal is credited only when it is FACT (rule 2 already guarantees this). The combined effect is the load-bearing invariant: unverified input can lower a score but never raise it. This kills path (c).

# bad signal: penalize on is_known, provenance-independent
if maintainer_count.is_known and maintainer_count.value <= 1:
    demote("single-maintainer-risk")
# good signal: crediting is FACT-only, enforced by num() above

## Why one round of review is not enough: 58 → 71 → 96

These defects do not surface in a single pass. They are layers of the same threat — "this deterministic gate is laundering judgment" — and each adversarial round peels back the next layer. In our hardening of a generic adoption-scoring pipeline, an adversarial reviewer (you can use a second model, a colleague, or a structured red-team checklist) was asked each round to break the determinism claim, and its satisfaction climbed across three rounds: 58 → 71 → 96.

Round 1 (58): caught label forgery — the gate trusted the provenance tag. Fix: host-verified whitelist (rule 1).
Round 2 (71): caught source laundering — values claiming an official-docs origin were accepted without host checks, so the attacker spoofed the source type rather than the label. Fix: fail-closed host matching extended to every source class.
Round 3 (96): caught score laundering — the FACT-count floor passed a score driven by non-FACT values. This is the deepest layer and the one most teams miss: a count of trusted inputs is not the same as a score built only from trusted inputs. Fix: FACT-only consumption (rule 2) plus the asymmetric penalty.

A single round of review would have shipped after fixing label forgery and declared victory at 58, leaving the actual laundering channel wide open. The lesson generalizes: when the threat is "my safety boundary is being bypassed," iterate the adversary until its satisfaction plateaus, because each fix exposes the next assumption.

Pin the defense with an attack matrix regression test

Once you have the three rules, freeze them as tests so a future refactor cannot quietly re-open a laundering path. Encode the attack matrix directly: each row is an adversarial input, each expected outcome is the gate's correct response.

# attack matrix as a regression suite
cases = [
    ("normal FACT, verified host",        EXPECT_PASS),
    ("label forgery (INFERENCE -> FACT)",  EXPECT_DEMOTE_AND_GAP),
    ("source spoof (fake official host)",  EXPECT_REJECT),
    ("score laundering (FACT count met,"
     " score driven by non-FACT values)", EXPECT_NO_CREDIT),
    ("penalty dodge (bad signal as weak"
     " source)",                          EXPECT_PENALTY_APPLIED),
    ("missing key signal",                 EXPECT_DEMOTE),
]

The two tests most worth writing are the ones that catch the subtle paths: (1) a candidate whose FACT count is satisfied but whose numeric inputs are non-FACT must receive conservative defaults, not credit; and (2) a bad signal arriving as INFERENCE must still trigger its penalty. If both pass, your gate has the invariant "unverified input can only lower the score." Re-run the matrix on every change to the gate.

Honest limitation: code cannot close the injection boundary

These three rules close the laundering paths that live inside the scoring pipeline. They do not — and cannot — close the boundary one layer up: the honesty of the session that injects the inputs. Host verification proves a value came from registry.example.com; it does not prove the model fetched the right page, reported the response faithfully, or didn't omit a contradicting field it saw. A sufficiently capable injecting agent that controls its own tool calls can fetch a real primary source and still cherry-pick or misreport from it.

The correct posture is to separate this residual boundary honestly rather than pretend code closed it. Mark it explicitly as an intrinsic trust boundary in your threat model — the same way you would label "we trust the OS kernel" — and mitigate it with controls that live outside the deterministic function: pinning the exact fetch URL and re-fetching it independently at verification time, diffing the model's reported value against a second independent fetch, sampling outputs for human spot-checks, and logging raw responses so a claimed FACT is auditable after the fact. Claiming a deterministic gate fully neutralizes a dishonest injector is itself a form of laundering — of your assurance to whoever consumes the score.

More notes at hexisteme.github.io/notes.

macOS: nslookup works but curl and Python "Could not resolve host" — the mDNSResponder zombie

John — Fri, 26 Jun 2026 00:00:06 +0000

Originally published on hexisteme notes.

If nslookup resolves a host fine but curl, pip, and Python (requests/httpx) fail with "Could not resolve host," your mDNSResponder daemon has almost certainly entered a non-responsive "zombie" state — and the fix is to restart it (sudo killall -9 mDNSResponder), not to touch your API keys, SDK versions, or code.

This failure is maddening because every signal points the wrong way. nslookup example.com returns a clean IP, so DNS "works." Your network is up. Your code didn't change. Yet curl, pip install, and every Python HTTP call die with "Could not resolve host." People burn hours rotating API keys, downgrading SDKs, and editing config files. None of that is the problem.

The two-DNS-path asymmetry (your fastest diagnosis)

macOS resolves DNS over two independent paths, and the asymmetry is the diagnosis:

Path	Who uses it	Goes through mDNSResponder?
Direct	`nslookup`, `dig`	No — queries DNS servers directly
System resolver	`curl`, Python, `pip`, most apps (`getaddrinfo()`)	Yes — routes to the daemon

The daemon can keep its PID alive while silently refusing to answer. So the direct path (nslookup) succeeds and the system-resolver path (curl) fails on the exact same host. When one path works and the other fails, you are not looking at a network, key, or code problem — you are looking at a sick daemon.

Confirm it in 10 seconds

Use the daemon path directly so you're testing the same route curl uses:

# Uses the mDNSResponder path (same as curl/Python). Empty result = zombie.
dscacheutil -q host -a name example.com

# ...while the direct path still returns a valid IP:
nslookup example.com

If dscacheutil comes back empty but nslookup returns an IP, the daemon is confirmed. One more check proves routing and TLS are fine and isolates the daemon as the sole cause:

curl --resolve example.com:443:<IP-from-nslookup> https://example.com
# Succeeds? Then DNS resolution is the only broken thing.

Fix it weakest-tool-first

# 1) flush + reload (least disruptive)
sudo dscacheutil -flushcache && sudo killall -HUP mDNSResponder

# 2) kickstart the service
sudo launchctl kickstart -k system/com.apple.mDNSResponder

# 3) universal hammer (if kickstart prints "Could not find service" —
#    the service path differs across macOS versions)
sudo killall -9 mDNSResponder mDNSResponderHelper
#    launchd's KeepAlive immediately restarts it with a fresh PID.

# verify
pgrep -l mDNSResponder && dscacheutil -q host -a name example.com

Why agents and long-lived dev boxes hit this repeatedly

The zombie is usually triggered by connection-pool exhaustion. Many concurrent outbound long-poll connections — several MCP servers, a local inference/LLM server, and a background job all holding sockets open at once — push mDNSResponder into high CPU until it stops answering.

Measured case: 30+ concurrent long-poll connections drove mDNSResponder to 77% CPU and into the non-responsive state. The recovery alone doesn't stop recurrence — you have to find and cut the connection source. Watch for the active connection count climbing (lsof -i -P -n | wc -l in the 80+ range) and mDNSResponder CPU above ~30%.

"Disabled" in config does not mean the process is dead

A subtle trap when hunting the connection source: a server can be flagged off at the code/config level (an ENABLED=False switch) while its OS process keeps running and keeps holding its long-poll connections. The flag stopped new work from being dispatched but never killed the process, so it kept feeding the pile-up. When auditing, check ps/pgrep for the actual process and its elapsed time — not just the config flag.

pgrep -lf <server-name>        # still there?
ps -o pid,etime,%cpu,command -p <pid>

Prevent recurrence — and the honest limitation

Two durable fixes: reduce concurrent long-poll load (trim always-on servers/MCP endpoints, kill stale inference servers, and on a box that runs for many days, restart mDNSResponder on a schedule), or install a watchdog LaunchDaemon that kickstarts the daemon when its CPU crosses a threshold.

Honest caveat: this exact two-path asymmetry is macOS-specific — it's mDNSResponder/getaddrinfo behavior, not Linux's nsswitch/resolv.conf. And a watchdog that calls launchctl kickstart needs root: implement it as a LaunchDaemon running as root with a narrowly scoped script, not a broad sudo NOPASSWD rule. The watchdog is a band-aid — the real fix is capping concurrent connections.

More notes on building & running an AI agent fleet at hexisteme.github.io/notes.

A file-based work-bus for orchestrating a fleet of agent CLIs — coordination without a message broker

John — Wed, 24 Jun 2026 00:00:05 +0000

Originally published on hexisteme notes, part of a series on building and running an AI agent fleet.

To coordinate a fleet of independent AI agent CLIs without a message broker or a heavy framework, use a filesystem work-bus: the orchestrator decomposes a goal into a graph of subtasks, writes a Task file per subtask, and polls for the Result file each worker writes back — every file written atomically. The durable coordination state lives on disk as files: language-agnostic, debuggable with ls, surviving restarts, and self-healing because an absent worker is skipped and logged instead of failing the run.

Say you have several AI agents, each an independent installed CLI — one gathers information, one writes copy, one builds an app scaffold — and you want to run a goal that needs several of them in sequence. The heavyweight answers are an in-process framework (LangGraph, an AutoGPT-style loop) or a message broker (Redis, Kafka, RabbitMQ). Both are more than a single-operator fleet needs: a framework couples your workers into one process and one language, and a broker is infrastructure you now have to run, secure, and monitor.

There's a lighter primitive that fits this shape: a work-bus made of files.

The mechanism

A conductor process owns a shared directory — the bus. To run a goal:

Decompose the goal into a directed acyclic graph of subtasks (e.g. gather → narrate → build).
For each ready subtask, write a Task file into the bus, tagged with the capability it needs.
Poll for the matching Result file, with a short backoff.
Absorb each result, validate it, and release the next subtasks in the graph.

The one rule that makes this safe is atomic writes: write each record to a temp path and rename it into place. Rename is atomic on POSIX filesystems, so a reader either sees the whole file or nothing — never a half-written record. Task and Result are typed records (a small pydantic schema), and the conductor keeps a registry of what's in flight.

# atomic publish — a reader never sees a partial record
def publish(path, record):
    tmp = path.with_suffix(".tmp")
    tmp.write_text(record.model_dump_json())
    tmp.rename(path)          # atomic on POSIX

# the conductor loop
for task in topo_order(dag):
    publish(bus / f"{task.id}.task.json", task)
    result = poll(bus / f"{task.id}.result.json", backoff=...)   # durable: waits for the file
    absorb(result)

This is state, not events — by design

It's fair to ask whether a file work-bus is just an event bus in disguise. It isn't, and the distinction is the reason it works. An event bus is push: producers emit ephemeral events, and anything not listening at that instant misses them. A file work-bus is state: the Task and Result records are durable files that stay until consumed. A worker that starts late, or restarts mid-run, still finds its task waiting. (I argued the same principle for monitoring a fleet in state is truth, events are rumors — here it shows up again for coordinating one.)

Why push is fine here but not for monitoring. You build and control these workers, so you can make them read and write the bus. Monitoring is the opposite case — you watch components you don't control, so you pull their state instead. Coordination of owned workers via durable files, monitoring of unowned components via state scans: both lean on durable state over ephemeral events.

Routing by capability, not by name

The conductor doesn't hard-wire "send step 2 to worker X." Each worker advertises capabilities; each subtask declares the capability it needs; the conductor matches them at dispatch time by finding a healthy worker that advertises the required capability. Add or remove a worker and routing adapts — there's no wiring diagram to edit. This is what lets one conductor coordinate a heterogeneous, changing fleet through a single uniform contract.

Graceful degradation: skip the absent worker

The most important behavior for a fleet that's still being built: a missing worker must not fail the run. If a subtask needs a capability no healthy worker currently advertises, the conductor marks that node skipped (a logged worker_absent), continues the rest of the graph, and synthesizes from whatever completed. On day one, when most workers don't exist yet, the conductor still runs end-to-end and produces partial output — and the skip log is a precise to-do list of which capabilities to build next. A gap is reported, not crashed on.

Trust the bus like a network boundary

Worker output is untrusted input crossing a boundary, and the bus treats it that way. Every result is parsed into a strict schema before absorption; mismatches (say, casing differences between the wire format and internal enums) are coerced and normalized at the seam. Load-bearing claims carry a provenance label and must include evidence — a claim that arrives marked FACT with no evidence IDs is rejected at parse time, not trusted. The typed contract is what lets independent workers, written in different languages by you at different times, interoperate without the conductor having to trust any of them blindly.

The honest limitation

⚠️ No stop condition on re-routing. Capability-based routing has a sharp edge: if a node can be re-routed to "any worker advertising capability C" and results keep failing validation, a naive conductor can re-route in an unbounded loop. A file work-bus needs an explicit per-node attempt budget (and a dead-letter outcome) or it can spin. Durability and decoupling are the wins; a bounded retry policy is the cost you must pay to claim them safely.

When to reach for a real broker

This pattern fits a small, heterogeneous fleet running tasks that take seconds to minutes, coordinated by one operator. If you need high-throughput, low-latency fan-out across many producers and consumers, run a real message bus — the file-bus's polling and single-conductor model won't keep up. Match the tool to the failure that hurts: for a solo fleet, the pain is operational overhead and brittle coupling, and a directory of atomic files removes both.

More notes on building an AI agent fleet — why I rejected an event bus for monitoring, labeling facts vs inferences, reusable decision units — at hexisteme.github.io/notes.

DEV Community: John

Stop Hooks as Hard Constraints: Enforcing Claude Code Behavior Outside the Model

How stop hooks actually work

The AND-gate pattern

Production hook 1: decision-ownership enforcement

Production hook 2: sycophancy prevention

The nag-once pattern

What hooks can't do

Combining hooks

Why I Rejected an Event Bus for My Solo Agent Fleet: State Is Truth, Events Are Rumors

The architecture I almost built

Reason 1: the instrumentation tax is a project killer

Reason 2: state is truth, events are rumors

Reason 3: orchestration schizophrenia

Reason 4: the bus itself becomes an unreviewed dump

What pull looks like instead

Freshness is "unread," not "old"

The boundary that keeps it honest

When push is actually right

When a Site Blocks Your Scraper, Read the Browser Tab You Already Have Open

The wall: the data is behind everything that stops a scraper

The move: you already have the page open

Why it sidesteps every wall at once

The honest edges

The Silent 10 Tax: How a Nondeterministic System Prompt Voids Your LLM Prompt Cache

How prefix caching actually works

The silent failure: cache_read stays zero and nothing tells you

The invalidators, in order of how often they bite

The audit: make the cache assert itself

A Fair Coin Isn't Enough: When a Perfectly Randomized Experiment Is Impossible to Analyze

Why randomize at all: to earn one causal word

The coin was provably fair — and that was the easy 10%

The fatal bug: nothing joined the draw to the task it dispatched

Three ways the ledger couldn't defend itself

What instrumenting a randomized experiment actually requires

Model or Method? Building a Deterministic Lab to Measure an AI Agent Fleet's Own Behavior

The trap: "quality is the method" is unfalsifiable until you measure both axes

Why measuring an agent fleet is harder than it looks

The method fingerprint: what discipline did this session actually use?

The first lesson wasn't in the data — it was to distrust our own catalog

Honest limitations (the part that keeps it a science)

How to Stop AI Agent Skills, Hooks, and Cron Jobs from Silently Conflicting Over Where They Run and What Data They Trust

The failure mode: components that work alone but leak when combined

The fix: four invariants every component must declare

Two real leaks, one missing declaration

Where the declarations live, by component type

Backfilling existing components and keeping it honest

Honest limitations

Do Multiple Personas on One LLM Give Real Diversity, or Do You Need Different Model Families?

The short answer: same weights, same blind spots

The measured ceiling from an 8-round self-audit

Three axes of real diversity

The Aggregator Bottleneck

The ΔEVD test: is your reframing real or theatrical?

Honest limitations and when persona councils are still fine

Claude Code '400: no low surrogate in string' on every turn: repairing a permanently broken session transcript

Why every turn fails: the surrogate is in the transcript, not the network

The safe repair: strip only U+D800–U+DFFF, leave real emoji intact

Close the session before you touch the file

Make it cheap enough to auto-repair on every session start

If an LLM Extracts the Inputs, Is Your Deterministic Score Really Deterministic? Stopping Provenance Laundering

The trap: "the LLM only extracts, Python decides" is not enough

Three laundering paths a deterministic score is exposed to

The fix: three rules plus one asymmetry

Pin the defense with an attack matrix regression test

Honest limitation: code cannot close the injection boundary

macOS: nslookup works but curl and Python "Could not resolve host" — the mDNSResponder zombie

The two-DNS-path asymmetry (your fastest diagnosis)

Confirm it in 10 seconds

Fix it weakest-tool-first

Why agents and long-lived dev boxes hit this repeatedly

"Disabled" in config does not mean the process is dead

Prevent recurrence — and the honest limitation

A file-based work-bus for orchestrating a fleet of agent CLIs — coordination without a message broker

The mechanism

This is state, not events — by design

Routing by capability, not by name

Graceful degradation: skip the absent worker

Trust the bus like a network boundary

The honest limitation

The silent failure: `cache_read` stays zero and nothing tells you