DEV Community: Jeremy Longshore

Run the Readiness Audit Before You Flip DNS

Jeremy Longshore — Sat, 04 Jul 2026 13:00:29 +0000

The cutover that would have taken the money and then broken

The DiagnosticPro migration moved a live product off Firebase, Firestore, GCP, and Vertex AI onto a single self-hosted VPS. New database engine, new secrets model, new LLM client, new proxy, new deployment shape — the whole substrate replaced at once. The plan ended with the usual last step: flip diagnosticpro.io DNS from the old host to the new one and watch the traffic move over.

That last step is the one that cannot be un-done cheaply. The moment DNS propagates, real customers hit the new stack with real credit cards. Everything before that moment is reversible: the old host is still authoritative, rollback costs a config revert. Everything after it is a live incident.

So before the flip, the new stack was put through an adversarial readiness audit: not a smoke test against the happy path, but a deliberate, multi-lens attempt to find how the migration would fail in production — run against the deployed stack while the old host was still authoritative and rollback was still free. It found a failure, and it was the worst possible kind.

The single most damning finding

The live VPS database was missing every column the payment and membership write-paths depended on — nine columns on the submissions table, three on the analyses table. The schema the code was written against and the schema actually deployed on the box had drifted apart.

Trace what that means through a real Stripe checkout:

Customer enters card details and completes checkout. Stripe charges the card.
Stripe fires checkout.session.completed to the webhook.
The webhook handler tries to write the submission row — including columns that do not exist on the live table.
The write throws. The handler returns HTTP 500.

The customer has already been charged. The system then errors out and cannot record the purchase, cannot queue the diagnostic, cannot deliver the report. This is the worst kind of failure: the money moves, the system errors, and no record survives on either side. Not a declined card, not a graceful "try again" — a completed charge followed by a server error, with no record on your side that the transaction happened.

Every single paid checkout after the DNS flip would have hit this. Not an edge case. The main revenue path, guaranteed to fail, on a stack that had already collected the customer's money.

The fix: migrate the schema on boot, idempotently

The repair was not a one-time hand-run ALTER TABLE on the box — that fixes today's database and silently rots the next time a fresh environment comes up. The fix was to make the application upgrade its own schema on startup, so any database it boots against converges to the schema the code expects.

The migration reads the current shape of each table, compares it to what the code needs, and applies only the missing changes:

function migrateSchema(db) {
  const cols = db.prepare("PRAGMA table_info(submissions)").all();
  const have = new Set(cols.map((c) => c.name));

  const wanted = {
    stripe_session_id: "TEXT",
    payment_status:    "TEXT",
    membership_tier:   "TEXT",
    // ...the rest of the drifted columns
  };

  for (const [name, type] of Object.entries(wanted)) {
    if (!have.has(name)) {
      db.exec(`ALTER TABLE submissions ADD COLUMN ${name} ${type}`);
    }
  }
}

PRAGMA table_info gives the current columns; the loop only issues ALTER TABLE for the ones that are absent. That have.has(name) guard is load-bearing: SQLite's ALTER TABLE ADD COLUMN has no IF NOT EXISTS clause before version 3.37, so it throws if the column is already there — the idempotency lives in the JavaScript check, not the SQL. Run it twice and the second pass is a clean no-op. A brand-new database converges to the full schema; an old drifted one gets exactly the missing columns added in place. (The ${name} ${type} interpolation is safe only because both come from a hardcoded object in the same file — never substitute column names or types from untrusted input; SQLite won't bind them as parameters.)

Verified against the live VPS database, it applied 12 migrations in place — the exact drift the audit had predicted, closed before a single customer touched the new stack. A regression test locks the behavior in: create an old-shape database, boot the app, assert the columns exist, boot again, assert the second boot is a clean no-op.

What else the audit dragged into the light

The schema drift was the headline, but an irreversible cutover has more than one way to go wrong. The audit was multi-lens on purpose — many independent passes, several review angles, each finding adversarially re-verified rather than trusted on first sight. Several of the others were also invisible until someone actually paid.

A GCP secrets client on a non-GCP host. The code carried a Google Secret Manager client inherited from its Firebase/GCP life. On a GCP host that client silently uses the platform's identity. On a self-hosted VPS there is no GCP metadata server — so the client hunts for one, and that hunt can hang process startup while it waits on a network endpoint that will never answer. You inherit this hazard for free when you self-host code that assumed it was running inside Google's platform. The fix was an env-first secrets model: secrets materialized at deploy time from an encrypted store into the process environment, and the cloud SDKs removed entirely — @google-cloud/secret-manager and google-auth-library deleted, pruning 63 npm packages from the install.

A dead gateway URL on the success page. The deployed frontend bundle was still calling a decommissioned GCP gateway host — a stale fallback URL baked into the post-payment success page. It would only fire after a customer paid, which is precisely why a normal click-through of the site never surfaced it. The path a paying customer takes is often the least-tested path on the whole site.

A login guaranteed to throw. The Whop login flow referenced an undeclared membership variable — residue from the deleted Firestore code path. Every login attempt would hit a ReferenceError and break. Not intermittent, not conditional: a guaranteed crash on a core flow, left behind by the migration itself.

Webhook replay with no idempotency guard. Stripe re-delivers webhooks; checkout.session.completed can arrive more than once for the same session. Without a guard, a duplicate delivery re-queues the diagnostic work and re-runs the LLM — double cost, double side effects. The fix keys on the session ID and treats any already-seen event as a no-op acknowledgement.

Two testing traps the cold-CI sweep caught

The same discipline — replicate the real environment instead of trusting the local one — surfaced two test bugs a warm laptop hides:

An exact-string environment gate. The backend gates its mock-LLM path on the exact string 'true'. The end-to-end test set TEST_MOCK_LLM=1. Environment variables are always strings in Node, so the gate compares "1" !== 'true' — the test would have driven a real, keyless LLM call and failed the full-flow run for a reason that had nothing to do with the code under test. The lesson is unforgiving and portable: know exactly which value your gate compares against. "1", "true", "yes", and true are four different things, and a strict-equality check honors exactly one of them.

"Passes locally" that CI never saw. A ts-jest inline tsconfig omitted lib. TypeScript derives its default lib from target, and target: es2022 defaults to ["ES2022"] — which has no DOM. Locally the project-level tsconfig still supplied DOM, so compilation passed; the inline ts-jest config didn't inherit it, so CI type-checked against the bare es2022 defaults and the DOM globals — window, IntersectionObserver — failed to resolve. The green checkmark on the laptop came from config the CI transform never read. The fix pinned lib: ["ES2022", "DOM", "DOM.Iterable"] and turned on isolatedModules. A related trap in the same suite: the Playwright job ran vite preview without building dist first, so the preview server errored with "directory dist does not exist." Both share a moral — "it works on my machine" can quietly mean "it works with config and build artifacts CI never sees." CI is cold and empty by design — which is exactly why an adversarial audit replicates the deploy environment instead of reading the code and trusting it.

The remediation discipline

The audit ran independent, adversarial review of the new stack before the irreversible step, while rollback was still free. Many separate passes, several review lenses, every finding re-verified against the actual live artifacts rather than the code as written. The drift between "what the code assumes" and "what is actually deployed" is exactly the gap that a happy-path smoke test steps right over.

Finding the bugs was half the job. The other half was making sure they stayed fixed. Before the flip, a revenue-path test suite was installed and gated in CI:

7 backend test suites, 92 tests total.
Stripe — first delivery, replay/idempotency, signature validation, and the checkout-session contract.
Whop — Standard-Webhooks verification, legacy HMAC, a length-mismatch path that must return 401, and the OAuth flow.
Reports routes, plus a rate-limit test asserting the 11th submission returns 429.
Schema-migration regression — old database upgrades cleanly, and a double boot is idempotent.
PDF generation — output exceeds 10 KB and starts with the %PDF- header.
Backend coverage: 84.6% of lines, with a 60% floor gated in CI.

The pre-remediation grade was recorded as D+ in a TEST_AUDIT.md. That honest starting grade mattered: it named the gap in writing so the revenue-path P0s could be tracked and closed rather than waved through. By the flip, every revenue-path P0 was closed.

The stack the audit ran against

For context on the scope of the drift, here is the substrate the traffic landed on — every layer new relative to the Firebase/GCP original, which is why nothing about the old deployment could be assumed to still hold:

SQLite via better-sqlite3 in WAL mode for the database, with the local filesystem for artifacts — no managed database, no object store.
OpenAI gpt-4o through an OpenAI-compatible client. The old callVertexAI became callLLM; the provider is now a pure environment swap, not a code change.
Env-first secrets, materialized from an encrypted (SOPS) source at deploy time — no cloud secrets SDK in the running process.
A multi-stage Dockerfile with docker-compose, fronted by Caddy proxying /api on the VPS, on Node 20.
Firebase and Firestore deleted from the client entirely, and every Google Cloud dependency removed.

The result: diagnosticpro.io DNS was flipped off Firebase and went live on the VPS on 2026-07-01.

Before you flip DNS — a short checklist

Take this to your own cutover:

Audit the deployed artifact, not the source. The schema drift was invisible in the code — it lived in the gap between the repo and the live box. Point your verification at what is actually running.
Walk the money path end to end. The success page, the webhook write, the receipt — the paths a paying customer takes are often the least-exercised on the whole site, and the most expensive to get wrong.
Grep the new host for assumptions from the old one. Cloud SDKs, metadata-server calls, gateway URLs, identity clients — anything that quietly depended on the previous platform is now a boot hazard or a dead call.
Make schema changes converge, not command. A boot-time idempotent migration fixes every environment forever; a hand-run ALTER TABLE fixes exactly one database until the next fresh deploy rots it.
Trust cold CI over a warm laptop. "Passes locally" can mean "passes with config and build artifacts CI never sees." Run it clean before you believe it.

The rule

The moment before an irreversible cutover is the cheapest time you will ever have to find catastrophic bugs. After the DNS flip, the schema drift is a live payment incident with charged customers and no records. Before it, it is a diff. The distance between those two costs is one adversarial audit of the new stack plus a revenue-path test suite — run while rollback is still nothing more than reverting a config. Buy the safety while it is free.

Surviving CoreWeave: the GPU failures that burn your hours

Jeremy Longshore — Thu, 02 Jul 2026 22:13:15 +0000

Your 64-GPU job just threw Xid 94. Reschedule, or is the node dead?

It comes down to one bit, and most people get it wrong. They see "memory error," panic, drain the whole node, and lose an hour babysitting a machine that was fine. Or they do the opposite — retry blindly onto a GPU that's quietly returning corrupt gradients.

CoreWeave gives you world-class hardware with very sharp edges. The failure modes aren't exotic. They're just undocumented in the place you're looking when a run dies at 2 a.m. Here are the five that burn the most hours, and how to read them fast.

Node forensics: read the Xid before you touch anything

When a GPU misbehaves, the driver logs an Xid to dmesg. The Xid number is the whole story. Learn three of them and you'll triage 90% of node incidents without a support ticket.

Xid 79 — "GPU has fallen off the bus." The driver can't reach the GPU over PCIe anymore. The NVIDIA catalog's recommended action is RESTART_BM — restart the bare metal. The card is gone until the node reboots. Don't fight it; the node needs a power cycle.
Xid 94 — "Contained memory error." A memory error the hardware isolated to one application. Action: RESTART_APP. Every other process on that GPU keeps running. You restart your job and move on. The node is not dead.
Xid 95 — "Uncontained memory error." The error escaped the faulting process. Other work on that GPU may have read corrupt data. Action: RESET_GPU. Now the GPU is suspect and everything that touched it is suspect with it.

That's the bit everyone gets wrong. 94 is contained — reschedule. 95 is uncontained — the GPU is out. Same three words in the log ("GPU memory error"), opposite responses.

The other pair worth memorizing is Xid 63 / 64 — row remapper events. Ampere and newer GPUs quietly retire bad memory rows. When you suspect a flaky card, read the remap state directly:

nvidia-smi -q -d ROW_REMAPPER

Two fields decide the verdict. Pending: Yes means a remap is scheduled but won't take effect until the GPU is reset — the repair is durably recorded, just not live. Remapping Failure Occurred: Yes is the one that ends the conversation: the remap couldn't be written (that's the Xid 64 case), and the card needs to come out of service. Pending is routine. Failure is terminal.

One CoreWeave-specific rule sits on top of all this: never manually uncordon a health cordon. CoreWeave's Node Life Cycle controller cordons nodes automatically on GPU hardware errors and reboots them. Those node conditions are internal machinery, not a knob for your automation. If you kubectl uncordon a node CoreWeave cordoned for a hardware fault, you're scheduling work back onto a known-bad card. Let the controller clear it.

The fabric lies quietly: RDMA that fell back to TCP

This one doesn't crash. It just makes your training 5x slower and says nothing.

GPUDirect RDMA over InfiniBand is what makes multi-node training fast. When it silently falls back to TCP sockets, your all_reduce still completes — over Ethernet, at a fraction of the bandwidth. No error. No log line screaming at you. Just a job that's mysteriously behind schedule.

Three things have to be true, and all three fail quietly if you miss one.

Request the InfiniBand resource in both requests and limits. It's a boolean scheduling flag, not a count:

resources:
  requests:
    rdma/ib: 1
    nvidia.com/gpu: 8
  limits:
    rdma/ib: 1
    nvidia.com/gpu: 8

Miss it in either block and your pod schedules onto a node without IB, and NCCL shrugs and uses TCP.

Point NCCL at the right interfaces. On CoreWeave that's:

export NCCL_IB_HCA=ibp
export NCCL_SOCKET_IFNAME=eth0

Then confirm it actually took. Turn on debug output and read one line:

export NCCL_DEBUG=INFO

Look for NET/IB. That's InfiniBand — good. If you see NET/Socket, you're on TCP and every second of that run is wasted. CoreWeave's own docs put it plainly: if a multi-node job runs but throughput is far below expectations, NCCL has usually fallen back from InfiniBand to TCP.

Prove the fabric before you trust it. Run all_reduce_perf from CoreWeave's nccl-tests and check the reported bus bandwidth. On a 64-GPU H100 run over InfiniBand you should see hundreds of GB/s — the exact number moves with your NCCL version, so baseline against CoreWeave's published manifests, not a screenshot from someone's blog. If your busbw is an order of magnitude low, you already know why: go back and read the NET/ line.

Launch: MPIJob, not PyTorchJob

The obvious path — Kubeflow's PyTorchJob — is not the path CoreWeave paves. Their reference manifests for multi-node NCCL runs use the MPI Operator (MPIJob), with SUNK/Slurm as the other supported route. Follow the paved road; the sharp edges are already sanded off it.

Two traps live here.

slotsPerWorker must equal the GPUs per pod. The MPI Operator launches one rank per slot. Set eight GPUs per worker and four slots, and mpirun spins up the wrong number of ranks — half your GPUs sit idle while the job reports "running."

Don't reach for the SUNK pod scheduler to gang-schedule a distributed job. Its own docs are blunt: "No gang scheduling. The scheduler schedules each Pod as a separate Slurm job. Multi-node PodGroups aren't supported." It's built for single-node work like inference. Point it at a multi-pod training job and you get piecemeal scheduling — some ranks start, the rest stay Pending, and the ranks that started block forever at NCCL rendezvous waiting for peers that were never scheduled. Your job hangs at init with no error. Use MPIJob or a real Slurm allocation, which grab the whole gang at once.

Storage: the endpoint decides your throughput

CoreWeave AI Object Storage is S3-compatible, so your existing code "just works." That's the trap — it works at the wrong speed if you use the wrong hostname.

Inside the cluster, hit http://cwlota.com. That's LOTA, the Local Object Transport Accelerator — a proxy that caches objects on the GPU nodes' local NVMe. The public https://cwobject.com endpoint works too, and it'll route you around the accelerator you're paying for. In-cluster: cwlota.com. Full stop.

LOTA has rules. It only caches objects larger than 4 MB — anything smaller bypasses the cache entirely. So the classic "millions of tiny files" dataset gets zero benefit. Consolidate small samples into larger archives (WebDataset, TAR, TFRecord) so your objects clear the 4 MB bar, and use a minimum 50 MB multipart part size to cut request overhead.

The payoff is real once the cache warms. CoreWeave's own benchmark on 160 H200 GPUs measured about 24 GiB/s aggregate cold (first minute, cache empty) climbing to 368 GiB/s warm — roughly 2.3 GiB/s per GPU. Cold reads are network-bound; warm reads come off local NVMe. Structure your data loader to warm the cache early.

For weights, Tensorizer loads faster than the standard path — CoreWeave clocked GPT-J-6B at ~8.2 s median versus ~15 s for HuggingFace on A40s, by streaming tensors at wire speed instead of deserializing a checkpoint blob.

For resilience, use PyTorch Distributed Checkpoint. torch.distributed.checkpoint.async_save writes checkpoints on a background thread so training barely stalls, and DCP's load-time resharding lets you save on one topology and resume on another. That last part matters here: when CoreWeave cordons a node mid-run and you come back on a different world size, a DCP checkpoint loads anyway. A rigidly-sharded one doesn't.

Cost: there is no dashboard — you build it

Here's the one that surprises every finance-conscious team: CoreWeave ships no cost dashboard and no billing API. You want to know what you're spending? You build it yourself, in PromQL, against their managed Grafana.

The usage metrics are there. The one you start with is billing:instance:total — running instances by cluster — alongside billing:object_storage_used_bytes:total and friends. Multiply usage by your rate card inside the query to synthesize a cost estimate; there's no single "dollars" metric to read. You'll need to be in the admin, metrics, or write group to see them.

Two levers keep the number down.

Reserve carefully. Reserved capacity runs 25–60% off on-demand, with the deepest cuts on multi-year H100/H200 commitments. But reserved means paid whether or not you use it — an idle reservation is worse than on-demand. Reserve only your steady-state floor; burst the rest.

Right-size the GPU to the model. An H100 is wasted on small-model inference. Third-party 2026 benchmarks consistently put the L40S ahead of the H100 on cost-per-token for models in the ~7B–30B range — reach for the H100 only when you genuinely need ultra-low latency. (Treat the exact dollar figures as directional; they swing hard by benchmark and by month.) And remember the hardware line: FP8 lives on Hopper and Ada, not Ampere — if your inference plan is FP8, an A100 can't run it, so match the silicon to the numeric format before you commit.

The short version

Read the Xid first. 79 → node reboot. 94 → contained, restart your app. 95 → uncontained, the GPU is out. nvidia-smi -q -d ROW_REMAPPER: Pending is fine, Remapping Failure is terminal. Never uncordon a health cordon.
Confirm NET/IB, not NET/Socket. RDMA fails silent and slow. rdma/ib: 1 in requests and limits.
Launch with MPIJob. slotsPerWorker = GPUs per pod. The SUNK pod scheduler won't gang-schedule — your job will hang at rendezvous.
Use cwlota.com in-cluster. Keep objects above 4 MB, parts above 50 MB. async_save + DCP resharding survives node loss.
Build your cost view in PromQL. No dashboard exists. Reserve only your floor; put small-model inference on cheaper silicon.

None of this is in one place, which is exactly why it costs people days. Now it's in one place.

Jeremy Longshore builds Claude Code skills for infrastructure platforms at Intent Solutions — the CoreWeave GPU-ops pack lives in claude-code-plugins. Community-contributed. Not affiliated with, endorsed by, or sponsored by CoreWeave, Inc.

Gate the Statement, Not the Tool Name

Jeremy Longshore — Thu, 02 Jul 2026 21:35:25 +0000

The original safety gate on the Dolt-over-MCP plugin tried to keep a Claude Code agent harmless by excluding "history-affecting tools" from its MCP grant. It was the wrong granularity, and it did nothing.

MCP exposes the entire database through one tool — query / exec — and that tool carries every SQL verb. SELECT rides it. So does CALL DOLT_PUSH, CALL DOLT_RESET('--hard'), DROP DATABASE, and CALL DOLT_BRANCH('-D', 'main'). Excluding "dangerous tools" from the grant accomplishes nothing, because the dangerous verbs live inside the one tool you already granted. The destructive operations were never separate tools to exclude.

This is the reframe the whole Phase 0 hardening pass turned on: a tool-name allowlist is meaningless for any tool that carries a sub-language. SQL is a sub-language. So is the shell behind a Bash tool. So is anything behind an eval. If the tool can run arbitrary statements in some grammar, the only boundary that means anything is one that reads the statement. It is the move from tool-name allowlisting to capability-based security: the grant stops being "you may call the query tool" and becomes "you may run these statement classes inside it."

Why not just allowlist the safe tools?

Because there is exactly one tool, and it is not safe or unsafe — it is whatever statement you hand it. You cannot partition a single door into a safe door and a dangerous door by naming. The same logic kills the next-obvious fix: a denylist of dangerous verbs. Blacklist DOLT_PUSH, DOLT_RESET, DROP... and miss DOLT_REBASE, or the proc Dolt ships next quarter, or a CALL whose name your regex didn't anticipate. A denylist is only as good as your imagination on the day you wrote it.

The fix inverts that. You add safety by enumerating what is safe, not by blacklisting what is dangerous. Anything you cannot positively classify as safe is treated as the most dangerous thing it could be. Default-deny the unknown. It's least privilege applied to a grammar: the agent gets only the verbs it can prove it needs.

The classifier: three verb classes, decided before the server sees it

scripts/sql_classifier.py is 259 lines of pure stdlib. No import side effects, so the 22 unit tests in tests/test_sql_classifier.py import it directly and hammer it in isolation. scripts/dolt-mcp-client.py makes it the chokepoint — every statement is classified before it reaches the dolt-mcp server, into one of three classes:

read — SELECT / SHOW / DESCRIBE / EXPLAIN / read-only table functions → executes freely.
safe-write — INSERT / UPDATE / DELETE / CREATE TABLE / CALL DOLT_COMMIT / DOLT_CHECKOUT / DOLT_BRANCH (create) → executes only on an agent-owned branch (never main) and only under --allow-mutation. On pre-GA / alpha database flavors it's refused entirely — read-only there.
history-affecting — CALL DOLT_PUSH / DOLT_PULL / DOLT_MERGE / DOLT_REBASE / DOLT_RESET('--hard') / branch-or-tag delete / DROP DATABASE / GRANT / any unknown CALL … → always refused. The classifier is recommend-only here: it surfaces the exact command it would have run and a human runs it.

The agent gets to mutate its own scratch branch. It never gets to rewrite shared history. That line is drawn by reading the verb, not by trusting a tool grant.

The fail-safe details are the whole point

A statement-level gate is really an input-validation boundary — every statement is validated before it executes — and a classifier is only as good as its failure mode. This one fails closed, and the details are where a naive regex gate quietly gets it wrong.

Default-deny the unknown. Any CALL with no resolvable procedure name, and any unrecognized CALL DOLT_*, is classified history-affecting — refused. When Dolt ships a new stored proc, it lands on the deny side automatically, with no code change. That's the enumerate-the-safe principle paying rent.

Batch = max severity. A multi-statement batch is classified at the severity of its most dangerous statement. A read prefix cannot smuggle a write past the gate:

# A batch is as dangerous as its worst statement.
# "SELECT 1; CALL DOLT_PUSH(...)" classifies as history-affecting, not read.
return max(
    (classify_statement(s) for s in split_statements(sql)),
    key=lambda c: SEVERITY[c],
    default="read",
)

Comment-stripping is quote-aware. /* */, --, and # comments are stripped before classification, so a verb hidden behind a comment can't mask the real leading verb. But string literals are preserved — which matters more than it looks. The --hard inside CALL DOLT_RESET('--hard') must not be mistaken for the start of a -- line comment. Get that wrong and a hard reset reads as a soft one. The contract:

def strip_sql_comments(sql: str) -> str:
    """Remove -- , # , and /* */ comments. Quote-aware.

    Inside a '...' or "..." string literal, comment markers are
    inert: the --hard in CALL DOLT_RESET('--hard') survives intact.
    Backslash and doubled-quote escapes are honored so a quote inside
    a literal doesn't prematurely end it.
    """

DOLT_RESET is severity-split on its argument. Soft reset is safe-write. --hard is history-affecting. Same proc name, two classes, decided by reading the argument the literal preserved above.

Cannot prove it's a read → at least safe-write. Ambiguity loses. A WITH (CTE) resolves to whatever it ultimately wraps — WITH x AS (...) SELECT is read; WITH x AS (...) DELETE is safe-write. The classifier never guesses in the agent's favor.

The Bash door: the §10 union gate

Hardening the MCP path left a second door open. The original safety check inspected only mcp__* grants. It was blind to the fact that an agent could still be handed Bash(dolt:*) or Bash(bash:*) and reach dolt push — or anything — straight through the shell. Same destructive operation, different surface, completely unguarded.

scripts/check-agent-safety.sh is the CI gate that closes it. It asserts the mutation-verb taxonomy across both surfaces — every agent .md and the core SKILL.md:

No Bash(<cmd>:*) wildcard that can reach a history-affecting op. bash/sh are arbitrary by definition; dolt/bd/bd-sync/git reach push / reset / branch -D / killall. Banned.
No granted MCP tool outside the read/safe set — so a future …__exec / …__merge / …__push / …__reset grant fails the build the moment someone adds it.

The subtlety that makes it correct: it scans the allowlist only, never the denylist. A destructive pattern in disallowedTools is the mitigation, not a violation — flagging it would be backwards. The gate only cares what a config permits.

# Scan ALLOWED grants only. A destructive pattern under
# disallowedTools is the fix, not the finding.
grep -oE 'Bash\(([^):]+):\*\)' "$agent_md" | while read -r grant; do
  cmd=$(printf '%s' "$grant" | sed -E 's/Bash\(([^):]+):\*\)/\1/')
  case "$cmd" in
    bash|sh|dolt|bd|bd-sync|git)
      fail "$agent_md grants Bash($cmd:*) — reaches a history-affecting op" ;;
  esac
done

That landed by replacing Bash(bash|dolt|bd:*) wildcards in 5 agents plus the SKILL.md with explicit read-only subcommand allowlists. Wildcards are an unbounded grant; an enumerated subcommand list is a bounded one.

Invariants become mechanisms, not comments

A second blocker had the same shape at a smaller scale: scripts/dolt-push-dolthub.sh documented its safety invariants in comments and trusted them. A safety invariant written only in a comment is not enforced. So the comments became mechanisms:

A failed bd export used to be swallowed with || true. Now a failed flush aborts the push — you never push on an unverified flush.
A flock idempotency guard makes overlapping scheduled runs a no-op, so a double-fire can't double-apply.
On an ambiguous push failure, it polls the DoltHub SQL API for the real terminal state instead of blind-retrying into a possible double-push.

And the supply chain got pinned: dolt-mcp-server@v0.3.6 plus a Go module checksum, consistent across README / SKILL / client. No @latest in anything security-sensitive — @latest means "I'll run whatever you publish next," which is not a thing you say to a binary that can rewrite a database.

The general lesson

Tool-name allowlisting works when each tool is a single, fixed capability. It collapses the instant one tool carries a grammar. SQL over MCP is the case here, but a Bash tool over a shell is the same hole, and so is any eval-style tool that takes a string and runs it. For those, the tool name tells you nothing about what's about to happen. Only the statement does.

So gate the statement. Enumerate the safe verbs, default-deny everything you can't prove safe, classify batches at max severity, and make sure your parser is honest about quotes and comments — because the one place a lazy gate breaks is the --hard it mistook for a comment.

Also shipped

governed-second-brain — /teamkb-compile, a nightly job that compiles the day's work into the governed team brain (auto-graduates itself, fixed a tenant-mismatch bug); a follow-up review locked down the scratch dir and made paths portable with glob/dir guards.
intent-mail — migrated to React 19 + Ink 7, removed a dead @anthropic-ai/claude-agent-sdk integration, batch-adopted 11 gate-passing Dependabot bumps, and made the OSV scan report-only to kill a phantom red check.
claude-code-plugins — databricks-pack was the Killer Skill of the Week (W27); the dolt-mcp-vcs rename landed as a non-breaking install-slug alias, so the old beads-dolt slug still resolves.

{
"@context": "https://schema.org",
"@type": "BlogPosting",
"headline": "Gate the Statement, Not the Tool Name",
"description": "When one MCP tool carries every SQL verb, allowlisting tool names is theater. The safety boundary has to read the statement — here's how that gate was built.",
"datePublished": "2026-06-29T08:00:00-05:00",
"author": {
"@type": "Person",
"name": "Jeremy Longshore",
"url": "https://startaitools.com/about/"
},
"publisher": {
"@type": "Organization",
"name": "StartAITools",
"url": "https://startaitools.com"
},
"articleSection": "Technical Deep-Dive",
"keywords": "ai-agents, claude-code, security, mcp, architecture",
"mainEntityOfPage": {
"@type": "WebPage",
"@id": "https://startaitools.com/posts/gate-the-statement-not-the-tool-name/"
}
}

Coverage Said 69%, Mutation Testing Said 25%

Jeremy Longshore — Wed, 01 Jul 2026 13:00:26 +0000

Sunday 2026-06-28. intent-mail repo, fresh Stryker baseline run. The coverage gate reported green: 69.09% line coverage. Three seconds later, mutation testing reported 24.88%. The rules engine—the code that actually mutates user email—reported 0.00%.

That zero is the story.

The Gap

Coverage counts lines. Mutation testing counts assertions. When a 535-line end-to-end suite exercises a piece of code but doesn't assert on its internal logic, coverage sees a line executed and calls it a win. Mutation testing inverts a single boolean operator in that line, runs the suite again, and if the outcome is the same, marks that mutant as survived. The engine had 301 mutants and zero of them were killed—not because there were no tests, but because the tests that ran the code never asserted on the code's decision logic.

Here's the real shape of it:

File	Line Coverage	Mutation Score	Killed	Survived	No-coverage Mutants
`connectors/shared/retry.ts`	—	76.27%	44	9	5
`storage/token-crypto.ts`	—	52.53%	52	17	30
`ai/daily-digest.ts`	—	34.66%	61	85	30
`rules/engine.ts`	—	0.00%	0	0	301
Overall repo	69.09%	24.88%	157	111	366

The engine had no co-located unit tests. It was exercised end-to-end—so it counted toward line coverage—but its specific logic was never pinned. Flip a condition in the engine, and the outcome (was the email moved?) often stays the same. The assertion lives at the wrong level.

Code Coverage vs. Mutation Testing: What's the Difference?

Code coverage counts whether a line executed. Mutation testing counts whether that line's behavior is actually asserted. Run both on one repo and you can get 69% line coverage beside a 24.88% mutation score—because a line can execute in dozens of tests while none of them pin down what it should do.

A mutant is a one-character change: === becomes !==, && becomes ||, a > becomes >=. Stryker performs fault injection—it injects the mutant, runs the test suite, and counts:

Killed: a test failed (the mutant was caught).
Survived: all tests passed (the mutant hid).
No-coverage: no test executed that line at all.

The 301 mutants in the engine were all no-coverage. Stryker never even got to run them against a test, because there was no unit test visiting that file.

Meanwhile, the repo's overall line coverage sat at 69.09%—and the engine's lines were executed via the E2E path, counting toward that number. Coverage says "this line ran." Mutation testing says "this line's behavior is pinned down by assertions"—call it assertion coverage. They are not the same metric, and a green CI run only tells you about the first.

The Stryker Setup

{
  "packageManager": "npm",
  "testRunner": "vitest",
  "reporters": ["html", "clear-text", "progress"],
  "coverageAnalysis": "perTest",
  "mutate": [
    "src/rules/engine.ts",
    "src/connectors/shared/retry.ts",
    "src/storage/token-crypto.ts",
    "src/ai/daily-digest.ts"
  ],
  "thresholds": {
    "high": 80,
    "low": 60,
    "break": null
  }
}

The mutate array targets four high-value pure-logic files. Stryker baseline runs in ~30 seconds. The break: null is deliberate: report the score, but do not fail CI. Establish a baseline first. Ratchet later.

This is the inverse of the "fail immediately on every finding" instinct. A fresh gate that blocks on day one gets disabled by the next engineer. Report-only first. Let the team see the numbers. Then make it enforceable—and honor the gate when its verdict is inconvenient.

Why E2E Wasn't Enough

The same day, a 535-line end-to-end suite was added:

describe("rules engine E2E", () => {
  const tmpDir = mkdtempSync(join(tmpdir(), "intentmail-"));
  const dbPath = join(tmpDir, "test.db");
  const testMasterKey = "e".repeat(64); // 64-char hex = 32-byte AES-256 key

  beforeAll(() => {
    process.env.INTENTMAIL_DB_PATH = dbPath;
    process.env.INTENTMAIL_MASTER_KEY = testMasterKey;
  });

  it("applies a rule and writes an audit log", async () => {
    // Create account → upsert emails → create rule → run it → assert outcome
    const emails = await db.getEmailsByRule(ruleId);
    expect(emails).toHaveLength(1);
    expect(emails[0].labels).toContain("archived");

    const auditLog = await db.getAuditLog(ruleId);
    expect(auditLog).toHaveLength(1);
    expect(auditLog[0].action).toBe("move");
  });
});

This is good and necessary. It proves the wiring: condition → action → side effect. But it asserts on the outcome (was the email moved?), not the logic (did the condition comparison return true or false for this specific case?).

Here's the subtle part. The engine's 301 mutants didn't even survive—they came back as no-coverage. At baseline time, Stryker found no co-located unit test pinning engine.ts, so it never ran those mutants against an assertion at all. The E2E suite executes the engine, but it asserts through the storage layer on the final state—which is exactly why the documented next step is a co-located engine.test.ts, not more end-to-end tests.

The other three files show the milder failure mode—a mutant that is covered but still survives. When a test executes a mutated line yet only checks the outcome, the mutant lives. Take a condition in the engine:

// Original
if (email.subject.includes(rule.condition.value)) {

// Stryker mutant: flip the condition
if (!email.subject.includes(rule.condition.value)) {

An E2E test that asserts only on the final database state can let this flip through. Unless a fixture exercises both the matching and the non-matching branch—full branch coverage—and checks each one, inverting the condition can still leave the rows where the test expects them. The test never asserted "for this subject, the condition must return true"—so the mutant lives. That's the failure mode that left 85 mutants alive in daily-digest.ts, where tests did run. The engine's case is worse: no unit test ran at all.

Line coverage: ✓ (the line executed via E2E).

Mutation coverage: 0 (the logic was never asserted at the unit level).

Why Not Just Block On It?

Because a gate that reports a baseline truth on day one and gets disabled by day three is worse than no gate at all. The 24.88% score is real and uncomfortable. Flip break from null to 60, and the suite fails immediately. The next engineer to touch this codebase sees a broken CI, disables the gate, deletes the stryker script, and ships. The baseline evaporates.

Instead: report-only for one development cycle. Let the team see the number. Add co-located unit tests to src/rules/engine.test.ts until the engine clears low: 60. Then flip break: 60 and make it a ratchet: the mutation floor can only move up.

This is the same discipline as the L5 security scan wired the same day—gitleaks + OSV, both report-only, both feeding into tests/TESTING.md as the single source of truth for "which gates are enforced now, which are baseline, which are deferred?"

Also Shipped

The same day closed two beads: GCP Deployment (won't-do, 592 lines deleted—deploy.yml, drift.yml, infra/), and Test Coverage (paid down lint debt, promoted no-case-declarations to error, adopted dotenv 17 + commander 15, 9 GitHub Actions bumps, closed children). The security scan job is non-blocking (continue-on-error: true) because 5 fix=NONE advisories live in duckdb's native build chain—accepted and documented until upstreams ship.

The Transferable Rule

Coverage measures attendance. Mutation testing measures whether anyone was paying attention. A green coverage gate tells you code was executed. A green mutation gate tells you code's behavior was asserted. They are not the same gate. If your mutation score is much lower than your line coverage, your tests are outcome-level and your logic is untouched. Add assertions closer to the decision points. Start with report-only. Then ratchet.

Green CI Proves Nothing: Why Your Tests Gate Zero Calls — a passing test suite that asserts on nothing.
Honor the Gate When the Verdict Is Inconvenient — the discipline of trusting the gate's verdict.
When LLM Output Lies Instead of Crashing — the same intent-mail codebase: "it ran without erroring" is not "it's correct."

{
"@context": "https://schema.org",
"@type": "BlogPosting",
"headline": "Coverage Said 69%, Mutation Testing Said 25%",
"description": "A repo at 69% line coverage scored 24.88% on mutation testing—and the rules engine that touches user email scored 0.00%. Coverage said fine; Stryker didn't.",
"datePublished": "2026-06-28T08:00:00-05:00",
"dateModified": "2026-06-28T08:00:00-05:00",
"keywords": "mutation testing, code coverage, mutation score, Stryker, test quality, CI/CD, unit testing",
"wordCount": 1200,
"author": {
"@type": "Person",
"name": "Jeremy Longshore"
}
}

Rent the Agent, Own the Proof

Jeremy Longshore — Wed, 01 Jul 2026 03:08:56 +0000

On June 23, 2026, Anthropic shipped Claude Tag: tag @Claude in a Slack channel and it works as an agentic teammate — running tool calls, following threads, building memory, acting under your org's identity. The legacy "Claude in Slack" chat app retires August 3, 2026; admins get 30 days to migrate. It is, by a distance, the best agentic teammate you can rent.

I want to be honest about that up front, because the interesting question isn't whether Claude Tag is good. It's good. The interesting question is the one it doesn't answer: when an agent has admin-scoped tools in your workspace, who owns the memory it builds, and who can verify the log of what it did?

Give Claude Tag its due

A hit piece would be easy and wrong. Claude Tag's engineering is strong, and the honest version of this argument starts by saying so:

Zero infrastructure. No servers, no deploy, no on-call. Tag the bot, it works. Nothing self-hosted competes on setup cost.
Multiplayer shared context. "One Claude that interacts with everyone" in a channel — anyone can see what it's working on and pick up where the last person left off.
Compounding memory. It "learns over time… builds more context about the work," and can learn across channels with permission.
Central governance. Admins specify which tools and data the model can touch, per channel. Usage is billed to the org. Admins can set token-spend caps per org and per channel.
A genuinely good security model. Work runs in an isolated sandbox on Anthropic's infrastructure. When it needs an external system, requests cross an Agent Proxy where — per launch coverage — credentials "stay in a store and are injected at the network boundary without the model receiving raw keys," with a default policy of deny for un-allowlisted hosts.

That last point deserves emphasis, because it's the part people underrate: keeping raw credentials out of the model and defaulting egress to deny is exactly the posture a security-minded self-hosted stack would build. Anthropic did the hard, correct thing. If you want an agent in Slack tomorrow with no infrastructure, Claude Tag is an excellent product and you should use it.

So this is not "their thing is insecure." It's a different question entirely.

The trade you're actually making

Every one of those benefits shares a root: the substrate is Anthropic's. The sandbox is on their infra. And so is the memory — as AlphaSignal argues, the real Claude Tag question is context ownership: the organizational knowledge Claude builds becomes the vendor's proprietary state, not an exportable dataset your next vendor can ingest. The principle it points to is simple — rent the agent, but own the memory — and Claude Tag inverts it: you rent the agent and the memory lives with the vendor. Critics have named the deeper trap the same way: not model lock-in but context lock-in. And they've been fair about it — the concern isn't that Anthropic is doing anything wrong, it's that the incentives are obvious.

Two more things live on that substrate, and they matter more than memory portability:

Governance is coarse. Admin control means which channels, which tools/data, and spend caps. That is scope plus budget. It is not a documented, mandatory human approval before each consequential tool call executes.
The audit log is the vendor's. Admins "can view a log of everything that @claude has done." You can view Anthropic's record. You cannot verify it, independently, with your own key, with no vendor in the trust path.

For a lot of teams, none of that is a dealbreaker — and they should use Claude Tag. But for a regulated shop, a security vendor, or anyone whose answer to "prove what your agent did" has to survive an adversary who controls the log's storage, coarse governance and a vendor-shown record aren't enough. That's the gap two open-source projects were built to fill.

Why filters were never the boundary

Before the alternative, the premise: for a tool-using agent reading untrusted text — every Slack message is untrusted input — content filtering is not a security boundary. This is true of any Slack-connected agent — CCSC and AGP included, not Claude Tag in particular — which is why the answer isn't a better filter, it's a different architecture. The literature is blunt about this. InjecAgent (Findings of ACL 2024) found ReAct-prompted GPT-4 vulnerable to indirect prompt injection 24% of the time, with private-data exfiltration a primary attack class. Worse, Adaptive Attacks Break Defenses Against Indirect Prompt Injection (NAACL 2025) bypassed all eight evaluated defenses, keeping attack success above 50%. And because tools reached over MCP are "first-class, composable objects with natural-language metadata," MCP Security Bench shows the standard enlarges the attack surface — name-collision, tool-description injection, out-of-scope parameters.

The takeaway isn't "add a better filter." It's that a probabilistic model cannot be the thing that decides whether a destructive action runs. You need a deterministic gate the model can't talk its way past, a human in the loop for the consequential calls, and a record of what actually ran that you can trust without trusting the runtime. That's three properties — and they're the three pillars of what I'll call customer-owned, verifiable governance.

Two reference implementations: CCSC (Claude Code Slack Channel), a small auditable governance kernel that puts Claude Code in Slack behind exactly these gates; and AGP (Agent Governance Plane), which reimplements-and-hardens that kernel with sandboxed execution across multiple agent harnesses. AGP doesn't vendor CCSC or depend on it — it's an independent reimplementation ("adapt-and-harden") pinned to CCSC's v0.10.0 design. Both are open source; you run them on your own infrastructure.

Pillar 1 — Deterministic, fine-grained policy (not a probabilistic scope)

The oversight literature lays out a spectrum, from human-out-of-the-loop through human-on-the-loop. Claude Tag's channel scope sits low on it: a coarse boundary set once, per channel. Fine-grained governance sits higher, and it looks like capability-based security — Miller's object-capability model and the principle of least authority, where authority is granted per operation, not per door. Established capability-security work is explicit that coarse role/scope access over-privileges by construction and that fine-grained capabilities are how you get least authority at scale.

In CCSC, that's a pure decision procedure: evaluate(call, rules, now) maps a tool call to allow | deny | require with no side effects — same inputs, same verdict, every time. Tiered rules resolve strictest-wins, then first-applicable. Out of the box, one tool — upload_file — defaults to fail-closed (denied unless a rule allows it); every other tool follows the policy you author, so making the actions that matter in your workspace fail-closed is a rule you write, not a default you inherit. AGP takes the harder line and bakes it in: its engine is default-deny with deny > require > allow, so a call that matches no rule fails closed to deny, not allow. That's precisely what "AGP hardens the kernel" means. Neither ships an if (model === "claude") branch anywhere near the decision; the gate is data, evaluated before execution.

The difference from a spend cap is the whole point. A token budget limits how much the agent can do. A policy engine decides whether this specific action is allowed — and can force a human into the loop for the ones that matter.

Pillar 2 — Human-in-command approval, per tool call

On the oversight spectrum, the mode that matters for consequential actions is human-in-command: the action does not execute until a human approves it. Not "review the log afterward." Not "set a budget." A blocking approval on this call, before it runs.

CCSC's require verdict routes to a human approver: approval is a reply-code the gate recognizes, with quorum counted by distinct user IDs — and because permission-reply messages from peer bots are dropped at the inbound gate, an injected message can't cast an approving vote. A separate, stronger handshake guards the admin command path (!restart): a single-use 64-bit nonce DM'd out of band, TTL-bounded, checked against wrong-channel replay — built specifically to defeat a same-channel-approval attack, an EchoLeak-class injection where the request and the approval both originate in a compromised channel. AGP renders the per-call approval as Slack Block-Kit Approve/Deny buttons whose nonce is bound to (messageId, sessionId) — and crucially, a bot can never approve, and a bot's click doesn't even burn the nonce.

This is the pillar Claude Tag most visibly doesn't have. Admin scope decides what tools are reachable; there's no public evidence of a mandatory per-action human approval before each consequential call. For a broad class of "summarize the thread" work you don't want one. For "open this PR," "message this customer," "run this migration," a lot of orgs very much do.

Pillar 3 — A signed audit you can verify without trusting the runtime

This is the sharpest line between renting and owning, so be precise about what "verifiable" means. Claude Tag gives admins a log they can view. CCSC and AGP produce a log you can verify — a different verb.

CCSC's journal is a hash chain: each entry commits sha256(prevHash ‖ canonicalJson), so any reordering or edit breaks every downstream link. Each entry is then Ed25519-signed over RFC 8785 JSON Canonicalization, and carries a policy_attestation digest recording exactly which rules were in force when the decision was made. You verify the whole thing offline — bun server.ts --verify-audit-log <path> — with only the public key. No vendor. No live service. Key rotation is supported, so the verifier still works across a rotation. This is the Certificate Transparency design lineage — append-only logs a third party can check without trusting the operator — the same lineage as verifiable ledger databases like GlassDB and recent work on Certificate-Transparency-style provenance for agent action chains, where external verifiers cryptographically validate what an agent did without access to the runtime.

And here's the honest, load-bearing detail most "verifiable log" claims skip: a bare hash chain does not stop truncation. If an attacker drops the last N entries, the remaining chain still verifies clean — CCSC documents this as threat T8, out loud, in its threat model. AGP closes exactly that hole: it writes a signed HEAD checkpoint — a <journal>.head file pinning {seq, hash} — so the offline verifier catches a truncated tail a chain alone cannot. agp verify checks the chain, the signatures, and the signed head — offline, with only the public key. That's what owning the proof buys you: not "trust our log," but "here is a record, check it yourself, and check that none of it was quietly cut off."

The fourth property AGP adds — a sandbox that proves egress is off

Claude Tag's sandbox is on Anthropic's infra. AGP's runs on yours, and it doesn't assume isolation — it proves it. Containers launch with --network none, --cap-drop ALL, no-new-privileges, and pid/memory limits, with no silent fallback to the host if hardened flags fail. Then a network preflight actually runs an egress probe to a TEST-NET-1 address and fails closed — tearing the container down — if traffic is not isolated. Images are pinned — by digest, or at minimum a specific version tag, never latest — and host secrets are mounted through a deny-list. Same "gate, don't impersonate" credential posture as Claude Tag's Agent Proxy: AGP holds no model credentials, and {{secret:NAME}} is resolved only after the gate, at the exec boundary — the journal records secret names, never values.

The part where I don't oversell it

The reason to trust any of this is that both projects are relentless about their own limits, and I'm not going to break that streak here.

CCSC does not protect you against a compromised host OS, a same-UID process, a supply-chain attack, or a socially-engineered operator at their own terminal — its threat model says so explicitly. Its manifest consumer doesn't ship yet, so cross-bot identity is still just bot_id. And T8 truncation, again, is a real gap in CCSC alone — it's AGP that fixes it, which is the whole reason AGP exists.

AGP's sandbox is namespace/cgroup isolation, not a VM — a kernel exploit or container escape defeats it, and the code comments say exactly that. Its multi-harness claim rests on a deterministic reference contract and a conformance test that drives Claude Code and Codex through identical governance; live Codex interception is provisional and not yet CI-validated. AGP's own docs rate the current test suite a B− (78/100). And its marketing-claims scanner bans the words "tamper-proof," "forensic-grade," and "compliance-grade" in CI — which is why I've said signed and offline-verifiable throughout, and not one of those. A log can be signed and still be stolen wholesale if your host is owned. Verifiability is a property of the record, not a force field around your infrastructure.

That discipline is the point. A governance layer that oversells itself is worse than none, because it manufactures the exact false confidence it was supposed to remove.

So: rent, or own?

Both stacks share one architectural signature — prove-don't-trust: a deterministic gate, a human-in-command approval, and a signed record you can check. AGP takes it furthest and makes fail-closed the default: default-deny, no-decision-means-deny, egress runtime-verified off rather than assumed, an unresolved secret throws, an unpinned image throws. CCSC brings the deterministic policy, the human-in-command gate, and the signed record — and leaves which operations fail closed to the policy you author. Put together, it's one posture: don't trust the runtime — make it prove itself, and keep a record you can check. The single sharpest thing they do that a rented, vendor-logged agent doesn't is produce a publicly verifiable, offline audit — and AGP's signed-HEAD checkpoint closes the exact truncation hole CCSC documents. That's the difference between a log you're shown and a proof you hold.

Claude Tag is the best agent you can rent, and for most teams that's the right answer — take the zero-setup, the multiplayer memory, the strong sandbox, and go. But if your governance model needs the agent — and its memory, and its audit trail — to be yours, on your infrastructure, and verifiable with your own key, then you host it. You give up the free memory and the zero-ops UX; you get a substrate no vendor can revoke, relocate, or rewrite.

Both are legitimate engineering positions. The question was never which agent is smarter. It's who has to own the substrate — and who has to be able to prove what happened.

CCSC and AGP are open source: claude-code-slack-channel (the governance kernel) and agent-governance-plane (sandboxed, multi-harness). A companion post, Gate the Statement, Not the Tool Name, walks through the capability-based gate at the statement level.

References

Anthropic (2026). Introducing Claude Tag. anthropic.com/news/introducing-claude-tag
Wulf, J., Meierhofer, J., & Hannich, F. (2025). Architecting Human-AI Cocreation for Technical Services. arXiv:2507.14034 — the human-in-command / human-on-the-loop oversight spectrum.
Zhan, Q., Liang, Z., Ying, Z., & Kang, D. (2024). InjecAgent: Benchmarking Indirect Prompt Injections in Tool-Integrated LLM Agents. Findings of ACL 2024. arXiv:2403.02691.
Zhan, Q., Fang, R., Panchal, H., & Kang, D. (2025). Adaptive Attacks Break Defenses Against Indirect Prompt Injection Attacks on LLM Agents. NAACL 2025. arXiv:2503.00061.
Zhang, D., et al. (2025). MCP Security Bench (MSB). arXiv:2510.15994.
Reddy, P. & Gujral, A. S. (2025). EchoLeak: Zero-Click Indirect Prompt Injection in Microsoft 365 Copilot (CVE-2025-32711). arXiv:2509.10540.
Zigmond, E., Chong, S., Dimoulas, C., & Moore, S. (2019). Fine-Grained, Language-Based Access Control for Database-Backed Applications (ShillDB). arXiv:1909.12279.
Yue, C., et al. (2023). GlassDB: An Efficient Verifiable Ledger Database System Through Transparency. PVLDB. doi:10.14778/3583140.3583152.
Malkapuram, S., Gangavarapu, S., Kavalakuntla, K. R., & Gangavarapu, A. (2025). Context Lineage Assurance for Non-Human Identities in Critical Multi-Agent Systems. arXiv:2509.18415.
Laurie, B., Langley, A., & Kasper, E. (2013). Certificate Transparency. RFC 6962, IETF.
AlphaSignal (2026). The Real Claude Tag Question Is Context Ownership. alphasignalai.substack.com

The LLM Should Never Do the Math

Jeremy Longshore — Mon, 29 Jun 2026 13:00:20 +0000

A CFO will not act on a number an LLM eyeballed. They will not act on a number the model "estimated" by reasoning over a usage dump. And they should not — because the moment a language model emits a dollar figure it computed itself, that figure is a guess wearing the costume of a fact.

This is the design constraint behind databricks-cost-leak-hunter, the pilot skill of the databricks-pack v2 rebuild shipped in the claude-code-plugins marketplace (PR #906). Given a live, authenticated Databricks workspace, it surfaces real cost leaks across four named categories, ranks them by monthly dollar impact, and emits a report a finance reader can act on. The marketplace validator graded it B (88/100, zero errors). The SKILL.md is 329 lines. The single most important thing in it is a rule the model is structurally prevented from breaking: the LLM never does the dollar arithmetic.

Why not just let the agent read the bill and summarize it?

Because that is exactly how you ship a confidently wrong cost report.

Hand a model a few thousand rows of system.billing.usage and ask it for the top cost leaks, and it will give you a fluent answer. It will add DBUs. It will multiply by a price it half-remembers. It will round. Every one of those steps is a place the model can be plausibly, invisibly wrong — and the output reads identically whether the math is right or hallucinated. The failure mode of an LLM doing FinOps is not a crash. It is a clean, well-formatted, wrong number.

The fix is architectural, not prompt-engineering. The model is allowed to decide what to look for and how to explain it. It is never allowed to be the calculator.

The dollar primitive: confirmed, never estimated

Every confirmed figure comes from the customer's own billing tables — system.billing.usage joined to system.billing.list_prices. Not a model estimate. Not a public price list. The number Databricks actually billed.

That join is defined once, as a priced CTE, and reused by every category query. Usage is multiplied by list_prices.pricing.default, matched on both sku_name and usage_unit, inside the price-effective window, filtered to currency_code = 'USD'. Define the dollar primitive once; every downstream query inherits it. There is exactly one place where a DBU becomes a dollar, and it is a SQL join against the source of truth.

Before any of that runs, Step 1 is a fail-fast grant check. system.billing.usage sits behind a metastore-admin grant chain, so the skill probes it first:

SELECT 1 FROM system.billing.usage LIMIT 1;

If that errors, the skill reports the exact missing GRANT USE CATALOG / USE SCHEMA / SELECT chain verbatim and stops — rather than limping forward and failing mid-analysis with a half-built report. It also requires DATABRICKS_WAREHOUSE_ID, a running SQL warehouse to execute against. Fail at the door, with the precise remediation, or not at all.

Two data planes: dollars and evidence

The skill reads from two MCPs, and the split is the whole architecture.

Dollars come from the Databricks CLI Statement Execution API — databricks api post /api/2.0/sql/statements — reading the system.* tables. Auth is the CLI's own DATABRICKS_HOST + DATABRICKS_TOKEN (or databricks auth login), and Unity Catalog enforces the metastore-admin grant chain on every read. This plane answers how much.

Evidence — the live config that explains why a leak exists — comes from a custom databricks-workspace-mcp control-plane server. It reads the live REST API for the auto-termination setting, the node type, the autoscale floor, the pool's min_idle. It has its own PAT / U2M / M2M auth and needs no system-table grants, because it never touches billing data.

The line that captures the division of labor: the SQL produces the number; the workspace MCP turns it into a verified, single-config-change fix. "$8,400/month on a cluster that never auto-terminates" is the SQL's job. "Set autotermination_minutes = 15" is the MCP's. And the degradation path matters: if the workspace MCP is absent, the skill still produces every dollar figure and falls back to accepting pasted config — it never fails silently mid-flow because one plane is missing.

Four leak categories, each labeled with what it actually is

The leaks are not a flat list. Each category carries an explicit confidence kind, and that label travels with the dollars all the way to the report.

1. Clusters that never auto-terminate — confirmed. Join priced ALL_PURPOSE usage to system.compute.clusters, flag auto_termination_minutes = 0, rank by 30-day idle spend. This is money actually billed for compute that sat idle:

SELECT p.usage_metadata.cluster_id AS cluster_id,
       COALESCE(c.cluster_name, 'unknown') AS cluster_name,
       c.auto_termination_minutes,
       ROUND(SUM(p.usd), 2) AS spend_30d_usd
FROM priced p
JOIN cluster_cfg c ON p.usage_metadata.cluster_id = c.cluster_id
WHERE p.billing_origin_product = 'ALL_PURPOSE'
  AND c.auto_termination_minutes = 0
GROUP BY p.usage_metadata.cluster_id, c.cluster_name, c.auto_termination_minutes
HAVING SUM(p.usd) > 0
ORDER BY spend_30d_usd DESC;

2. Scheduled jobs on All-Purpose compute — confirmed. A usage row with a job_id and billing_origin_product = 'ALL_PURPOSE' (~$0.55/DBU) instead of JOBS_COMPUTE (~$0.15/DBU). Re-price the exact same DBUs at the Jobs rate; the delta is the savings. Deterministic re-pricing, not a model's guess at "roughly 3x cheaper."

3. Overprovisioned clusters idling below floor — estimated. Mean CPU from system.compute.node_timeline, flag clusters under 25% utilization, est_overprovision = spend × (1 − CPU%). This is the one arithmetically modeled number in the skill — the at-risk Photon figure below is flagged for review, not model-computed — and it is labeled est_* everywhere it appears, so no one mistakes a model for a bill.

4. Photon premium without the speedup — at-risk. Photon is not a column. It is billing-visible via the SKU. Surface the ~2× premium portion as money to review against actual runtime gain — not confirmed waste:

SELECT p.usage_metadata.cluster_id AS cluster_id,
       ROUND(SUM(p.usd), 2)        AS photon_spend_30d_usd,
       ROUND(SUM(p.usd) / 2.0, 2)  AS photon_premium_at_risk_30d_usd
FROM priced p
WHERE p.sku_name ILIKE '%PHOTON%'
  AND p.billing_origin_product IN ('ALL_PURPOSE','JOBS_COMPUTE')
  AND p.usage_metadata.cluster_id IS NOT NULL
GROUP BY p.usage_metadata.cluster_id
HAVING SUM(p.usd) > 0
ORDER BY photon_premium_at_risk_30d_usd DESC;

Three kinds — confirmed, estimated, at-risk — and a report that never blurs them together:

Confidence kind	What it means
Confirmed	Money actually billed, from `system.billing.usage` joined to `system.billing.list_prices`
Estimated	A modeled figure from utilization data (CPU %), labeled `est_*` so no one reads it as a bill
At-risk	Spend flagged for business review (the Photon premium) before it counts as recoverable waste

The ranker that the model cannot fudge

The pipeline is detect → compute → rank → report. The detect and report stages are the model's. The compute and rank stages live in scripts/rank-and-report.py, whose docstring opens with the thesis:

The LLM does NOT do the dollar arithmetic.

The script ingests per-category JSON — each item {category, root_cause, fix, waste_30d_usd, kind} — converts 30-day spend to monthly, ranks descending, and renders the report. And it enforces the invariant the whole skill is built on: never sum confirmed and unconfirmed dollars under one verb.

UNCONFIRMED = ("estimated", "at-risk")

# Split sum — confirmed and unconfirmed dollars are NEVER added under one verb.
confirmed_monthly   = sum(c["monthly"] for c in ranked if c.get("kind") == "confirmed")
unconfirmed_monthly = sum(c["monthly"] for c in ranked if c.get("kind") in UNCONFIRMED)

That split is why the headline reads honestly. For a $100K/month workspace it renders something like: "burning *~$19,000/month** (confirmed), plus up to ~$8,000/month pending review"* — with a trailing-30-day window stamp, and a ranked table whose Confidence column is load-bearing:

#	Where it's leaking	$/month	Confidence	The fix

Root-cause cells use plain business language. No raw DBU in any CFO-visible text — that translation happens before the number reaches finance, not in the reader's head.

Why not let the model add the numbers?

Because a model that can add can also mis-add, and you cannot tell which run you got. Deterministic arithmetic outside the LLM is not a performance optimization. It is the difference between a report and a guess. The model's strengths — pattern-matching, explanation, ranking by business relevance — are real and used here. Its weakness — silent numerical confabulation — is engineered out by never giving it a calculator in the first place.

This is the same instinct that runs through the rest of the skill-marketplace work: deterministic math outside the model, every figure labeled with its confidence, and the model confined to the parts of the job where being articulate is an asset rather than a liability.

The adversarial review that caught the LLM inventing a schema

The skill was built through a 10-agent workflow: research → author → four adversarial review lenses → revise. That review stage is not polish. It is load-bearing — and here is why.

The review caught a hallucinated system.compute.clusters.runtime_engine column. The model had confidently written its first Photon-detection query against a column that does not exist. Left unreviewed, the skill would have shipped a cost report built on a fictional schema — and it would have looked completely plausible doing it. The fix is the Photon query above: detect via sku_name ILIKE '%PHOTON%' on the priced billing row, because Photon is billing-visible by SKU and never a system-table column.

That is the entire argument for adversarial review of LLM-authored systems in one bug. The failure mode is not a stack trace. It is confident plausibility — output that looks right, reads right, and is wrong.

Two more from the same pass:

A non-deterministic jobs-rate join fan-out. The jobs-rate CTE could match multiple price rows per usage unit, multiplying rows and inflating savings — the kind of un-deduped join a model reaches for because the simpler version reads correctly to anyone skimming it. Fix: dedupe to one USD rate per usage_unit so the join can't fan out.
CFO-clarity gaps in the narrative cells.

Then a follow-up Gemini code review caught three more. Two worth naming:

CRITICAL: spend-baseline.sql.json had ${DATABRICKS_WAREHOUSE_ID} inside a --json @file template. The Databricks CLI does not expand env vars inside a JSON file, so the call would fail with an invalid warehouse id. Fix: jq-inject the id at call time — jq --arg wh "$DATABRICKS_WAREHOUSE_ID" '. + {warehouse_id: $wh}' template.json.
HIGH: the renderer, at one point, summed confirmed and estimated/at-risk dollars under a single total with no Confidence column — violating the exact invariant the skill exists to enforce. The model regressed against its own design rule. Review caught it.

That last one is the lesson stated cleanly: even a system explicitly designed around "never sum confirmed with modeled" will, under LLM authorship, drift back toward doing it — because summing everything into one big number is the locally fluent move. Adversarial review is the gate that catches the regression. A hallucinated column and a silently-fanned-out join both produce output that looks right; only a hostile second read finds them.

The same instinct, elsewhere

Precision over blunt tooling ran through the rest of that day's work too. A dependency-vuln triage in a Node mail service refused a blind npm audit fix --force — which would have major-bumped googleapis and typescript-eslint — and instead cleared 17 of 22 advisories via npm overrides and targeted direct bumps, documenting the 5 accepted residual install-time-only vulns in tests/TESTING.md. The destructive automatic fix is rarely the right one, in dependency graphs or in cost reports. A version-controlled document-store initiative got consolidated into a single synthesis hub, and partner FinOps research moved a step forward.

The transferable rule

If you want a number a decision-maker will act on, the model is not allowed to compute it. Join the source-of-truth tables for confirmed figures. Label every output with its confidence kind and never blur the kinds together. Run the arithmetic in deterministic code the model cannot reach. And put the whole thing through adversarial review before it ships — because the LLM's most dangerous output is the one that looks exactly right.

Govern at the Merge: The Untrusted Union — where governance belongs in a pipeline, and why the merge point is the gate. Same argument as the grant check and the confidence split: enforce at the boundary, not after.
The Bot Loop Circuit Breaker for Multi-Agent Slack — multi-agent review and the failure modes that only show up when agents check each other's work, the way the four review lenses caught the hallucinated column.
The API Is the Real Boundary — the two-MCP split is exactly this: dollars on one data plane, config evidence on another, each with its own auth and its own job.

{
"@context": "https://schema.org",
"@type": "BlogPosting",
"headline": "The LLM Should Never Do the Math",
"description": "A Claude Code skill that hunts Databricks cost leaks and reports confirmed dollars from the customer's own billing tables — never LLM estimates.",
"datePublished": "2026-06-26T08:00:00-05:00",
"dateModified": "2026-06-26T08:00:00-05:00",
"author": {
"@type": "Person",
"name": "Jeremy Longshore"
},
"publisher": {
"@type": "Organization",
"name": "Start AI Tools"
},
"url": "https://startaitools.com/posts/llm-never-does-the-math/",
"keywords": "LLM cost report, Databricks cost leaks, deterministic math, confirmed dollars, FinOps, Claude Code skill, AI agent reliability, adversarial review, hallucination"
}

N Bots in One Slack Channel Loop Forever: Three Gates

Jeremy Longshore — Fri, 26 Jun 2026 15:49:28 +0000

Put two AI agents in the same Slack channel and let each one reply to anything it sees, and you have built a perpetual motion machine. A answers a human. B sees A's message and answers it. A sees B's answer and answers that. Neither one is wrong. Neither one is broken. They will keep going until you kill a process, because each message is, from Slack's point of view, a legitimately distinct event that deserves a response.

This post is about claude-code-slack-channel — the substrate that lets humans, Claude Code sessions, and peer agents share one channel — and the three commits-worth of work it took to make that channel safe when more than one bot is in it. The thesis is one sentence: safety for a multi-agent channel is three moves, not one. You gate who gets to speak (mention-to-engage), you trip a breaker when speech runs away (a channel-wide circuit breaker), and you keep the whole thing observable and bounded under load (backpressure plus structured drop reasons). Pull any one and the other two don't save you.

The problem: bots answer each other forever

The peer-bot feedback loop is the failure that defines the domain. By default a channel ignores other bots entirely; the loop only becomes possible once an operator explicitly opts peer bots in via an allowBotIds allowlist. But that opt-in is exactly what you want — the whole point of a shared channel is agents collaborating — so "just don't allowlist bots" isn't a fix, it's a refusal to build the feature.

The moment two bots are on the allowlist, A's reply is an inbound event for B, B's reply is an inbound event for A, and the exchange has no natural terminator. Ordinary event-dedup doesn't help: dedup catches the same event arriving twice, but A's third message and B's fourth message are genuinely new events. The loop is made of distinct, valid messages. That's what makes it nasty — every individual step looks correct.

The obvious fix is "only respond when you're spoken to" — require an @-mention before a bot engages. And it works, right up until a human tries to actually have a conversation. A mention-gate that demands a fresh @bot on every single message is too dumb to use: the human mentions the bot, gets an answer, types a follow-up, and gets silence, because the follow-up didn't re-mention. So they re-mention. Every turn. The channel becomes a place where you shout the bot's name before each sentence. Mention-gating that's correct but unusable isn't a fix; it just trades one failure for another.

So the real problem has two halves that have to be solved together: stop bots from talking to bots, without making the channel miserable for the humans. Three moves do it.

Move one: mention-to-engage, but thread-sticky

The first move makes mention-gating usable. A requireMention channel drops any message that doesn't @-mention the bot — except once a human has engaged a thread by mentioning the bot, subsequent human messages in that same thread are delivered without a fresh mention. Mention once to open the thread, then converse. The engaged thread is recorded in a set keyed by the session thread, so a top-level mention and its in-thread follow-ups resolve to the same slot.

// Illustrative — the requireMention branch of the channel gate.
if (policy.requireMention && !isMentioned(ev, botUserId)) {
  // Thread-sticky: a human who already mentioned the bot in this
  // thread keeps the floor without re-mentioning every turn.
  if (!ev.bot_id && opts.engagedThreads !== undefined) {
    const threadTs = (ev.thread_ts ?? ev.ts) as string | undefined
    if (opts.engagedThreads.has(deliveredThreadKey(channel, threadTs))) {
      return { action: 'deliver', access }
    }
  }
  return { action: 'drop', dropReason: 'channel.require_mention' }
}

Three details in that branch are load-bearing. Peer bots are never sticky — the !ev.bot_id guard means an allowlisted bot has to mention the target on every message to be delivered, so a bot can't ride a human's engaged thread into a loop. Stickiness is a convenience extended to humans only. A mention is still required to open a thread — stickiness lowers the cost of staying in a conversation, never the cost of starting one. And the engaged set is bounded: it evicts its oldest entry past a cap, so a busy channel can't grow it without limit. A convenience feature that leaks memory is a slow outage.

The matching gate also has to be careful about what counts as a mention. A human pasting a code snippet or quoting an earlier message that contains <@bot> is displaying the token, not addressing the bot. So the matcher prefers Slack's structured blocks and prunes the rich_text_preformatted (code block) and rich_text_quote (blockquote) subtrees before looking for the mention — a <@bot> buried in a fenced code block can't falsely engage the channel. It falls back to substring matching only when structured blocks are absent.

Finally, the interaction mode itself is a first-class channel choice, and new channels default to mention-to-engage. The safe posture is the default; opening the floodgates is the thing you opt into, not the thing you forget to turn off.

Move two: a channel-wide circuit breaker for N-cycle rings

Mention-gating handles the well-behaved case. The breaker handles the runaway one — when something gets past the gate anyway, or when two bots are supposed to talk and the conversation goes feral.

The first layer is a per-(channel, bot_id) sliding window: track each bot's message timestamps, and once a single bot exceeds the threshold (default 10 messages in 60 seconds) in a channel, drop that bot's messages until the window slides back under. That's tuned well above any plausible legitimate cross-bot rate — humans interleave their messages, loops don't — and it cleanly breaks the pairwise A→B→A case.

But the pairwise limit has a blind spot, and it's the kind you only see when you reason about it adversarially. A longer A→B→C→D→E→A ring keeps every individual sender under its own per-bot cap. Five bots each posting every seven or eight seconds sit at roughly eight messages a minute apiece — comfortably under the cap of ten — so a counter that only ever watches one bot at a time reports all-clear while their combined velocity runs away. That's the threshold math, and it's worth saying out loud: a two- or three-bot ring is caught by the per-bot counter, because for three bots to sum past a 40/60s aggregate at least one has to exceed its own 10/60s cap. The aggregate breaker earns its keep precisely when the ring is large enough — five or more — that every member can hide under the per-bot limit.

So a second counter sits on top: a channel-wide aggregate that sums peer-bot velocity across all bots in the channel and trips when the total runs away.

// Illustrative — two counters, two scopes, checked in order on the inbound gate.
if (!perBotWindow.check(channelId, botId, now, perBotLimit)) {
  return { action: 'drop', dropReason: 'rate.cross_bot_loop' }   // pairwise A→B→A
}
if (!channelWindow.checkChannel(channelId, now, breakerConfig)) {
  return { action: 'drop', dropReason: 'rate.channel_cycle' }    // N-cycle ring A→B→C→D→E→A
}

The aggregate default is roughly 4× the per-bot limit (40 messages in 60 seconds), chosen so a legitimate two- or three-bot exchange interleaved with real human work never trips it, but a runaway ring — which fires as fast as the pipeline allows — trips it within seconds.

It helps to trace the ring against both counters to see why the second one earns its place:

# Illustrative — an A→B→C→D→E→A ring under a 10/60s per-bot cap and a 40/60s breaker.
# Five bots, round-robin, one message every ~1.5s.
t=00.0s  A posts   per-bot[A]=1   channel=1
t=01.5s  B posts   per-bot[B]=1   channel=2
t=03.0s  C posts   per-bot[C]=1   channel=3
t=04.5s  D posts   per-bot[D]=1   channel=4
t=06.0s  E posts   per-bot[E]=1   channel=5
t=07.5s  A posts   per-bot[A]=2   channel=6     # each bot only ~ every 7.5s — per-bot stays near 8, never 10
...
t=58.5s  ...        per-bot[*]≈8   channel=40    # every bot under its cap, but the ring trips the AGGREGATE → rate.channel_cycle

Each bot paces itself to about eight messages in the window — under its own cap of ten — while the channel total climbs to 40 and the breaker fires. The per-bot counter, watching each bot in isolation, would have let this five-way ring run indefinitely. Three properties keep the aggregate honest:

Humans always post freely. Both counters only apply to events where bot_id is set. A human in a melting-down channel is never rate-limited; the throttle is aimed exclusively at the machines.
It's disablable, with one coherent semantic. Setting the config to { count: 0, windowMs: 0 } short-circuits to "always allow" in both the store and the gate. (That non-positive-config path was a follow-up fix — the original code had a dead branch where a zero config didn't actually disable cleanly. To deny all peer-bot delivery you don't zero the limit; you set allowBotIds: []. Two intentions, two switches.)
It's deliberately not persisted. A process restart resets every counter. Same posture as the channel's nonce store: a fresh process means no stale state for a misbehaving bot to game, and the breaker re-derives from live traffic in one window.

Counter	Scope	Catches	Default
Per-bot sliding window	one `(channel, bot_id)`	pairwise A→B→A loops	10 / 60s
Channel circuit breaker	all bots in a channel	N-cycle A→B→C→A rings	40 / 60s

Neither counter alone is sufficient. The per-bot limit can't see a ring; the aggregate limit can't tell you which bot is the offender. Layered, they cover both the pair and the ring.

Move three: backpressure and observability

The third move is what turns a channel you've gated into one you can actually operate. Loops aren't the only way a multi-agent channel hurts you — a burst of legitimate new conversations can exhaust resources just as effectively, and a channel where you can't see why a bot went quiet is one you can't debug.

A global backpressure cap. The session supervisor takes a maxConcurrentSessions limit; once live plus in-flight activations hit the cap, a genuinely new session is refused rather than spun up. The load-shed decision is both logged to the operator and written to the audit journal — a refusal is a governance event, not a silent drop. One subtle correctness note: the cap counts distinct in-progress sessions, because a session that just finished activating is briefly present in both the "live" set and the "activating" set, and naively summing the two would double-count it and reject sessions spuriously near the cap.

// Illustrative — refuse a NEW session at the cap; count distinct in-flight keys.
let inFlightNew = 0
for (const k of activating.keys()) if (!live.has(k)) inFlightNew++
if (live.size + inFlightNew >= maxConcurrentSessions) {
  journalWrite({ kind: 'session.activate_rejected', outcome: 'deny', /* ... */ })
  return Promise.reject(new Error('at maxConcurrentSessions cap'))
}

A structured drop reason on every gate drop. Every inbound message the gate turns away now carries a typed dropReason, and that reason lands in the journal. The reasons are grouped by the gate stage that produced them — which is itself a readable map of how an inbound event is evaluated, in order:

bot-event stage — self.echo (our own message), bot.not_allowlisted, admin.muted, rate.cross_bot_loop, rate.channel_cycle, bot.permission_relay.
top-level stage — subtype.filtered (an edited or deleted message, intentionally ignored), event.no_user.
DM stage — dm.policy_closed, dm.pairing_cap, dm.pending_full.
channel stage — channel.not_opted, channel.allowfrom_miss, channel.require_mention.

The payoff is operational: when a bot stays silent, you can always answer "why didn't it respond?" by reading one field instead of guessing. Silence stops being ambiguous. An enum beats a free-text string here because the reasons are a closed set you want to count, filter, and alert on — rate.channel_cycle spiking is a signal you can build a dashboard on; a string blob isn't.

Read-only admin verbs from Slack. Operators can ask the channel about itself without changing anything: !mute-status lists active peer-bot mutes and their remaining TTL, !rate-limit shows the effective thresholds, !agents lists the peer bots seen active recently. The !agents view is derived from the per-(channel, bot) activity the rate limiter already tracks — the same timestamps that feed the breaker double as an "agents online" read-out, no separate bookkeeping. These are pure observation verbs — no argument, no state change — sitting alongside the destructive !clear/!restart commands but carrying none of their risk. You shouldn't have to SSH into a box to find out which bots are live in a channel.

A targeted mute, between the allowlist and the breaker. The allowlist is all-or-nothing per bot; the breaker is all-or-nothing per channel. Between them sits !mute @bot, which silences one specific peer bot in one channel for a TTL — its messages drop with dropReason: 'admin.muted' until the mute expires. That's the surgical option: when exactly one bot is misbehaving, you don't have to choose between de-allowlisting it everywhere or tripping the whole channel's breaker. You mute the one offender and leave the rest of the conversation running.

Per-user session isolation, opt-in. A channel can opt into keying sessions by (channel, thread, userId) instead of the shared (channel, thread), so two humans working in the same thread get independent sessions that don't observe each other's state. It's default-off and default-safe: no flag means the legacy shared key, behavior unchanged. The guard validates the sender id is a non-empty string before keying on it, so a malformed event falls back to the shared session rather than constructing a broken key.

Why the three compose

It's tempting to ship one of these and call the channel safe. Each one alone leaves a hole the other two close.

Mention-gating without a breaker assumes every actor honors the gate. The moment a bot is meant to be in the conversation — allowlisted, mentioning correctly — the gate waves it through, and two such bots can still loop. The breaker is the backstop for actors the gate legitimately admits.

A breaker without mention-gating is all backstop and no front door. You'd let every bot respond to everything and rely on rate limits to clean up the mess — which means the channel is constantly tripping its own breaker, and the humans are drowning in bot chatter below the trip threshold. Gating keeps the normal case quiet so the breaker only fires on genuine runaways.

And both of those without observability is a system that's safe but unoperatable. When a bot goes quiet you can't tell whether it hit the mention gate, tripped the breaker, got muted, or crashed — so you can't tune any of it. The structured drop reasons and the read-only verbs are what let you see which gate is firing and adjust the thresholds with evidence instead of superstition.

The ceiling, named honestly

What this earns is bounded, and worth stating plainly so nobody oversells it.

The breaker is velocity control, not intent control. It stops a fast loop — the kind that fires as fast as the pipeline allows. Two bots having a slow, wrong conversation at one message every ten seconds stay under every threshold and will happily waste tokens all afternoon. The defense against that is the mention gate and the allowlist, not the rate limit; the breaker is specifically the runaway-velocity backstop, and calling it a general "bad conversation detector" would be a lie.

The counters are in-memory and per-process. A restart resets them — deliberately, but it means a multi-process deployment doesn't share a breaker, and the cap is per-supervisor, not global across a fleet. At single-channel, single-process scale that's the right amount of machinery; the day this runs as a horizontally-scaled service, the counters need a shared backing store, and that's a real piece of work, not a config flag.

And the thresholds are heuristics. 10-per-60s and 40-per-60s are tuned for the realistic loop shape, but they're operator-tunable for exactly the reason that the right number depends on your channel's legitimate traffic. The defaults are a safe starting point, not a proof.

The principle

A shared channel is a place where independent actors — human and machine — touch the same surface, and the reflex is to assume each actor will behave. For humans that's mostly true. For bots it is exactly false: a bot does precisely what it's told, forever, including answer another bot that's answering it.

So you don't make a multi-agent channel safe with one clever gate. You gate who may speak, you trip a breaker when speech runs away, and you keep the whole thing observable so you can see which defense is doing the work. Mention-to-engage keeps the normal case calm. The circuit breaker catches the loops the gate admits. Backpressure and structured drop reasons keep it bounded and debuggable under load. Three moves, each covering the other two's blind spot. Anyone putting more than one bot in one room needs all three.

Related posts:

MCP Server Auth: The API Is the Real Boundary — the same instinct applied to auth: knowing which layer is the actual boundary and which is just UX.
Honor the Gate When the Verdict Is Inconvenient — a gate is only worth building if you respect its drops.
The Merge Is the Trust Boundary: Re-Derive as Untrusted — properties don't compose across a boundary you didn't enforce yourself.

The Merge Is the Trust Boundary: Re-Derive as Untrusted

Jeremy Longshore — Fri, 26 Jun 2026 15:49:20 +0000

A single-writer knowledge store with a hash-chained audit log has a clean story: every governance event folds the previous entry's hash into its own, so an in-place edit breaks the chain and verify catches it. That story holds right up to the moment a second writer appears.

The instant two clones of the same store can each promote facts independently and later reconcile, the audit chain stops being a single line. Clone A has a chain that verifies clean against itself. Clone B has a chain that verifies clean against itself. Merge them, and you have a question neither green checkmark answers: is the union governed?

This post is about the EPIC 1 work in qmd-team-intent-kb that took the position that it is not — and that the only honest fix is to treat the merge itself as the trust boundary, re-derive the union as untrusted, and re-govern it from scratch. The thesis underneath is one sentence: in a multi-writer knowledge store, the merge is where trust has to be rebuilt, because it is the one place trust cannot be inherited.

The problem: audit chains don't compose across a merge

The seductive assumption is that two trustworthy halves make a trustworthy whole. A clone's chain verified; the other clone's chain verified; surely their union is fine. Two independent failures kill that.

Failure one: a content-level merge runs no policy. The decisive finding came from a throwaway spike against Dolt's native three-way merge. The spike fed two branches into Dolt's row-union and watched a secret-bearing row — one that a governance gate would have rejected — land in the merged database with zero governance applied.

The union operator reconciles rows; it does not re-run your disclosure checks, your dedupe, your tenancy rules. Version control is not governance. Anything either branch promoted (or had slip past a weaker gate) rides into the merged state unchecked, because the merge operator has no concept of a policy to enforce. This is not a Dolt defect — it is true of every content-level merge operator, which is exactly why the fix has to live above the substrate rather than inside it.

Failure two: the linear verifier breaks at the seam — for an innocent reason. The store's audit verifier is a linear walker: it sorts rows by (timestamp ASC, id ASC) and asserts each row's prev_entry_hash matches its predecessor. That contract is exactly right for one clone's own history and exactly wrong for a merge.

# Illustrative — why a linear walk fires a FALSE break at a merge seam.
clone A chain (its wallclock):   a0 → a1 → a2
clone B chain (its wallclock):   b0 → b1
merged, sorted by timestamp:     a0  b0  a1  b1  a2     # interleaved by wallclock
linear walk: a1.prev_entry_hash == a0  but predecessor is now b0  →  PREV_LINK_MISMATCH

Two clones evolve on independent wallclocks; concatenating their chains by timestamp interleaves rows that never linked to each other, and the walker fires PREV_LINK_MISMATCH at the boundary. So the naive merge produces a chain that looks tampered even when nobody tampered — which means a real tamper at the seam would be indistinguishable from the normal noise of merging. A verifier that cries wolf at every merge can't be trusted to catch a wolf.

Put together: the merge admits ungoverned data and destroys the signal that would tell you. You cannot patch this by trusting either input chain harder. The trust has to be rebuilt at the merge, not inherited through it.

The design: re-derive the union as untrusted

The move is to stop treating a row that crossed a clone boundary as "already governed." At merge time the gate (mergeGovern) takes the union of both clones' promoted rows and re-runs every one of them through the same front-door governance it would apply to a brand-new capture — as if none had ever been trusted.

# Illustrative — the merge gate, in five moves.
union = dedupe_by_content_id(cloneA.promoted ++ cloneB.promoted)
union = sort_by_content_id(union)              # makes A∪B and B∪A byte-identical
for index, row in enumerate(union):
    cand = project_to_untrusted(row)           # strip governance metadata; trustLevel = untrusted
    if not disclosure_clean(cand):  quarantine(row); continue   # fail-closed secret/PII choke point
    if not policy_pipeline(cand):   quarantine(row); continue   # dedupe, sensitivity, tenancy, ...
    promote(cand, clock = merge_clock(index))  # canonical promotion path, deterministic timestamp

The load-bearing detail is project_to_untrusted. A promoted row carries governance metadata — its lifecycle, its prior policy evaluations, its trustLevel. The gate strips all of that, keeps only the source content and provenance, and forces trustLevel = untrusted, so the source_trust rule evaluates the row as if it had no standing. The merge trusts nothing that came across a boundary. Survivors get promoted through the canonical path; a secret that rode in on a clone is quarantined, never written. The merged state is, by construction, indistinguishable from one assembled by feeding every row through the front door one at a time.

Quarantine has its own non-leak contract, and it matters more than it looks. A rejected row is recorded as a QuarantinedRow carrying the offending memory's content-derived id, a category (disclosure or policy), and a human-readable reason — the disclosure category, or the id of the rule that rejected it. What it never carries is the matched secret or PII value.

The whole point of the disclosure choke point is to keep secrets out of the governed store; a quarantine record that quoted the secret back in its reason field would re-leak exactly what it just blocked, into the very audit trail meant to be safe to share. So the gate reports that a row was turned away and why category, never what the offending value was. Refusing to leak on the rejection path is as much a part of the design as refusing to admit on the success path.

The gate also fails loud, and atomically, on a row it cannot reason about. Before any promotion runs, it pre-passes the entire union and checks that every row's id reproduces under re-derivation — that id == deriveMemoryId(candidateId, contentHash). A row whose id does not reproduce did not come from the canonical promotion path; it carries a stray, non-content-derived id (a crypto.randomUUID() v4 from some out-of-band code path), and that id is poison: it would survive de-dup as its own twin and sort to a per-clone position, breaking both de-dup and commutativity.

Rather than corrupt the merge quietly, the gate throws at entry — before a single row is written — so one bad row aborts the whole merge instead of silently skewing it. Walking the already-sorted union makes the first reported offender deterministic across clones, too: even the failure is reproducible. The error carries the offending id and the id it should have been — both already-public identifiers — but never the row's content, the same non-leak discipline the quarantine path follows.

Critically, the gate invents no new governance primitive. It re-runs the existing disclosure choke point and the existing policy pipeline. That is a deliberate honesty constraint, and it sets a ceiling I'll name later: re-governing the union is exactly as strong as the governance you already had, and not one bit stronger.

That "front door" is itself worth a sentence, because it's what makes re-running it meaningful. The same day this merge gate landed, the store's EPIC 0 hardening put real teeth on the intake path: a fail-closed PII/secret/comp choke point at CandidateRepository.insert() that rejects on detection rather than flagging-and-continuing, token-bound tenancy so a row can't claim a tenant its caller isn't scoped to, and token hashing plus revocation so a leaked credential can be killed.

The merge gate's whole value proposition — "re-derive the union through the front door" — only pays off because the front door is fail-closed. Re-running a permissive gate over the union would launder bad rows, not catch them. The order of operations matters: harden intake first (EPIC 0), then point the merge at it (EPIC 1).

Why content-derived ids make the re-derivation reproducible

Re-running policy is necessary but not sufficient. For the result to mean anything across clones, two clones re-deriving the same union have to land on the same answer — same ids, same hashes, same verdict. That reproducibility is bought by three deterministic choices.

Content-derived UUID v5. Every memory and audit-event id is derived from its content (deriveMemoryId(candidateId, contentHash)) under a locked namespace, not minted from crypto.randomUUID(). So the same logical memory produces the same id in every clone. That makes the union's de-dup a real set operation — two clones holding the same fact collapse to one row — and it makes a sort-by-id traversal deterministic. The gate even refuses, loudly, any row whose id does not reproduce under re-derivation: a stray v4 id would survive de-dup twice and sort to a per-clone position, so the gate aborts the whole merge atomically rather than corrupt it silently.

An entry hash that excludes the wallclock. The audit chain's entry_hash was split into versions: v2 canonicalizes the row body without its timestamp, so the same logical event hashes identically no matter when each clone happened to write it. (v1 rows are deliberately frozen, never rehashed — their original hashes are their tamper-evidence; v1 and v2 coexist and verify in one pass.) Determinism here is a deliberate substitution: the timestamp leaves the hash body so two clones agree, and a separate deterministic merge clock puts ordering back.

A deterministic merge clock. Promotion still needs to write some timestamp. Sourcing it from new Date() would make the merged DB differ run-to-run. Instead merge_clock(index) derives a strictly-monotonic timestamp from the row's position in the id-sorted traversal. Because the sort is content-stable, a given memory lands at the same index — and gets the same timestamp — on every clone. This is what makes mergeGovern(A, B) and mergeGovern(B, A) produce byte-identical durable state and an identical audit chain. The merge is commutative, by construction, not by luck.

Three clones, one pass, same answer

Two-way commutativity is the property you can demo. The one you actually need in a team is that it doesn't matter how the merge was assembled — three people's clones reconciled in any grouping, any order, must land on the same governed state. The N-way path (mergeGovernFold) gets this almost for free, and the "almost" is instructive.

The obvious implementation is an iterated reduce: govern A and B, then re-govern those survivors against C, writing the running database each pass. That is rejected for a concrete reason. The deterministic merge clock restarts at its epoch on every call, so a second pass would mint audit timestamps that collide with or precede the first pass's — and the verifier re-walks the chain by (timestamp ASC, id ASC) while the store anchors each new row to the most-recent by (timestamp DESC, id DESC). Colliding per-pass clocks desynchronize insertion order from chronological order, and the chain breaks.

So the fold instead concatenates every clone's rows into the inputs of a single mergeGovern call: one union, one id-sorted traversal, one strictly-monotonic clock over the whole thing. Because the first step is a set union and the traversal erases input order, fold([X, Y, Z]), fold([Z, Y, X]), and either grouping of the pairwise merges all produce byte-identical durable state and an identical audit chain. Associative and commutative, proven by a fold-of-three test that asserts every ordering verifies with zero breaks. The reduction shape is the only new code; the governance is reused verbatim.

The signed DAG anchor: binding the merge to who made it

Determinism gives reproducibility. It does not give attribution. A wholesale rewrite that recomputes every hash is internally consistent and self-verifies clean — the same gap a single-writer external anchor closes for a linear chain. A merge needs more, because a merged head has two parents and the question "who reconciled these two histories?" has no answer in the hashes alone.

So each merge appends a per-actor Ed25519-signed anchor record (SignedMergeAnchorRecord). It binds the merged chain head to a parents set — the two pre-merge clone heads, sorted so the set canonicalizes identically regardless of input order — a per-actor Lamport clock, the signer's public key embedded inline, and a detached Ed25519 signature over the canonical body. Two independent guards then ride every record: the existing SHA-256 integrity hash (the bytes weren't accidentally corrupted) and the signature (a forger who edits the merged chain and re-hashes it forward still cannot produce a valid signature without the actor's private key).

# Illustrative — what the merge-aware verifier checks, and what each catches.
per-clone linear walk        →  tamper inside one clone's own history
id-sorted merged re-walk     →  merged DB NOT produced by mergeGovern (or reordered after)
Ed25519 signature verifies   →  WHO anchored this merge — not just that the bytes agree
parents == {headA, headB}    →  the anchor attests to the correct two chains
chainHead == merged head     →  the anchor describes THIS merged head, not a different one

The parents field is what makes this a DAG anchor rather than a fancier linear one. A merged head has two ancestors, and the anchor records them as a set, not a tuple — sorted before the body is serialized — so the same two clone heads canonicalize to identical signable bytes no matter which clone was passed to mergeGovern first. The Lamport clock alongside it is a per-actor monotonic counter the caller owns and bumps before each signed anchor, so an auditor can order one actor's anchors causally even when wallclocks across machines disagree. And an optional commitHash records the Dolt or git SHA the merge landed on, giving an external auditor a thread back into version-control history. None of these are required to verify a single merge; all of them are what let a sequence of merges across actors be reconstructed after the fact.

The merge-aware verifier owns the canonical id-based ordering as a first-class contract — it re-walks the union by content-derived event id and checks prev_entry_hash against the id-sorted predecessor, not the wallclock one. It validates each clone's chain linearly first (a break there is tamper in that clone's own history, surfaced before any merge-level reasoning), then re-walks the merged union, then cross-checks the signed anchor. That ordering cures the false PREV_LINK_MISMATCH from earlier: a clean merge re-walks to byte-identical hashes and reports zero breaks, so a real reorder or tamper now stands out instead of drowning in merge noise. Against a keyless adversary, the skeptic harness that gated this work reported 20,000 brute-force signature attempts with zero accepted, and a tamper-then-rehash trips HISTORY_REWRITTEN or DAG_HEAD_MISMATCH.

Five layers stack into the property, and it helps to see them as a unit — each earns one specific guarantee, and none is load-bearing alone:

Layer	Mechanism	What it earns
Content-derived UUID v5	same content derives the same id in every clone	union de-dup is a real set op; id-sorted traversal is reproducible
Deterministic entry hash (v2)	canonical hash body excludes the wallclock timestamp	the same logical event hashes identically across clones
Govern-at-merge gate	re-derive the union as untrusted through the front door	ungoverned rows are quarantined, never admitted
Signed DAG anchor (Ed25519)	parents set + signature over the merged head	which key reconciled the histories — identity needs a trusted key roster on top
Merge-aware verifier	per-clone walk + id-sorted re-walk + anchor cross-check	a real reorder or tamper stands out instead of drowning in merge noise

Pull any row and the property degrades: without content-derived ids the merge isn't commutative; without the deterministic hash the merged chain won't re-walk clean; without the gate the union is ungoverned; without the signature you can't attribute the merge; without the merge-aware verifier you can't tell a clean merge from a tampered one. They were built and merged as one epic for that reason.

Why it matters: verification that travels with the content

The payoff is that verification is no longer tied to a place or a trusted party. Any clone, anywhere, that re-derives the same content reproduces the same ids, the same entry_hash, the same merged chain, and the same quarantine-or-promote verdict. "Is this merged store governed?" becomes a question you answer by recomputation, not by trusting the machine that produced it. The signed anchor adds the one thing recomputation can't: evidence that a specific key performed the reconciliation. The signature is self-contained — verifiable against the public key embedded in the record, with no key server in the loop — but that earns integrity plus "one keyholder signed this," not identity. Attributing that key to a named actor still needs a trusted roster of which public key belongs to whom: the anchor proves a key signed; the roster says whose key it is. That roster is the out-of-band trust this design still leans on, and naming it is the honest version of the claim.

Determinism also makes the merge safe to preview. The gate takes a dryRun flag that runs the entire pipeline — validation, de-dup, disclosure check, policy, audit-event construction — and writes nothing. Because the result is byte-identical to what a real run would produce, a dry run is a faithful forecast, not an approximation: you can see exactly which rows would promote and which would quarantine before committing the merge. A non-deterministic gate couldn't offer that; its preview and its real run could disagree.

That is the whole shape of the thing. Where a version-control merge says "I reconciled the rows," this says "I re-governed the union from scratch as untrusted, here is the deterministic result you can reproduce, and here is my signature over the head." The merge stops being a place where trust is silently inherited and becomes a place where it is explicitly rebuilt.

The ceiling, named honestly

The discipline this work inherited from its single-writer predecessor is sizing the claim to the mechanism. So, plainly:

# Illustrative — the ceiling.
forger WITHOUT the actor's private key   →  cannot mint an accepted anchor   (caught)
the key-holder edits a row and RE-SIGNS  →  a self-consistent signed anchor  (NOT caught here)

This is tamper-evident, not tamper-proof. A legitimate key-holder — or an exfiltrated key — can still edit a row, recompute the chain, and produce a fresh signed anchor that verifies perfectly. The signature proves the edit was made by the key, not that the edit was honest.

That gap is mitigated, not eliminated, by two things outside the cryptography. First, key custody: the Ed25519 private key lives only in a SOPS/age-encrypted file, mode 600, loaded at sign time and never written to disk in plaintext. Second, committing the anchor log somewhere append-only that the local writer cannot quietly rewrite — a git push, an OpenTimestamps proof. The anchor is only as trustworthy as the place it is committed to; an anchor that lives next to the chain it witnesses is no witness at all. Naming where the stronger claim would need more machinery is part of the claim — the same recursion the single-writer version faced: where do you anchor the anchor?

Two more ceilings worth stating. The re-derivation is only as strong as the detectors it re-runs — the disclosure choke point inherits whatever recall ceiling its secret/PII pattern set has, and re-governing the union doesn't improve that; expanding the detectors is the only lever. And the content-derived ids use UUID v5, which is SHA-1 under the hood per RFC 4122 — that is deterministic namespacing, not a security primitive (the static analyzer's weak-crypto alert on it is a dismissed false positive); the audit integrity that actually matters runs on SHA-256. Calling the namespacing "cryptographic integrity" would be the kind of overstatement this whole design exists to avoid.

One more piece of honesty belongs in the same breath, about the substrate. The clone-and-merge story sounds like it demands a distributed database, and the Dolt spike that exposed the ungoverned-merge problem made Dolt the obvious destination. The decision recorded the opposite: the store of record stays SQLite, and the merge gate operates on the store's types, not on any one engine's merge — so it is substrate-agnostic, and adopting Dolt later is a swap, not a rewrite.

Migrating the brain onto a distributed substrate is explicitly demand-gated: not built until a real multiplayer need is logged — two people actually blocked on a shared brain, or a real cross-person recall miss. Provisioning a distributed control plane before that signal is exactly the premature optimization the de-risked plan exists to prevent. The merge gate is the part you need first, because the governance gap is real the moment two clones exist; the distributed plumbing is the part you can defer until someone is genuinely blocked without it.

The forbidden-words list is short and load-bearing: tamper-proof, immutable, blockchain, and — for the local single-actor case — non-repudiation. What the merge gate earns is real and worth having: governance that survives a merge, deterministic and commutative re-derivation, and cross-actor attribution of the merge event once the anchor is externally committed. Sized to that, the claim holds. Inflated past it, it would be the falsifiable kind that ages into a liability the day someone asks you to prove it.

The principle

A merge is the one operation where two trust domains touch, and the reflex everywhere — version control, caches, replicas — is to assume the operation preserves whatever properties the inputs had. For data you actually govern, that reflex is the bug. Properties don't compose across a reconciliation you didn't perform yourself.

The fix is not a stronger merge operator; it's refusing to inherit trust at all — re-deriving the union as untrusted, re-running the governance you already trust, anchoring the deterministic result with a signature, and being honest that the signature proves authorship, not virtue. The merge is the trust boundary. Treat it like one.

Related posts:

A Second Brain You Can Audit Beats One You Must Trust — the single-writer half of this system, where one external anchor closes the silent-rewrite gap for a linear chain. This post is what happens when a second writer arrives.
Governed Knowledge: Two Releases, a Freshness Daemon, and Export Gating — earlier groundwork on governing a knowledge corpus rather than just retrieving from it.
MCP Server Auth: The API Is the Real Boundary — the same knowledge system, and why the governance audit trail must stay pure for verification to mean anything.

An Agent Allowlist Is a Comment Until a Gate Checks the Body

Jeremy Longshore — Fri, 26 Jun 2026 15:45:26 +0000

A Claude Code agent declares the tools it is allowed to use in its frontmatter. That tools line looks like documentation — a courteous note about what the agent touches.

It is not documentation. It is a runtime allowlist: tools not listed are blocked at runtime. The agent can write all the instructions it wants in its body about calling some MCP tool; if that tool isn't in the allowlist, the call never happens.

Which means a Claude Code agent has two surfaces that can disagree. The frontmatter declares a capability. The body exercises a behavior. Nothing in between proves they match. And because the allowlist is enforced at runtime, a mismatch isn't a cosmetic doc-drift bug — it's a latent failure. An agent that invokes a tool it forgot to declare doesn't error loudly. It silently can't do its job.

At one agent, you catch that by reading the file. This repo ships 317 of them. This post is about the gate that made all 317 declare the truth about themselves in one automated sweep, instead of someone hand-auditing 317 files and hoping.

The problem: a runtime gate that nobody checked against behavior

Every agent file is two parts. Frontmatter — name, description, tools, model, and the rest — and a body, which is the prose that actually instructs the agent what to do and which tools to call. The tools field is the allowlist, and the allowlist is load-bearing in a way a comment never is: the runtime reads it and blocks anything not on it.

So there are exactly two ways for the frontmatter and the body to fall out of sync, and both are silent:

The body uses a tool the allowlist doesn't grant. The agent's prose says "collect the analytics with the umami tools," but tools lists only Read. At runtime every umami call is blocked. The agent is shipped, looks fine, validates against a frontmatter-only schema, and cannot perform its one job.
The allowlist grants a tool the body never uses. Over-declaration. The agent claims privilege it doesn't exercise. The allowlist is now lying in the other direction — it overstates the blast radius of the agent, which is exactly the wrong thing for an allowlist to do.

Neither of these shows up in a frontmatter-only validator. You can check that tools is a well-formed list of real tool names and still have an allowlist that has nothing to do with what the agent does. The schema was green and the agents were wrong, and the only thing that would have caught it was a human reading each body against each allowlist — which does not scale to 317.

The fix is the same move I keep coming back to: a declared property is only worth what the check that compares it to behavior is worth. Without that check, the allowlist is a comment that also happens to block your runtime calls — the worst of both, because being wrong is invisible until the agent fails quietly in front of a user.

The approach: a kernel-strict gate that lands with the fleet

The check didn't arrive in one commit. It arrived as a schema-versioned progression — 3.9 to 3.10 to 3.11 — where each step tightened what "a valid agent" means, and the fleet was dragged up to meet each new bar in the same branch that raised it.

The version number isn't decoration either. Every change to what the validator accepts bumps SCHEMA_VERSION and lands a changelog entry, so "the day the agent contract got stricter" is a diff with a number, not a vibe. When an agent that passed last month fails today, the version delta tells you which rule moved and where it was signed off. A quality gate that changes silently is its own kind of drift; versioning the gate is how the gate stays auditable while it tightens.

Schema 3.10.0 — flip the gate from lenient to kernel-strict. The agent validator used to require almost nothing: name, description, and a warning if you used a banned field. That's a gate in name only. 3.10.0 flipped it to a two-layer required set, all of it errors at every tier (agents aren't tier-gated the way skills are):

The kernel floor — 8 fields: name, description, tools, model, color, version, author, tags. This is the @intentsolutions/core agent-definition required set, consumed from the kernel with an inline fallback mirror so the validator and the spec can't drift apart.
The enterprise live set on top: disallowedTools / skills / background on every agent, plus hooks / mcpServers / permissionMode on standalone agents (those three are plugin-level and ignored at runtime on a plugin agent, so they're only required where they mean something).

It also promoted the banned fields — capabilities, expertise_level, activation_priority, type, category, when_to_use and friends — from WARN to ERROR, and added fable to the model enum.

The discipline that makes this real is in the commit note: CI goes red until the in-repo agents are remediated in the same branch. The gate and the fleet land together. A gate you merge before the fleet conforms is just a broken main with a TODO. A gate you merge with a deterministic remediation of all 317 agents is enforcement from the first commit — the bar and everything held to it move in one motion.

The two-pass order matters and is worth slowing down on. The remediation that turned CI green could not also be the remediation that got things right, because "green" and "correct" are different bars and conflating them is how a big migration goes sideways.

So the green pass was deliberately dumb and deterministic: every agent missing a tools field got the full canonical tool set (a faithful default that preserved the inherit-everything behavior those agents already had), tags scaffolded from plugin category plus name, version set to 1.0.0, a stable color, and empty disallowedTools / skills / background false. Nothing clever, nothing that required judgment per agent — just enough to satisfy the new required set so the gate could go live with a green fleet under it.

The A-grade pass — narrow to least privilege. Then a second pass took the allowlists from "technically valid" to "actually minimal." It did three things to every agent: rewrote description into capability plus an explicit "Use when…" plus trigger phrases; replaced the scaffolded category tags with at least two real lowercase topic tags; and narrowed tools from full-canonical to least-privilege. The result: median allowlist around five tools, and zero agents retaining the full ten-tool set, down from 311 inheriting all of it.

The edits were surgical — only description, tools, and tags changed; every body and all other frontmatter byte-identical across all 317. Splitting it this way meant the risky, judgment-heavy narrowing happened against an already-green baseline, so any regression had a single obvious cause instead of being buried in the same diff that introduced the gate.

And here's the gap that remained. After the A-grade pass, every allowlist was structurally valid and least-privilege — and still nothing proved any of them matched the body. Least privilege you assert by hand is just a tighter comment. A smaller allowlist that the body contradicts is, if anything, more dangerous than an over-broad one: the tighter you draw the grant, the more likely an honest behavior in the body falls outside it and gets blocked.

The check: body-vs-allowlist consistency (schema 3.11.0)

3.11.0 is the step that closes the gap. validate_agent() stopped reading only the frontmatter and started reading the body too, then cross-checking what the body does against what the allowlist grants. The structure, paraphrased:

# Illustrative — the shape of the body-vs-allowlist consistency check.
def check_agent_body_vs_allowlist(frontmatter, body):
    # Example code isn't behavior. Strip fenced ```
{% endraw %}
 …
{% raw %}
 ``` blocks first
    # so a tool shown in a usage snippet doesn't read as a tool call.
    body = strip_fenced_code(body)

    declared   = mcp_tools_in(frontmatter["tools"])     # mcp__server__tool entries
    fq_in_body = find_fully_qualified_mcp_refs(body)     # mcp__server__tool used in prose

    errors, warnings = [], []

    # CHECK 3 (ERROR): zero MCP tools declared, but the body calls them.
    if not declared and fq_in_body:
        errors.append("body references MCP tools but the allowlist declares "
                      "none — every call would be runtime-blocked")

    # CHECK 1 (ERROR): a specific tool the body invokes isn't on the allowlist.
    for ref in fq_in_body:
        if ref not in declared:
            errors.append(f"body invokes '{ref}' but it is not in the allowlist")

    # CHECK 2 (WARN): heuristic short-name mention, only on MCP-oriented agents.
    for name in backtick_verb_camelcase(body):
        if not matches_any_declared(name, declared):
            warnings.append(f"body mentions `{name}` (tool-call-shaped) "
                            "but no declared tool matches")

    # WARN: declared but never used — over-declared privilege / drift.
    for tool in declared:
        if tool not in fq_in_body:
            warnings.append(f"declares '{tool}' but the body never uses it")

    return errors, warnings

The severities are sized to the confidence, and that's the part that makes it a usable gate rather than a noise machine:

A fully-qualified mcp__server__tool reference in the body that isn't on the allowlist is an ERROR. That string is unambiguous — it's the exact runtime name, it doesn't appear by accident, and if it's in the body and not the allowlist, the runtime will block it. High confidence, hard failure.
An allowlist with zero MCP tools whose body still calls them is an ERROR. Same reasoning, whole-agent scale: every MCP call the agent makes is dead on arrival.
A backtick short-name that looks tool-call-shaped is a WARN, and only on MCP-oriented agents. This is the heuristic, and it's deliberately timid. An earlier, broader version of this check would have flagged getStaticProps and every other verbCamelCase token in a body as a phantom tool. So the short-name heuristic is gated to agents that actually deal in MCP tools, and it only warns — because a check that cries wolf gets switched off, and a switched-off check protects nothing.

Two engineering details earn their place. Fenced code is stripped before any matching — a tool shown inside a triple-backtick usage example is documentation of how to call it, not the agent calling it, so it shouldn't trip the gate. And the strong signal (exact mcp__ strings) carries the errors while the weak signal (short names) carries only warnings. Both choices come from the same instinct: a precise gate that fails hard on certainty and merely whispers on a guess is one people leave on.

The result: one real defect, found by machine

The check ran against all 317 agents and surfaced a genuine pre-existing defect that had been sitting in the repo, green, the whole time. data-collector.md invoked six mcp__umami__* tools in its body and declared only Read in its allowlist.

Read that again in runtime terms: the agent whose entire purpose is collecting analytics data could not make a single analytics call. Every one would have been blocked. It wasn't doc-drift — it was a latent outage the frontmatter-only schema had no way to see. The fix was to declare the six tools, because the agent genuinely needed them.

That single catch is the whole argument for the check. One agent, wrong in a way that looked fine, in a fleet too large to read by hand, found deterministically the moment the validator was taught to compare declaration against behavior. And notice it was an under-declaration — the body asked for more than the allowlist granted. A privilege auditor that only worried about agents asking for too much would have walked right past it. The check has to run both directions because the silent runtime failure lives in the direction most reviewers don't think to look. Nine new unit tests pin the check's behavior; all 317 agents validate to zero agent-level errors.

Surface	What it claims	What enforces it
`tools` frontmatter	the agent's allowed capability	the runtime — undeclared tools are blocked
agent body	the agent's actual behavior	the model, following the prose
consistency gate (3.11.0)	declaration and behavior agree	CI — ERROR on a fully-qualified mismatch

The principle: declared capability must be machine-checked against behavior

Strip away the specifics and this is the same boundary lesson as an MCP server's write gate. There, the client-side tool gate was UX and the server-side gate was the boundary, and the mistake was confusing the two. Here, the allowlist is a declaration and the body is the behavior, and the mistake is assuming they agree because you wrote them in the same file.

They don't agree until a check makes them. A declaration nobody verifies decays into a comment — and a comment that's also a runtime gate is a latent failure waiting for the worst moment to surface.

The move generalizes past Claude Code agents. Any place a system declares a capability separately from where it exercises it — an RBAC scope versus the endpoints it guards, a capability manifest versus the syscalls a process makes, a type annotation versus what a dynamically-typed function actually returns, an OpenAPI contract versus the handler — has this same fault line. The declaration is cheap to write and ages into a liability the day it stops matching the behavior, because nobody was checking.

The fix is never "be more careful when you write the declaration." Care doesn't survive contact with 317 files and a year of edits. The fix is a deterministic check that reads both sides and fails when they diverge — and it has to live in CI, not in a reviewer's discipline, because the whole problem is that the discipline doesn't scale. Once the check exists, the property it guards stops being a thing you hope is true and becomes a thing that can't merge while false.

Scale is what turns this from nice-to-have into non-negotiable. At one agent, the author is the auditor and the body fits on a screen. At 317, the only auditor that scales is a check in CI that reads every body against every allowlist on every push — and that's the difference between asserting your agents are correct and proving it.

Tradeoffs

The body check is heuristic, not an AST. It matches text patterns — fully-qualified mcp__server__tool strings and backtick verbCamelCase mentions — so it cannot catch a tool invoked through a dynamically constructed string in the body, and it scopes the short-name heuristic narrowly on purpose.

That's a deliberate trade of recall for precision: a gate that throws false ERRORs gets disabled, so the certain signal fails hard and the uncertain signal only warns. Strong over complete.

Stripping fenced code means a tool that appears only inside an example block is invisible to the check. Acceptable, because an example isn't behavior — but worth naming, because it's the one place a real call could hide.

And the check bites hardest on MCP tools specifically, because their fully-qualified namespace makes them matchable with confidence. A plain Bash-vs-Read mismatch isn't gated the same way, since those canonical tools don't carry an unambiguous call signature in prose. That's fine by design: the MCP surface is the privileged, high-blast-radius one, and that's exactly where you want the allowlist to be provably honest.

Also shipped

The same day, the team knowledge-base work continued on a parallel track: the qmd adapter got a native FTS5/BM25 keyword backend with no external binary, a retrieval eval harness reporting Recall@10 and nDCG@10, and SHA-256 pinning of the retrieval-model weights that fails closed if the hash doesn't match. Plus a round of CI cleanup across the plugins repo. Each is its own thread; here they're the backdrop to a day whose lead story was making 317 agents' declared tools provably match what they actually do.

Related posts:

MCP Server Auth: The API Is the Real Boundary — the same lesson on a different surface: the declared gate and the enforced gate are not the same thing, and confusing them is how security theater ships.
A Second Brain You Can Audit Beats One You Must Trust — replace every promise with a command that checks it, applied to a governed knowledge store.
Green CI Proves Nothing: Why Your Tests Gate Zero Calls — the same instinct turned on tests: a green check that verifies nothing is worse than no check at all.

A Second Brain You Can Audit Beats One You Must Trust

Jeremy Longshore — Fri, 26 Jun 2026 15:45:25 +0000

Most "second brain" products ask for the same thing in the end-user license agreement: trust us. Trust that your notes stay where you put them. Trust that the model only saw what we told you it saw. Trust that the record we keep is the record that happened. None of those are verifiable from the outside — they're promises, backed by a privacy policy and a logo.

The Governed Second Brain plugin shipped this day inverts that contract. It's a local-first, in-process MCP server — read and write, daemon-free — that you install over a folder of your own files and query inside Claude Code.

The whole design is organized around one substitution: replace trust me with check it yourself. Where a hosted RAG product would say "your data is safe," this says "here is the command that proves what ran, what left, what changed, and what you installed." Four audit questions, four design choices. This post is about why each one is structural and not a feature you could bolt onto a trust-me product after the fact.

The problem: governed knowledge can't run on faith

Retrieval over a personal or team knowledge base is a solved demo and an unsolved product. The demo is easy: embed some documents, do nearest-neighbor search, stuff the hits into a prompt. The product is hard because the moment the knowledge is governed — client material under contract, regulated data, an institutional memory other people rely on — the interesting questions stop being about retrieval quality and start being about provenance.

Second brains attract the trust-me default more than almost any other category, and the reason is a mismatch. The data is the most sensitive a person owns — private notes, half-formed ideas, client material under NDA — while the architecture most products wrap around it is the most opaque: a hosted index, a sync loop, an embedding API, all behind a dashboard. You are asked to put your most guarded material into the system you can see the least of. That trade is fine for a to-do list and unacceptable for governed knowledge.

Three of the questions that follow have no good answer in a hosted, trust-me architecture:

What actually ran against my files? A cloud service runs code you can't see on data you can't withhold. "Local-first" in marketing copy frequently means "we cache locally and sync everything anyway."
What left the machine? RAG pipelines egress document text to an embedding or completion API by default. The user usually finds out which bytes left by reading a SOC 2 report, not by reading their own logs.
Did my record of what happened get quietly rewritten? This is the subtle one. A knowledge store accumulates a history — what was promoted, transitioned, superseded. If that history can be edited after the fact and nobody can tell, then "audit trail" is a decoration.

These three plus a fourth — what did I install — are the questions a governed knowledge store has to answer to be worth governing. The rest of this post is the four answers, each a design choice rather than a disclosure.

The plugin's umbrella repo got renamed this day from Compile-Then-Govern to Governed Second Brain with an explicit commit note: "Apache-2.0, honest audit claim." That second clause is the thesis in three words. The rename forced the README to stop overstating what the audit does and start stating exactly what it does — because the previous claim turned out to be falsifiable, and an audit feature you can falsify is worse than none.

Choice 1 — local-first and daemon-free: you can see what runs

The plugin is an in-process MCP server. There is no background daemon, no :3847 HTTP service to keep alive, no separate process holding your data open. The earlier incarnation of this system split the write path across a long-running curator daemon; folding it in-process removed a whole class of "is the service up, is it the right version, what is it doing while idle" questions.

The full loop runs in one process: capture a fact, govern it through dedupe and policy, promote it, write the hash-chained audit entry, export, index, and answer a query with a qmd:// citation pointing back at the source. That entire path — capture → govern → promote → audit → index → cited search — executes against local SQLite and a local vector index with no service in the middle. The verification criterion for the first phase was exactly this loop returning a cited hit from local data, with the audit chain verifying intact afterward. No daemon ran; nothing was waiting in the background to be trusted.

Daemon-free is an auditability choice before it's an operability one. A background service is a thing that runs when you aren't looking. An in-process tool runs only when a tool call invokes it, inside a session you initiated, and stops when the call returns. The surface you have to reason about is the tool list, not a process tree. "What runs against my files" collapses to "what tools did this session call" — which is a question the transcript already answers, without you having to take anyone's word for it.

This is the difference between "local-first" as an architecture and "local-first" as a marketing word. Plenty of products that wear the label cache locally and sync everything to a server anyway; the local copy is a convenience, not a boundary. Here the local copy is the system — there is no server-side index to drift out of sync with, because there is no server. The data plane and the trust plane are the same machine.

Choice 2 — no egress by default: you can see what leaves

Building the brain has two modes, and the default is the conservative one:

# Build a governed brain from a folder — zero LLM egress.
npx governed-second-brain init ./notes --index-only

# Opt into the full compile (document text egresses to the model).
npx governed-second-brain init ./notes        # pre-flight consent first

init <folder> --index-only builds the entire brain — index, governance, cited retrieval — with zero egress. No document text leaves the machine; the index is built locally and queries resolve locally. This is the mode for regulated or client data, and it is the default precisely so that "local-first" cannot quietly imply "nothing leaves" while a compile step ships your files to an API behind your back.

The richer mode is the full ICO (Intentional Cognition OS) compile, which runs on DeepSeek as an opt-in egress path. That mode genuinely sends document text to a model — and the design's honesty rule is that this fact is loud, not buried. The egress mode is gated behind a pre-flight consent step that names what it does on the tin: it reads every file under the target folder, runs local tooling, and — in this mode only — sends document text to the model. Consent is a step you pass through, not a flag you forget you set.

This is also why the default is index-only rather than full-compile. Defaults are the decisions users don't make, so the default has to be the one that's safe to not think about. The README is correspondingly forbidden from letting "local-first" do the work of implying "air-gapped." Two modes, one honest sentence each: index-only sends nothing; full-compile sends document text to the model, and asks first.

The pre-flight summary is itself an auditable artifact. It is a printed contract — these are the files I will read, these are the tools I will run, this is what I will send and to whom — presented before you approve, not logged after you've already lost the choice. A consent screen you can't decline is a notification; a consent screen that gates the action is a decision point. This one gates.

The point isn't that egress is bad. It's that egress should be a decision the user makes with the facts in front of them, not a default they discover later in someone else's compliance document. Auditability of what leaves means the answer is knowable before the bytes move, not reconstructable after.

Choice 3 — the external audit-chain anchor: you can see what changed

This is the design choice that earned the "honest audit claim" rebrand, because the original claim was wrong in an instructive way.

The knowledge store keeps a hash-chained audit log: every governance event (a memory promoted, transitioned, superseded) is an entry whose hash folds in the hash of the previous entry. The intuition is the familiar one — change any record and the chain breaks, so the log is tamper-evident. That intuition is half right, and the missing half is the whole problem:

# Illustrative — what a self-contained hash chain does and does not catch.
chain:  e0 → e1 → e2 → e3      entry_hash = H(row || prev_entry_hash)

# Edit e1 in place but DON'T re-hash e2, e3  →  chain breaks  →  caught. Good.
# Silently REWRITE the whole chain (recompute every hash)    →  internally
#   consistent again  →  self-verify still passes.            →  NOT caught.

A self-contained chain only proves internal consistency. It catches a sloppy edit that doesn't recompute everything downstream. It does not catch a wholesale rewrite, because a rewrite recomputes every hash and produces a new chain that is perfectly consistent with itself. A local writer can throw away the history, fabricate a clean replacement, and verify returns green. Calling that "tamper-proof" was the overstatement the rebrand corrected; the accurate word is tamper-detection, and only against in-place edits.

The external anchor closes the rewrite gap. The mechanism is two commits working as a pair: govern commits the chain head to an external anchor log, and brain_audit_verify checks the live chain head against that anchored value. The companion store change in qmd-team-intent-kb is described exactly this way — "external anchor log for the audit chain (detect silent full rewrites)."

# Illustrative — the anchor is an outside witness to the chain head.
govern:               anchor.log  ←  head_hash(e3) = "9f66…"   (committed externally)
brain_audit_verify:   live_head == anchored_head ?  intact  :  SILENT REWRITE DETECTED

A wholesale rewrite is internally consistent, but it produces a different head hash than the one already committed to the anchor. The anchor is an outside witness: it remembers what the head was, so a fresh chain that disagrees with it is caught even though the fresh chain agrees with itself.

The trust model then gets stated honestly per mode rather than as one blanket claim. A local single-writer install gets integrity, ordering, and rewrite-detection — that is genuinely what the chain plus anchor deliver, and it's a real guarantee worth having. Non-repudiation across multiple distrusting actors is a different, stronger property that needs keyed signatures and a tamper-resistant anchor, and the local mode does not pretend to it. Stating the weaker-but-true claim for the common case, and naming where the stronger claim would require more machinery, is the honesty the rebrand was about. The forbidden-words list is short and deliberate: tamper-proof, immutable, blockchain. The feature is real; the claim is now sized to it.

Choice 4 — reproducible, provenanced install: you can see what you installed

Auditability that stops at runtime leaves the biggest hole open: the install itself. A second brain you can audit, delivered by a supply chain you can't, is still trust-me — you've just moved the trust from the data plane to the package manager.

So the install is pinned and provenanced end to end:

gsb.lock.json pins the exact version tuple — the plugin runtime, the ICO compiler, and qmd (pinned to the canonical 2.5.3). The spool contract between compile and govern is a versioned wire format, so the lockfile owns a single known-good combination rather than letting three components drift independently.
A hermetic full-chain CI smoke runs the entire capture-to-cited-search loop against that pinned tuple before any release, with a real corpus and a real audit verify. The reproducible build is tested as a whole, not asserted component by component.
npm provenance is generated by the CI release workflow, so the published package carries a verifiable link back to the commit and workflow that built it.
The package ships only plugin-runtime/governed-brain.cjs, not a vendored node_modules. A stripped, self-contained runtime artifact is one file you can inspect, not a dependency forest you have to take on faith. (An early publish accidentally externalized dependencies that weren't bundled — an inert package — which is exactly the failure a single self-contained artifact prevents.)

The tuple matters more than a normal lockfile because the three components talk to each other over a versioned wire format. The compile stage produces a spool that the govern stage consumes; that spool contract can change shape between versions. Pinning the plugin runtime, the compiler, and qmd as a single tuple — rather than letting each float to its own latest — is what keeps the contract coherent. The lockfile owns a combination that was proven to work together, not three independently-blessed versions that happen to be installed at the same time.

These read like packaging chores. They're the fourth audit axis. "What did I install" has a verifiable answer — a provenanced package, a pinned tuple, one inspectable runtime file, a CI smoke that proves the whole chain works at those versions — instead of "whatever npm resolved today."

The result

By end of day the plugin was live as npx governed-second-brain init, version strings aligned at 0.1.4 (a finding the validate-plugin gate surfaced), the MCP server auto-registering with Claude Code so a fresh install can /brain "..." against its own data without manual config. The umbrella repo carries the productization epic and its phased child work; the audit-chain anchor moved from "on the roadmap" to "implemented" in the same pass that shipped it.

The auto-registration is a small thing that matters for the thesis. The installer doesn't hand you a block of .mcp.json to paste and hope you got right — it registers the server with Claude Code itself, prints a first /brain query and a capture example, and the loop is live. An honest, checkable system that's also annoying to wire up loses to a trust-me product that just works; closing the setup gap is what lets the auditable option win on convenience too, not only on principle.

The honest summary is the four audit questions, each with a command behind it rather than a promise:

Audit question	Design choice	How you check it
What runs against my files?	Local-first, daemon-free in-process MCP	The session transcript — no background process
What leaves the machine?	No-egress `--index-only` default; opt-in compile with consent	The mode you chose, stated before bytes move
Did my history get rewritten?	External audit-chain anchor	`brain_audit_verify` vs. the committed head
What did I install?	`gsb.lock.json` pin + npm provenance + single `.cjs`	The lockfile, the provenance, one inspectable file

The principle: replace the promise with the command

Strip away the specifics and the design is one move applied four times. Every place a hosted product would hand you a promise, this hands you a command.

"Your data stays local" is a promise. "There is no daemon and the loop runs in-process" is a fact you can confirm from the tool list. "We only access what you authorize" is a promise. "Index-only egresses nothing; the other mode asks first" is a behavior you can observe before any bytes move. "We keep an immutable audit log" is a promise — and, as it turned out, a falsifiable one. "brain_audit_verify checks the live chain head against an externally committed anchor" is a command that returns a verdict. "Our package is trustworthy" is a promise. "Here is the provenance, the pinned tuple, and the one file we ship" is a manifest you can read.

The pattern generalizes past second brains. Any system that touches data you don't fully control faces the same fork: it can ask you to trust an unobservable property, or it can expose an artifact that makes the property checkable. The first is cheaper to build and ages into a liability the day someone asks you to prove it. The second costs more up front — an anchor log, a consent gate, a lockfile, a provenance step — and the cost buys you a claim that survives scrutiny.

The discipline that made this honest was the willingness to shrink a claim to fit its mechanism. The original "tamper-proof" language was aspirational; the mechanism underneath only delivered tamper-detection, and only against careless edits. Rather than build toward the bigger claim or quietly keep the wrong word, the rebrand sized the words to the substrate — tamper-detection where the chain earns it, rewrite-detection where the anchor earns it, and a short list of words the system is forbidden from using because it hasn't earned them. A claim you can defend is worth more than a claim that sounds better, and the gap between the two is exactly where trust-me products live.

Tradeoffs

None of this is free, and the costs are worth naming. The external anchor adds a write on every govern and a check on every verify — cheap, but it means the anchor log is now load-bearing state the system has to keep somewhere durable. And it raises an honest recursion: where do you anchor the anchor? An anchor log that lives next to the chain it witnesses is no witness at all — a rewriter who can touch one can touch both. The substrate that makes it real is an external, append-only, shared history that the local writer doesn't unilaterally control — committing the head into git history is the cheap version, a timestamping service the stronger one. The anchor is only as trustworthy as the place it's committed to, and naming that ceiling is part of the honest claim.

Index-only mode trades retrieval richness for zero egress; the fuller compile is genuinely better at some queries, which is why it exists, which is why the consent prompt has to be honest rather than discouraging. The in-process design has its own ceiling: there's no shared service, so a brain that a whole team queries at once wants the hosted path — and that path is deliberately out of scope here, a client opt-in rather than the default, because hosting reintroduces exactly the trust surface this version removed. A single pinned tuple in gsb.lock.json buys reproducibility at the cost of staleness — the known-good combination is known-good until a security fix lands upstream and the pin has to move deliberately. Each trade leans the same direction: a little more ceremony in exchange for a claim you can actually back.

Also shipped

contributing-clanker cut its v0.2.0 release with new contribution-quality gates and a versioned product-landing README. intent-outreach got a ground-up rebuild — provider-pluggable LLM seam, a deterministic pipeline behind an MCP server, the GCP stack retired — and reached marketplace grade B on validate-skillmd. Both are their own posts; here they're context for a busy day whose lead story was making a knowledge store you can check instead of one you have to believe.

Related posts:

MCP Server Auth: The API Is the Real Boundary — the same knowledge system, and why the governance audit trail must stay pure for verification to mean anything.
Governed Knowledge: Two Releases, a Freshness Daemon, and Export Gating — earlier groundwork on governing a knowledge corpus rather than just retrieving from it.
Green CI Proves Nothing: Why Your Tests Gate Zero Calls — the same instinct applied to tests: a green check that verifies nothing is worse than no check.

Green CI Proves Nothing: Why Your Tests Gate Zero Calls

Jeremy Longshore — Fri, 26 Jun 2026 15:43:59 +0000

Your test passed. It gated zero tool calls. It proved nothing.

What the Dogfood Had to Prove

The agent-governance-plane (AGP) is a Slack-native, OSS (Apache-2.0) governance gate for Claude Code. It runs the agent inside a Docker sandbox, checks every tool call against a policy engine, gates suspicious ones through human approval in Slack, and writes each decision to an Ed25519-signed, hash-chained audit journal you can verify offline with agp verify. It holds no credentials. It fails closed on anything unverified. Phase B, pre-1.0.

The live-dogfood task was straightforward: prove that a REAL Claude Code harness actually obeys the gate end-to-end, not in a mock test. Three increments, each independently shippable.

Increment 1 wired the gate. A hook-bridge.ts parses Claude Code's PreToolUse hook (the real measured contract), asks AGP's gate over a Unix socket, and translates the verdict back. Allow = exit 0. Deny = exit 2 + reason on stderr. Fail-closed by construction: a missing verdict is never an allow.

// allow → exit 0; deny → exit 2 with the reason on stderr (the measured contract).
const ALLOW: BridgeResult = { exitCode: 0 };
const deny = (reason: string): BridgeResult => ({ exitCode: 2, stderr: reason });

// Fail-closed: connection failure, timeout, a deny verdict, or any non-allow
// response all return a deny. A missing verdict is never an allow.
if (res.kind === "policy_verdict" && res.verdict.decision === "allow") {
  return ALLOW;
}
return deny(`AGP gate denied: ${reason}`);

Increment 2 placed the harness in a network-enabled Docker container with real FS/process isolation. The hook still gated every call through the host control-plane gate over a bind-mounted Unix socket. The image was hardened: --cap-drop ALL, --security-opt no-new-privileges. API key passed by name, never in argv. Live validation on a real task: Read allowed, Bash denied.

Increment 3 was the reproducible CI dogfood. A .github/workflows/dogfood.yml workflow runs the harness and assembles an evidence bundle: journal, pubkey, agp verify output, event timeline, filled AAR. All artifact, cryptographic proof — no service dependency, since the signed hash-chained journal verifies offline. Run by hand first against an actual flake in the claude-code-slack-channel repo, it worked: 95 journal events, 47 tool calls gated — Claude tried Agent, Bash, and ToolSearch (denied), then dozens of Reads (allowed). agp verify confirmed chain, signatures, and signed head.

Why Not Run Claude Inside the Sandbox?

The obvious design is to run the harness inside AGP's own sandbox. That sandbox is --network none — an actively-verified, fail-closed default. But real Claude needs network egress to reach the model API. "Claude inside a --network none sandbox" is self-contradictory: the harness can't start.

So Topology B puts the harness in a network-enabled container with real FS and process isolation of tool execution, while every tool call still gates through the host control plane over the bind-mounted socket. The honest limitation: that container has full egress, not a model-only allowlist. A model-only egress allowlist (Topology C) is the north star — a real networking subsystem, filed as its own work, not a v0 blocker. Same class of honest limit as "Docker, not Firecracker."

The proof never depends on any of it. AGP's proof of what a governed run did is the signed, hash-chained journal, verifiable offline with the published key. No model provider, no hosted log, no third party is ever in the trust path. That independence is the moat — make provability depend on an external service and you weaken it.

Why the Green Run Proved Nothing

The first CI dogfood run went green. It was hollow.

Zero tool calls gated. The containerized intendant — AGP's per-harness adapter, the thing that actually runs Claude under the gate — could not connect to the host-process gate socket. The bind-mounted socket existed (ls confirmed it), but connect() returned ENOENT. The run completed. The workflow passed. It had exercised nothing. Note what did not happen: the gate's own fail-closed guarantee never fired, because the harness never reached the gate. The hollow green lived in the CI evidence path, one layer above the policy gate.

This reproduced on a clean CI runner. That mattered. An earlier ADR (037) had blamed the dev sandbox's filesystem virtualization for the same symptom; a clean-runner repro falsified that — there was no dev sandbox left to blame. It was a real socket-sharing bug: a single process, Bun, spawning Docker and multi-mounting one Unix socket. We corrected the ADR to name the real cause, reopened the tracking issue for the container socket bug (it had been closed prematurely on the false diagnosis), and made evidence-bundle.sh fail closed when the gate was never exercised.

event_count=$(wc -l < "$OUT/events.txt" | tr -d ' ')
gated=$(grep -cE 'gate\.(allow|deny)' "$OUT/events.txt" 2>/dev/null || echo 0)

# ... assemble journal, pubkey, agp verify output, AAR ...

# A dogfood that gated nothing proves nothing — fail closed (no fake-green).
if [ "${gated:-0}" -eq 0 ]; then
  echo "evidence-bundle: FAIL — 0 tool calls gated; the governed run did not exercise the gate" >&2
  exit 1
fi
echo "evidence-bundle: wrote $OUT ($event_count events, $gated gated, verify rc=$verify_rc)"

The Host-Path Pivot

We pivoted CI to the host path: Claude installed on the ephemeral runner, gated, signed. The runner itself is the isolation. That path works end-to-end and is genuinely reproducible. The container path (Topology B) stays the north star — the socket bug stays open, not papered over.

Preventing Hung Runners in CI

The first host-path dogfood hung. A fresh-runner Claude produced no output before stalling — likely first-run, no-TTY behavior, not AGP. A CI dogfood must never hang. We wrapped the run in timeout 360 and set the job timeout-minutes: 20, so a stuck harness fails fast instead of burning the runner.

jobs:
  dogfood:
    timeout-minutes: 20      # a stuck harness fails the job, never burns the runner
    steps:
      # ...
      - name: Governed run
        run: |
          timeout 360 bun src/cli/index.ts run --intendant claude-code \
            --task "$TASK" --repo "$REPO"

The Transferable Lesson: Assert the Work Was Done

The transferable lesson: an integration test that exercises zero of the thing under test is worse than a red one, because that fake-green reads as "covered" when it covered nothing. The fix is not "trust the green." It is to assert that the test did the work. For a governance gate, that assertion is gated_count > 0. A green check is a claim. The evidence bundle is the proof. This generalizes everywhere: any test whose green can be reached without exercising the property it claims should fail closed when the property was not exercised.

Also shipped: the sprite→intendant rename (ADR 038). Fly.io ships "Sprites" at sprites.dev—stateful sandboxes for AI agents with Claude Code as an explicit use case. A direct product/lane collision. "Intendant" is the agent-noun of Latin intendere, the root of intent, "one who executes on behalf of an authority." And the cross-repo echo: CCSC (AGP's substrate) shipped a "footgun-inversion" regression-test epic the same week, pinning fail-closed defaults (session isolation, "every policy decision is journaled—no gaps"). The same move as the no-fake-green guard: prove the safety property holds rather than trust it. This echoes the principle we follow in honoring the gate when the verdict is inconvenient — the gate is only as good as your commitment to enforcing it even when it blocks your path.

Related Posts:

{
"@context": "https://schema.org",
"@type": "BlogPosting",
"headline": "Green CI Proves Nothing: Why Your Tests Gate Zero Calls",
"description": "CI dogfood for AI-agent governance went green while gating zero tool calls. Here's why a passing test proving nothing is worse than a red one.",
"datePublished": "2026-06-11T09:00:00-05:00",
"author": {
"@type": "Person",
"name": "Jeremy Longshore"
},
"publisher": {
"@type": "Organization",
"name": "Start AI Tools",
"logo": {
"@type": "ImageObject",
"url": "https://startaitools.com/images/og-image.png"
}
},
"articleBody": "Your test passed. It gated zero tool calls. It proved nothing. The agent-governance-plane (AGP) is a Slack-native, OSS (Apache-2.0) governance gate for Claude Code. This post explores why a passing test that exercises zero of the thing under test is worse than a red one, and how to prevent fake-green dogfood runs in CI.",
"keywords": ["ci-cd", "ai-agents", "testing", "claude-code", "docker", "devops"]
}

Spec graduation: when a partner email rewrites architecture

Jeremy Longshore — Fri, 26 Jun 2026 15:43:56 +0000

The cleanest architectural moves of the last six months didn't come from a whiteboard. They came from a partner email that essentially said: hey, remember what we actually signed.

Yesterday's post was about coherence as a deliverable — a full audit-day where four advisor agents independently flagged the same drift in the same engagement. The drift was real, but the post stayed at the diagnostic layer. It identified the misalignment without committing to the structural change that misalignment implied. Today is the structural change.

The drift

The kobiton/CLAUDE.md for the M2 milestone work said, in plain text, that the primary venue for "Blog 1" was startaitools.com. That sentence had been there since the file was first scaffolded, and nobody had questioned it. It read like settled fact.

It wasn't. It was a gloss. Somewhere during SOW negotiation a verbal "we're flexible on venue, you do what makes sense" had been transcribed into the operating doc as "primary venue: startaitools.com." Internal interpretation became internal canon, and internal canon then started shaping how I scoped everything downstream — what the post would argue, what tone it would use, who the audience was, where it would link to.

The drift was small. The drift was three months old. The drift had compounded into structural assumptions about M3 and M4 already.

The check-in

Frank Moyer is the Kobiton stakeholder on the M-series. He replied to a thread that was nominally about WordPress publishing access. The actual content of his reply was a course correction: Kobiton's blog is the canonical home for Blog 1. Cross-publication elsewhere — startaitools.com, the personal portfolio, anywhere — needs to align on canonical strategy ahead of publication.

He was right. He didn't argue the point, he just stated the position the contract supported. My reply was "10-4." It was the only register that fit.

The re-read

This is the part where, if I'm being honest, the temptation was to negotiate. There was a half-formed counter forming in my head about cross-promotion, about audience reach, about how methodology content benefits both parties.

I closed the email tab and pulled the actual signed SOW from kobiton/000-docs/001-BL-CNTR-. I'd opened that document maybe three times since signing it.

The project title — the literal first line on the document — included the word "Promotion." The M2 milestone description required positive supportive promotional tone and content reinforcing plugin value. Venue was not specified, but the framing was unambiguous. Frank's read of the contract was well-supported. My kobiton/CLAUDE.md gloss was not.

This is a small humiliation worth sitting with for a moment. The contract is the artifact. The CLAUDE.md is a working note. When they disagree, the working note loses every time, and the working note had been silently steering decisions for a quarter.

The boundary

The fix was a new section in kobiton/CLAUDE.md titled "Content boundaries." Two columns:

Contracted M2 deliverables — three blogs, Frank-coordinated, canonical at kobiton.com/blog, vendor-specific framing per the SOW, positive promotional tone, plugin-value-reinforcing. This is theirs. I'm the author of record but Kobiton owns the venue, the timing, and the framing constraints.

Independent methodology track — anything that generalizes across engagements: the intent-eval-lab spec, the matcher-map template, public technical comments on GitHub, anything that lives at the "shape of the audit" layer rather than the "this specific plugin" layer. This is mine. Authoring domain, venue, timing all under my control.

The two columns aren't in tension. They're orthogonal. A finding I make while auditing the Kobiton plugin can produce both a Kobiton-specific blog (theirs) and a contribution to the vendor-neutral spec (mine), as long as I don't co-mingle the artifacts on one publication track or one CLAUDE.md row.

Writing that section took fifteen minutes. The clarity it produced is the part of this whole episode I keep returning to, because it's the part that unblocked everything else.

The graduation

Here's the spec graduation I hadn't seen until the boundary was on paper.

Four advisor agents — business-analyst, ai-engineer, architect-reviewer, content-marketer — had each separately produced reviews over the prior week recommending some variant of the same structural move: stop being the auditor of one plugin, become the methodology layer that Kobiton is one instance of.

The business-analyst framed it as positioning: "auditor of N plugins" doesn't compound; "author of the conformance spec N plugins are graded against" does. The architect-reviewer framed it as artifact taxonomy: the matcher-map template is being reinvented for every engagement and that's a sign it wants to be a vendor-neutral artifact. The ai-engineer framed it as eval-loop reusability. The content-marketer framed it as audience: methodology content has a different reader than vendor-specific content and the two were eating each other.

Four roads, same destination. The stall was recent — two weeks — but the misalignment feeding it was three months old: the same CLAUDE.md gloss that misread the SOW's venue position was also keeping the methodology entangled with vendor-specific framing. Pushing the methodology to a public vendor-neutral spec while simultaneously delivering vendor-specific promotional content for one of the vendors-in-question is a real tension if you don't have a content boundary.

With the content boundary in place, the tension disappears. The spec lives on the independent methodology track. The Kobiton blogs live on the contracted track. Neither contaminates the other.

This is the inversion.

Until Frank's email, the engagement architecture was: Kobiton M-series is the primary work, methodology extraction is a side effect. After the boundary, it's: Methodology spec is the primary structural asset, Kobiton M-series is one instance.

The work I do for Kobiton hasn't changed. What it produces, structurally, has.

The artifact

I executed what the architect-reviewer agent had labeled "Option B — soft spec move." Vendor-neutral spec, draft status, public artifact, no marketing announcement. The on-disk layout:

intent-eval-lab/
  specs/
    mcp-plugin-observability/
      v0.1.0-draft/
        SPEC.md
        matcher-map-template.md
        case-studies/
        conformance-test-suite/
    methodology/
      README.md
    validator-contract-reliability/
      README.md
    forecasting-drift-detection/
      README.md
    decentralized-crypto-evaluation/
      README.md

Module 1 — mcp-plugin-observability — ships as v0.1.0-draft. The other three placeholder modules live as siblings under specs/, not children of the first module, because the methodology umbrella is broader than any single shape. The cross-module methodology/ directory sits alongside.

The spec has five normative requirements, R1–R5. Most anchor to a section in Anthropic's published Claude Code monitoring, hooks, or plugins documentation; the structural ones cite RFC 2119 plus the spec's own anchoring rule. The discipline is: I'm not inventing prescriptions, I'm codifying what's canonical and pointing at the source. If a requirement can't cite a canonical doc section or an upstream MCP/OTel spec section, it doesn't go in.

The matcher-map template is the structural piece. Three columns:

finding-shape	hook matcher	OTel signal
race	`PreToolUse` on Bash	`mcp.tool.race.detected` span event
shape drift	`PostToolUse` on Edit	`mcp.context.shape.drift` metric

finding-shape — a class of failure mode that recurs across plugins.
hook matcher — the Claude Code hooks declaration that detects an instance.
OTel signal — what the hook should emit so the failure becomes observable downstream.

The v0.1.0-draft includes six baseline shapes — race, shape drift, cooldown, side-effect verification, mandatory context, strict-mode protocol. Each one is a row that's been instantiated against at least one real engagement. The Kobiton M2 audit produced four of them. The remaining two came from earlier work.

The three module placeholders — validator-contract-reliability (Polygon-shape), forecasting-drift-detection (Nixtla-shape), decentralized-crypto-evaluation (Lit-shape) — are README-only. Each README names a domain-specific failure mode and reserves the structural slot. No timeline, no commitment, no marketing. They exist so the methodology umbrella is visible without the umbrella having to ship every panel at once.

The methodology/ directory is the cross-module layer. Its README enumerates five patterns observed early — how to write findings (Toulmin's argumentation model — claim, grounds, warrant — from Stephen Toulmin's The Uses of Argument, 1958), why diagnostic precedes prescription, the distinction between conformance (does the plugin meet the spec?) and eval (does the plugin do its job?), the vendor-neutrality rule, and the anchoring rule that requires every normative claim to cite a canonical source. Each pattern is provisional today; the README states the promotion gate explicitly: when at least two modules ship to a release-candidate version, the patterns get promoted to properly authored documents.

The runner decision (separate)

The same boundary discipline that partitioned content also forced a smaller separation inside the spec itself: the language the runner is written in is not the language the spec speaks. A second multi-advisor pass — architecture, Go-fit, DX, business-strategy — converged on Go for the reference runner. Reasons stack neatly:

Single static binary distribution. No Python venv, no Node version dance. Plugin authors clone a release and run it.
OTel ecosystem is Go-native. Collector, Tempo, opentelemetry-go-contrib, and Honeycomb's Refinery proxy — all Go. The runner inherits the ecosystem instead of bridging to it.
Fast cold start matters when the runner is in a CLI eval loop being invoked dozens of times per session.
The official MCP Go SDK (modelcontextprotocol/go-sdk) gives clean protocol integration without me writing glue.
Matches the precedent set by the j-rig-binary-eval runner — same operational shape, same maintenance surface for one operator.

The critical separation: the runner-implementation decision is decoupled from the spec-implementation decision. The spec is language-neutral. The matcher-map template is language-neutral. The conformance reports are language-neutral JSON.

A plugin author writing in Python or TypeScript or Rust can author a conformant matcher map, run their own test harness, and produce a valid conformance report without installing Go. The Go runner is one reference implementation. A runner-py/ or runner-ts/ contributed later is a valid contribution, not a fork.

I want to be explicit about why this matters. The cardinal sin of methodology specs is conflating "the spec" with "the tool that checks the spec." When that happens, the tool's language choice becomes a barrier to spec adoption, and the spec dies on a toolchain hill. Keeping the layers separate is not architectural purity, it's adoption strategy.

The discipline holds the line

R3 — the M2 deliverable due May 25 — keeps the vendor-specific framing the SOW originally scoped. No reframing. No spec-aware asides. No "this is one instance of a broader pattern" gestures. Frank gets the deliverable he contracted for, in the tone he contracted for, on the schedule he contracted for.

The public announcement of the spec — a blog post, a tweet, an entry in the methodology hub on this site — is deferred until after M3 ships. At that point I'll go back to Frank, walk him through the spec, ask for consent on a co-credit announcement, and time it accordingly. Same compounding upside, lower partner-relationship risk, three or four weeks slower timeline.

The trade is correct. A clean structural reframing that risks the partner deliverable is strictly worse than a slower reframing that doesn't.

What this changes for future engagements

Every new engagement now lands as "an instance of a shape." The intake question is: which finding-shape does this engagement's failure modes correspond to? If the shape exists in the spec, the engagement inherits the matcher-map row, the hook patterns, the OTel signal definitions, and the conformance criteria for free. The engagement-specific work is then the actual audit: do the hooks fire, do the signals show up, what's the conformance gap.

If the shape doesn't exist, the engagement produces a new module. The Polygon engagement, when it activates, produces validator-contract-reliability content. The Nixtla engagement produces forecasting-drift-detection content. The Lit engagement produces decentralized-crypto-evaluation content. The shape gets added to the matcher-map. The next engagement of the same shape inherits it.

The auditor-of-one model required me to rebuild the audit framework for each engagement. The methodology-layer model means each engagement either is-a (inherits) or extends (contributes). N engagements produces an asset that compounds. N engagements under the old model produced N audit reports.

The takeaways, packaged tight

Six items I'd hand to anyone running a sole-prop AI consultancy or a small AI-engineering team:

Re-read the contract before arguing it. Internal interpretations of verbal context drift. The document doesn't. When a partner pushes back, the first move is the SOW, not the rebuttal.
Partition contracted-deliverable content from methodology-track content. Don't let them share a row in your operating doc, a publication channel, or an authorship surface. The boundary is what makes both possible.
A spec is a structural artifact, not a marketing artifact. Anchor every normative requirement to a canonical published source. If you can't cite it, don't prescribe it. The discipline protects the spec's integrity, and credibility follows.
Decouple the spec language from the runner language. Pick the runner language for the runner's reasons (distribution, ecosystem, cold start). Keep the spec language-neutral so contributors aren't blocked by toolchain. Reference implementations are references, not gates.
Reserve structural slots with placeholder READMEs. Three READMEs that name domain-specific failure modes signal "this is a methodology umbrella" without overcommitting to timelines for modules you haven't earned the right to write yet.
Hold the timeline when the reframing risks the partner deliverable. The spec is forever. The deliverable is May 25. Don't trade a fixed-date partner asset for a marketing window on something that's going to compound for years either way.

Also shipped today, in an unrelated repo: PR #707 on claude-code-plugins — a CSS grid overflow fix for the marketplace landing on iPhone 13 viewports. 1fr → minmax(0, 1fr) on the pcard-hosting grids, minmax(min(320px, 100%), 1fr) on the multi-column auto-fill grids, and min-width: 0; max-width: 100% on .pcard itself. Defense in depth against the nested-flex-grid overflow that 47 px of hidden width was causing. Worth noting only because it landed the same day; it has nothing to do with the rest of this post.

The version on disk is v0.1.0-draft. It will move. The spec that exists now is not the spec I would have written before Frank's email — that one would have been a Kobiton-shaped artifact dressed up as methodology. Instead it's a methodology layer that Kobiton, among others, will be instances of. The shape it took on this morning, after a partner email forced a contract re-read, is the shape it kept.