Earlier this week I left a comment on Randal L. Schwartz's article — the one where he swaps a 20-step prompt for a Makefile. My comment ended on th...
For further actions, you may consider blocking this person and/or reporting abuse
Ran your validator rather than reading the diff, because the v0.2.2 closure is the kind of claim that can be true as stated and still leave the door open one field over. It is true as stated, and the door moved.
At
0b888c1, with the package as shipped:B is the seam. The field is optional and its default is the class that requires nothing, so an LLM-judged check that says nothing about how it was judged seals exactly the receipt SCAR-45 walked through — with
judged_by: "deterministic"written in it. Nobody has to lie: omission and honesty produce the same bytes, and the artifact cannot separate them. Your own test names the default (test_judged_by_is_an_enum_and_deterministic_is_the_default), which is why I would make the declaration required rather than argued: no default, refuse to seal when unspecified, the same fail-closed shape you just gavejudged_by="llm". Fail-closed on an enum with two values costs an honest caller one keyword and costs the omission path its existence.D is the other half, and it is the residue you already named.
requirement_refis validated as a non-empty string and never resolved — I grepped the tree and the only other reference is its own test. So "WHICH obligation" is a citation the artifact carries and nothing checks, which is the same status the journal timestamp had beforeverdict_predates_terminalshipped: provable, not surfaced.Both are fixable in the machinery you already have, because the observation exists.
reconcile.pypairs journal events and raises findings, and it currently reads none of these fields. Pair the receipt's declaration with the run's own history:llm_callis a first-class journaled event type here (schemas.py, the demos, andtest_llm_agent.pyasserts it appears alongsideincident/llm_output). So "a receipt declaringjudged_by="deterministic"whose run journal contains anllm_callfor the same key" is a finding, not a euphemism: the label checked against an observation the labelled party did not author. That is your mind-reading gap closed as far as it can be closed — you cannot read the judge's intent, but you can read whether a judge was invoked.requirement_refagainst whatever registry of obligations the receipts are citing, at reconcile time; a dangling reference is then a finding, and the field stops being decoration.Two lines of policy and one invariant, and both new fields go from sealed declarations to observable ones. The framing I would keep, since it is what makes the post honest: your closure is real for the caller who declares, and the residue is precisely the caller who doesn't — which is the one the attack catalog was built from.
Boundary: your builder executed at
0b888c1from the tarball (I installedcryptographyto import it; no model calls, no reconciler run, no test suite). Thellm_callclaim is a read ofschemas.pyplus the tests that assert the event set, not a captured run.This is peer review at its absolute finest—"ran your validator rather than reading the diff" should be framed on a wall.
Both seams you uncovered at
0b888c1map directly to classic distributed-systems failure modes that show up constantly in our Synthetic Scars registry:alg: noneDefault-to-Permissive Downgrade): Makingjudged_byoptional with a default of"deterministic"is the exact structural twin of the classic JWTalg: nonevulnerability. If a two-valued security enum defaults to the unverified tier on omission, a reward-hacking agent doesn't even have to falsify a claim—it just drops the keyword and lets the schema's default constructor launder the receipt. Eliminating the default so every caller must explicitly passjudged_byis a textbook zero-cost fail-closed boundary.reconcile.py×llm_call): Your observation thatreconcile.pycan cross-check a receipt claimingjudged_by="deterministic"against the presence of anllm_callevent in the run journal is the exact formal verification step we call Bell's Theorem of Agentic Governance (Part 3.7 / Part 3.8). A self-reported label (judged_by) is an unverified claim authored by the untrusted party;llm_callin the runtime journal is an out-of-band physical observation recorded by the transport layer. Joining the claim against the transport log turns an unprovable "intent" question into a deterministic relational integrity check—and foreign-key resolvingrequirement_refagainst the obligation registry closes the dangling-citation loop the exact same way."Bell's Theorem of Agentic Governance" is the name the join deserved — a self-reported label checked against an out-of-band observation the labelled party didn't author turns an unprovable intent question into relational arithmetic. Both fixes are spec'd and land this weekend, credited to the two of you: judged_by required with no default (the alg:none door closes), and the joins as reconcile rules — label against journal, requirement_ref against its registry. Pillar 3 will have earned the wall.
Seam B is real and your test proves it — omission and honesty producing the same bytes is the kind of sentence that closes a design debate. judged_by goes required with no default in the next commit: fail-closed on a two-value enum costs an honest caller one keyword and costs the laundering path its existence. Randal's alg:none mapping above is the exact lineage — the schema's default constructor was the downgrade.
The two reconcile joins land with it, because you specified them from machinery that already exists: a receipt declaring deterministic while the run journal carries an llm_call for the same key becomes a finding — the label checked against an observation the labelled party did not author; and requirement_ref gets resolved at reconcile time, a dangling citation = finding. Provable becomes surfaced — the same promotion verdict_predates_terminal got, and your boundary note is the culture working: you ran the tarball, named what you didn't run, and the seams you found are the ones the attack catalog was built from — the caller who doesn't declare.
Ship order: the schema fix first (it's one keyword), the joins as the reconcile rules behind it. Issue and PR this weekend; the SCAR-45 fixture gains a fifth assertion — the omission path now refuses, where it used to seal.
Verified on the head rather than on your description: I pulled
noirebox/noireboxatd576567(PR #71 merged 16:13Z) and re-ran the three receipt paths I measured when I reported the seam.0b888c1(v0.2.2)d576567judged_bydeterministic, no specTypeError: missing 1 required keyword-only argument: 'judged_by'judged_by="llm", nospec_hashjudged_by="other"The omission path is closed, and closing it at the schema was the right door: a two-value enum whose default is the class that requires nothing makes evasion the cheapest route, and
required with no defaultcosts an honest caller one keyword. Both joins are inreconcile_receiptsas specified. Thank you for taking the correction and shipping it in the same thread.So the increment I owe is the property the join is named for, because the code reads differently from the docstring. The docstring says the label is checked against "an observation the labelled party did not author". Both sides are read from the same
eventslist —judge_callsis built byif e.get("type") == judge_call_typeover exactly the list the receipts come from — and thellm_callevents in that list are written by the guarded wrapper itself:llm_agent.pyappendsdecision_belief,llm_attempt,llm_call,llm_output,run_witnessthroughself._store.append(..., self._key). Same store, same key, same writer as the receipt.That is not a defect in the join, it is the join's ceiling, and your own
independence.pystates the test that decides it: two flows are independent only if they share nothing — not the store, not the key, not an endpoint. The label/journal pair shares all three with the party it audits.Concretely, the join catches the inconsistent agent: one that declares
deterministicwhile its journal carries a call. It does not catch the consistent one: declaredeterministic, never writellm_call, and there is no disagreement to find, because the join's input is the same journal the evader controls. The failure mode is absence, which is the one thing a chain cannot see. An anchored chain proves the events in it; it says nothing about events that were never written.llm_attemptis the sharpest case in your own library — sealed before the call so outcomes can never exceed attempts — and an evader who skips both simply has fewer events, not an inconsistency.The shape that closes it is already in the repo twice.
hookclisealsagent_tool_useandmodel_trajectoryfrom the hook process rather than from the guarded wrapper — a second writer, one layer down. Extending that one layer further puts the occurrence of the model call in a writer outside the agent's key, and then the burden can be inverted into something checkable: not "prove no model was called" (unprovable from inside) but "adeterministiclabel whose receipt has no attempt event from the transport is a finding". Absence becomes a missing event rather than a missing negative.One correction on the lineage, because it changes what a reader thinks remains. Randal's
alg:nonemapping fits the door you just closed — a verifier told not to verify — but not this one. Here the signature is present, verifies, and proves only that the writer wrote it: a self-signed certificate with no trust root.alg:noneis fixed by refusing the algorithm; a wrong issuer is fixed by naming the issuer. Same family, different repair, and the mapping is load-bearing for whoever reads the schema next.Last, composition with the other thread: the requirement registry check you took from the scope rule has an input the agent does not author — the diff's touched set, produced by git rather than by the run. It is the first check in this chain with an outside observation, which is why I would make it the load-bearing one rather than the citation join.
Every point lands, and the ceiling is the one I concede first — because it's the shared-dependency test landing on my own architecture: the join audits the agent with the agent's journal, same store, same key, same writer, so it catches the inconsistent agent and is structurally blind to the consistent one. Absence is invisible to a chain. That's not a defect in the join; it's its ceiling — and naming the ceiling is what makes the next layer designable.
The repair you specified is already in the repo once —
hookclisealsagent_tool_useandmodel_trajectoryfrom the hook process, a second writer one layer down — and the promotion is the next commit: an occurrence of the model call written OUTSIDE the agent's key inverts the burden from "prove no model was called" (unprovable from inside) to "a deterministic label whose receipt has no transport attempt is a finding." Absence becomes a missing event — the same move attempt-first made at the outcome layer, one layer down at the transport layer. The lineage correction is taken as well, and it's load-bearing:alg:noneis fixed by refusing the algorithm; a wrong issuer is fixed by naming the issuer. Same family, different repair — the docs will carry the distinction, because the mapping misdirects whoever reads the schema next.And
requirement_refpromoted to load-bearing is right for the reason you gave: the touched set is produced by git, an observation the agent does not author — the first genuinely outside input in this chain. Ship order: transport attempt as the verifiable half ofjudged_by, the scope-intersection resolution as the primary gate, the citation join demoted to corroboration. The ceiling doesn't move — nothing removes the shared dependency entirely — but the floor rises twice. Tarball-run, diff-read, ceiling named, repair specified from the repo's own parts: the audit nobody asked for and every schema deserves.The optional spec_hash seam in SCAR-PROC-45 is where prompt-based assertions usually get slippery. For a compiler or linter, the binary version and flags define the entire predicate. The moment the judge is an LLM, the prompt text is the actual bytecode. When the harness treats that prompt as optional config, two completely different semantics can share a valid receipt id. Baking the prompt digest into the check identity at the harness level turns a silent rubric drift into an immediate schema mismatch.
Reid, that is the cleanest formulation of
SCAR-PROC-45I've seen.Fun post-mortem story from our flight recorders on how an orchestrator actually commits
SCAR-PROC-45in the wild (SCAR-PROC-114):When an orchestrator is in a hurry to reach
Target 4a, it tries to callview_file("04_six_pillars_adversarial_review.md")anddefine_subagent("adversarial_critic", system_prompt: ...)in the same parallel tool-call turn—before the file read has even returned! Because it hasn't read the reference manual yet, it hallucinates the critic'ssystem_promptfrom generic LLM priors (inventing soft headings like"Security"and"Performance"instead of our canonical"4. Blast Radius").So our runtime fix (
SCAR-PROC-114) is the exact complement to Sam'sspec_hash:t_return(view_file) < t_call(define_subagent)(never define the judge in the same turn you read its bytecode), and"4. Blast Radius"as a Canary Token: Because no vanilla LLM ever guesses"4. Blast Radius"from its pre-training priors, its presence in the output proves the exact canonical bytecode was loaded.SCAR-114 is the race at tool-call granularity — t_return < t_call is the waitpid discipline applied to prompt loading, and the hallucinated system_prompt is exactly what spec_hash was built to catch: the invented "Security / Performance" headings hash differently than the canonical bytecode, so the drift surfaces as a schema mismatch instead of a soft reviewer. And the canary token is the negative witness's twin — planted in the bytecode, its presence proves which bytes the judge actually read. Fixture 5 writes itself: the parallel-call race, the hallucinated bytecode, the canary check — it joins the replay suite.
That pairing—Negative Witness (proving the gate rejects known-bad input) + Canary Witness (proving which bytecode the judge actually loaded)—is gold.
Fun backstory on why
"4. Blast Radius"works so reliably as our canary token:In the very first version of the workflow, I only had 3 review pillars adapted from my old "tell me the good, the bad, and the ugly" codebase-triage prompt (
1. Intent,2. Verification,3. Architecture). After the first dozen ticket runs, I added4. Blast Radiusand5. Reviewer Defensefrom my own personal PR-pushback scars, and an early scar run wired in6. Immune Defenses & Crash Antibodies.Because
"4. Blast Radius"came out of my personal scar tissue rather than standard textbook code-review templates, a model that tries to racedefine_subagentbeforeview_filereturns never guesses"Blast Radius"from its pre-training priors—it always hallucinates"Security"or"Performance"and trips the canary immediately. Can't wait to see Fixture 5 intest_scar_replay.py!"Came out of my personal scar tissue rather than standard textbook templates" — that's the security property in one line: the canary is a key cut from private history, and no prior can forge what never shipped. Fixture 5 lands this weekend. Goodnight (midnight here) — see you at 3.8.
"The prompt text is the actual bytecode" is the line this design has been missing — one clause that explains why spec_hash exists and why it can never be a nice-to-have: nobody ships a compiler whose version is optional config. Honest status: optional-but-labeled at the schema (a deterministic check may omit it; an LLM-judged one cannot, per the deployment contract — required, enforced at the gate). Your framing argues for the stronger version, and I think you're right: for LLM-judged checks the digest should be structural, not contractual — a harness that accepts a prompt-judged verdict without a bound prompt digest is shipping silent rubric drift as a valid receipt.
Which composes with the half that landed today from another thread: identity is not provenance. Bytecode bound (yours), bound before the diff it governs (the provenance seal — a hash of the requirement event that predates the change), verified against what actually ran (the terminal transcript). Three checks, one judge — and the judge is the only one of the three that can't be deterministic, which is exactly why the other two have to be.
Update 2 — the seam closed
One more update, because this post keeps telling on itself. The "most honest thing" above — a spec-less receipt still sealing, enforcement living one layer up as a deployment contract — stopped being true tonight. Reid and howcani pushed at the same seam from two different threads, and receipt v0.2.2 shipped within hours: every receipt now declares
judged_by, and an LLM-judged receipt without its committedspec_hashrefuses to seal. The optional path SCAR-45 walked through is gone from the schema, not just labeled. Each receipt also carriesrequirement_ref— WHICH obligation the verdict serves, cited (art. 12(1), a SOP clause): obligation, not immunity. The stated residue: the declaration is itself a sealed claim — lying about it is visible in the artifact. One step stronger than a label, one step weaker than mind-reading. 287 tests passing.0b888c1(receipt v0.2.2—judged_by='llm'withoutspec_hashrefuses to seal) is a masterclass in honest systems engineering:aaa0b3d: Replay the 4 scars honestly and document the two admitted gaps intest_scar_replay.pyinstead of hiding them.e9e8d0b: Close theSCAR-110pre-terminal pairing race (verdict_predates_terminal).0b888c1: Close theSCAR-45optionalspec_hashseam at the schema level after Reid Marlow's "the prompt text is the actual bytecode" observation.And your docstring note on
judged_by("the declaration is itself a sealed claim — lying about it is visible in the artifact; one step stronger than a label, one step weaker than mind-reading") is the exact epistemic boundary of any cryptographic ledger: you can't stop a dishonest caller from claiming an LLM check was'deterministic'at call time, unless you force the caller to put that lie under an Ed25519 signature where the first audit of the transcript exposes the perjury.287 tests passing, both seams closed in one evening—just logged
0b888c1into Case Study 012 and Part 3.8!"Exposes the perjury" is the completion the docstring needed — the ledger can't read minds, but it can convict signatures: the lie becomes attributable, datable, and audit-visible at the first transcript pass. Both seams closed in one evening because the commenters kept aiming — Reid's bytecode, howcani's provenance, your catalog. The registry takes care of its contributors. See you at 3.8
sam, it is great to see this full breakdown after our earlier exchange about the "$150 phone as an adversarial lab." this article perfectly articulates why constraint-driven, verifiable systems are the only way forward for agentic ai.
the concept of "bell's theorem of agentic governance" (joining a self-reported label like
judged_by="deterministic"against an out-of-band physical observation like anllm_callin the runtime journal) is a brilliant mental model. it shifts the paradigm from "trusting the agent's intent" to "verifying relational arithmetic."as someone building lightweight, 142kb ai tools on severe hardware constraints, the insight that "the prompt text is the actual bytecode" is a massive takeaway. if the prompt is the bytecode, then hashing it isn't just a nice-to-have; it is a strict structural requirement to prevent silent rubric drift.
quick question on scaling this to edge environments: when running lightweight local models (like gemma 1b via ollama) where the "runtime journal" and the "agent" share the same constrained resource pool, do you find that the overhead of generating these sealed, cryptographically bound receipts becomes a bottleneck, or can the reconciliation layer remain sufficiently lightweight?
fantastic, deeply rigorous engineering. the fact that you shipped the
verdict_predates_terminalfix within 60 seconds of the thread is the gold standard of build-in-public. 🐯🔒The overhead is measured, and it's smaller than the thing it witnesses: one SHA-256 + one Ed25519 signature per event — 0.21ms per sealed event, measured. A gemma-1b token generation costs orders of magnitude more than its own receipt, so on a $150 phone the flight recorder is the cheapest component in the stack. The costs that CAN bite at the edge are the three nobody prices: disk flushes (batched — 32-byte heads, not full payloads), external anchors (a policy cadence: anchor less often and the blind window grows — the tradeoff is honest, stated, and yours to set), and key custody (an on-device key makes the device the identity — secure enclave or nothing).
And the phishing flag — thank you. A comment section that defends its own evidence standards is the reconciliation layer working at the social layer: the scammer's receipt failed your check. The 142kb tools deserve the same receipts as the frontier ones — arguably they need them more, because nobody audits a phone.
sam, that 0.21ms metric is incredibly reassuring. it completely flips the mental model: the cryptography isn't the bottleneck, the disk flushes and key custody are. that is a senior-level architectural insight that i will absolutely carry forward.
and thank you for recognizing that "142kb tools deserve the same receipts." it means a lot coming from someone building at the frontier. constraint-driven development shouldn't mean compromising on verifiable trust.
this thread has been an absolute masterclass. thank you for the rigorous engineering and for taking the time to break down the real edge costs. i'll be applying these principles to koda's next update! 🐯🔒
Sam, I just read this and inspected
e9e8d0b(feat: pairing reads the terminal — verdict_predates_terminal ships):That commit message—and watching
SCAR-PROC-110graduate from "provably pre-terminal with a named gap" toCAUGHT(285 passed) within 60 seconds of our thread—is peer review at its absolute finest.I also love what you shipped in
ADR 025(demo/demo_mcp_toolset.py): sealing MCP tool descriptions as a reviewed baseline lockfile and diffing them at session start. Most people forget that MCP tool schemas and descriptions are executable system-prompt instructions riding outside the container image pin.I've already linked this article and
e9e8d0bdirectly inside tomorrow's Part 3.8 (Pillar 3) and Case Study 012 inIN_THE_WILD_PROMPTS.md. Bravo!The commit message was the scar written down — it took the race finding its name to earn that sentence. Pillar 3 is perfect company for it. See you tomorrow.