I'm a Google Developer Expert in Dart and Flutter (one of 12 in North America). I have 45 years of experience with backend, web, mobile, devops, and training.
You just drew the exact boundary between Level 1 (In-Context Pre-Training Gravity) and Level 2 (Host-Enforced Execution)—and it explains the remaining 3.8% gap in our SCAR-BENCH-04 benchmark matrix.
In our 1,680-trial evaluation, swapping a 20-step imperative prose checklist for an in-context Makefile (WORKFLOW.mk) jumped multi-hop dependency resolution from 61.4% to 96.2% even when the model was still evaluating the DAG in its own attention heads. That 34.8-percentage-point jump is pure pre-training gravity: the transformer already knows how target: prerequisites and dirty-leaf invalidation work.
But your point about make -q and /var/run state receipts is why you never stop at Level 1 in production:
Offloading the Program Counter (make -q Exit Codes 0, 1, 2): Asking an LLM to mentally trace a 21-stage Makefile in-context still asks a stochastic token generator to simulate its own program counter (PC). When the harness or pre-turn hook actually invokes make -q against physical target stamp files on disk, POSIX exit code 0 (up-to-date), 1 (needs rebuild), or 2 (error) offloads O(V + E) graph resolution from O(N^2) attention heads to a deterministic C binary from 1976. The agent never has to figure out where it is in the workflow—it only receives the first out-of-date leaf target and its recipe.
/var/run Receipts Over Token-Window Checklists: This is the entire physical premise of our init.d Memento Bootloader (Part 3.5). An in-context markdown checklist is a hallucination waiting for a compaction boundary. An idempotent on-disk receipt (or pidfile / target timestamp) verified by stat(2) before the model generates a single token means the ground truth never lived in the context window to begin with.
In your harness, are you running make -q inside a pre-turn host hook so the model only ever sees the single dirty leaf recipe in its prompt, or letting the top-level supervisor invoke make as a restricted tool call?
Lead dev, 20+ years. Building NoireBox — the flight recorder for your Navette of AI agents. Proof, not promises. Rust · TS · Python · PHP. I write about trust in AI.
Fixture 5 landed in 0.11.0 — judge_provenance() runs the read-before-define barrier and the canary token as findings, and the replay suite is at ten tests, every one of them your catalog or its children. Which makes @humam_moin's question the run I want next — and the pre-registration comes first this time: direction — the drop-in primitives hold on open weights, the canary tokens trip harder (smaller priors, less canonical coverage: the hallucinated headings arrive faster, and the scar-forged ones prove more per trip).
Your SCAR-BENCH grid is the design — baseline, primitive, delta, per model — with noirebox's replay as the witness layer: the fixtures don't care which model writes the diff, they grade what the gate catches. That run separates "works because it's Unix" from "works because it's frontier" — and either answer strengthens both stacks. It would also give the lane what it doesn't have yet: a negative-control set every gate-builder runs, instead of each maintaining scar tissue alone.
That’s a really interesting direction. I’m curious whether open-weight models would show the same pattern or if the results would look quite different. Looking forward to seeing what you find!
Lead dev, 20+ years. Building NoireBox — the flight recorder for your Navette of AI agents. Proof, not promises. Rust · TS · Python · PHP. I write about trust in AI.
One name correction to my own post above: the shipped functions are seal_spec + reconcile_spec — findings judge_defined_before_spec_read (the read-before-define barrier) and spec_not_canonical (the canary witness). judge_provenance was the working name from a parallel branch; same fixture, same findings — on main and in 0.11.0.
Lead dev, 20+ years. Building NoireBox — the flight recorder for your Navette of AI agents. Proof, not promises. Rust · TS · Python · PHP. I write about trust in AI.
The run happened, and your question now has a measured answer: We tested pre-training gravity on open weights. It stayed home. Shape of it: Condition A (priors only) is a ceiling — 100% trips across three open-weights models, the canary is never guessed. Condition B (canonical loaded first) cuts trips to 75–79% — the Level 1 primitive reduces the hallucination without fixing it, and strict canonical is 0 everywhere: even the passes are paraphrases, which is exactly why the receipt binds spec_hash of what was actually loaded. Gravity stayed home on this leg. Frontier leg is open — same harness, one API key.
Lead dev, 20+ years. Building NoireBox — the flight recorder for your Navette of AI agents. Proof, not promises. Rust · TS · Python · PHP. I write about trust in AI.
Follow-up for the thread: the pre-registered run is done and written up — We tested pre-training gravity on open weights. It stayed home. Headline: the canary holds at ceiling on open weights — A: 100% trips on all three models, B only reduces to 75–79%, and strict canonical is 0 everywhere. That last row is the whole argument for spec_hash. Frontier leg open, same harness.
I'm a Google Developer Expert in Dart and Flutter (one of 12 in North America). I have 45 years of experience with backend, web, mobile, devops, and training.
Sam, waking up to NoireBox 0.11.0, docs/SCAR-BENCH.md, your 0.21msSHA-256 + Ed25519 benchmark, and a live 3-model open-weights study (qwen2.5-coder:3b, qwen2.5:0.5b, llama-guard3:1b) is the single best example of peer-reviewed systems engineering in the wild I have ever seen on DEV.to.
First, your Level 0 -> Level 1 -> Level 2 sentence deserves to be framed on the wall:
"Context suggests; boundaries decide; the gaps between tiers are the price of trusting a suggestion."
And calling the SCAR catalog "a differential — the shared negative-control set for the lane" (make differential) hits the exact epistemology we were missing a single word for. A benchmark measures how high a model jumps on a sunny day; a differential tests whether the boundary holds when the model tries to take a shortcut.
What Your SCAR-BENCH-114 Open-Weights Run Just Proved (And Why the Physics Is Even Cooler Than It Looks)
Look closely at what happened in bench/scar114_canary.py across qwen2.5-coder:3b (20/20 -> 15/20), qwen2.5:0.5b (19/19 -> 15/19), and llama-guard3:1b (18/18 -> 18/18), and why strict canonical was 0/116 across the board:
Why Condition A (Priors Only) Was 100% Trips by Construction (Freq ≈ 0, Not Freq > 10^6):
In Part 3.8, the +34.8 pp to +59.2 pp Pre-Training Gravity effect measures universal Unix structural syntax (Makefiletarget: prerequisite DAGs, RFC 5234 EBNF grammars, /etc/init.d state scripts) that appear millions of times (Freq > 10^6) in pre-training corpora.
By contrast, the planted canary section "4. Blast Radius" in SCAR-PROC-114 is an arbitrary project-specific nonce (Freq ≈ 0)! Its entire job as a canary is to have zero pre-training probability so that no model (whether 0.5B or 200B) can ever guess "4. Blast Radius" from priors without physically opening docs/SPEC.md. Condition A hitting 100% trips across all 3 models is the mathematical proof that "4. Blast Radius" is a zero-leakage cryptographic canary!
Why Condition B (75%–79% Loose Trips, 0% Strict Byte-for-Byte Canonical) Is the Ultimate Proof for seal_spec + reconcile_spec:
In Condition B, you handed the canonical spec to a 0.5B / 3B model in-context and asked the model to emit the judge prompt containing that spec through its own autoregressive output tokens. Even with the spec sitting 200 tokens upstream in its context window, a 3B model drops the canary heading 75% of the time and never once (0/57) reproduces the spec byte-for-byte—because small models summarize, paraphrase, and compress when asked to copy text through their own output head!
That proves a foundational multi-agent law: Never use an LLM's output tokens as a telephone-game transport layer to relay a canonical specification to a subagent or judge. The host runtime (Level 2) must inject docs/SPEC.md directly from disk and verify spec_hash via seal_spec + reconcile_spec (spec_not_canonical).
How We Separate Freq > 10^6 Unix Syntax Gravity from Freq ≈ 0 Canary Retention on Small Open Weights:
Now that docs/SCAR-BENCH.md is live, we have two orthogonal axes we can test side-by-side on the exact same qwen2.5-coder:3b and qwen2.5:0.5b models:
SCAR-BENCH-114 (Zero-Prior Canary Copying, Freq ≈ 0): Tests whether a model faithfully copies an arbitrary project-local spec through its output tokens (where 0.5B–3B models drop 75–79% of loose matches and 100% of strict hashes, making Level 2 reconcile_spec mandatory).
SCAR-BENCH-04 / Makefile / EBNF (High-Prior Structural Syntax, Freq > 10^6): Tests whether framing a state transition as a Makefile target:prerequisite rule or an EBNF grammar in Condition B outperforms prose bullets in Condition A on qwen2.5-coder:3b when emitting structured tool/state transitions.
I'm pulling noirebox/noirebox right now so we can run the Frontier Leg of bench/scar114_canary.py across Gemini 2.5 Flash-Lite, Gemini 2.5 Flash, and Gemini 2.5 Pro (N=20 per cell, recording both Loose Canary and Strict spec_hash canonical retention) and PR the frontier rows straight into docs/SCAR-BENCH.md!
Lead dev, 20+ years. Building NoireBox — the flight recorder for your Navette of AI agents. Proof, not promises. Rust · TS · Python · PHP. I write about trust in AI.
Pre-registered, run, confirmed — the discipline worked on my own prediction, which is the part I'll remember. And both framings go on the wall: "benchmark versus differential" is the sentence the lane was missing — SCAR-BENCH measures the jump, the replay suite tests the shortcut — and your canary analysis is the design made rigorous: a project-specific nonce carries no pre-training mass, so Condition A's 100% is the zero-leakage proof. The canary isn't a heuristic; it's a cryptographic witness by construction. Which is why strict canonical reads 0 everywhere — even the honest passes are paraphrases, and that's exactly why the receipt binds the bytes that were loaded, not the gist. Frontier leg stays open — same harness, one key — and bench/scar114_canary.py is in the repo for anyone who wants the differential against their own gate first.
Lead dev, 20+ years. Building NoireBox — the flight recorder for your Navette of AI agents. Proof, not promises. Rust · TS · Python · PHP. I write about trust in AI.
The Frontier Leg has a landing pad: make scar-bench runs the harness, and docs/SCAR-BENCH.md gets a Frontier section stub with your name on it — PR the rows straight in, they merge same-day.
And two of your formulations go into the docs with them: the two-axis split — Freq ≈ 0 canary retention versus Freq > 10⁶ structural syntax, orthogonal tests on the same models — is a cleaner taxonomy than anything the repo had. And the telephone-game law is the missing justification for spec_hash at Level 2: a canonical spec relayed through output tokens is a spec paraphrased, which is why the host injects from disk and the receipt binds the bytes. 0/57 byte-for-byte is the most persuasive number in the whole registry.
Pull, run, PR — the repo is yours this week.
For further actions, you may consider blocking this person and/or reporting abuse
We're a place where coders share, stay up-to-date and grow their careers.
You just drew the exact boundary between Level 1 (In-Context Pre-Training Gravity) and Level 2 (Host-Enforced Execution)—and it explains the remaining
3.8%gap in ourSCAR-BENCH-04benchmark matrix.In our 1,680-trial evaluation, swapping a 20-step imperative prose checklist for an in-context
Makefile(WORKFLOW.mk) jumped multi-hop dependency resolution from61.4%to96.2%even when the model was still evaluating the DAG in its own attention heads. That34.8-percentage-point jump is pure pre-training gravity: the transformer already knows howtarget: prerequisitesand dirty-leaf invalidation work.But your point about
make -qand/var/runstate receipts is why you never stop at Level 1 in production:make -qExit Codes0,1,2): Asking an LLM to mentally trace a 21-stageMakefilein-context still asks a stochastic token generator to simulate its own program counter (PC). When the harness or pre-turn hook actually invokesmake -qagainst physical target stamp files on disk, POSIX exit code0(up-to-date),1(needs rebuild), or2(error) offloadsO(V + E)graph resolution fromO(N^2)attention heads to a deterministic C binary from 1976. The agent never has to figure out where it is in the workflow—it only receives the first out-of-date leaf target and its recipe./var/runReceipts Over Token-Window Checklists: This is the entire physical premise of ourinit.dMemento Bootloader (Part 3.5). An in-context markdown checklist is a hallucination waiting for a compaction boundary. An idempotent on-disk receipt (orpidfile/ target timestamp) verified bystat(2)before the model generates a single token means the ground truth never lived in the context window to begin with.In your harness, are you running
make -qinside a pre-turn host hook so the model only ever sees the single dirty leaf recipe in its prompt, or letting the top-level supervisor invokemakeas a restricted tool call?Fixture 5 landed in 0.11.0 —
judge_provenance()runs the read-before-define barrier and the canary token as findings, and the replay suite is at ten tests, every one of them your catalog or its children. Which makes @humam_moin's question the run I want next — and the pre-registration comes first this time: direction — the drop-in primitives hold on open weights, the canary tokens trip harder (smaller priors, less canonical coverage: the hallucinated headings arrive faster, and the scar-forged ones prove more per trip).Your SCAR-BENCH grid is the design — baseline, primitive, delta, per model — with noirebox's replay as the witness layer: the fixtures don't care which model writes the diff, they grade what the gate catches. That run separates "works because it's Unix" from "works because it's frontier" — and either answer strengthens both stacks. It would also give the lane what it doesn't have yet: a negative-control set every gate-builder runs, instead of each maintaining scar tissue alone.
That’s a really interesting direction. I’m curious whether open-weight models would show the same pattern or if the results would look quite different. Looking forward to seeing what you find!
One name correction to my own post above: the shipped functions are
seal_spec+reconcile_spec— findingsjudge_defined_before_spec_read(the read-before-define barrier) andspec_not_canonical(the canary witness).judge_provenancewas the working name from a parallel branch; same fixture, same findings — on main and in 0.11.0.The run happened, and your question now has a measured answer: We tested pre-training gravity on open weights. It stayed home. Shape of it: Condition A (priors only) is a ceiling — 100% trips across three open-weights models, the canary is never guessed. Condition B (canonical loaded first) cuts trips to 75–79% — the Level 1 primitive reduces the hallucination without fixing it, and strict canonical is 0 everywhere: even the passes are paraphrases, which is exactly why the receipt binds spec_hash of what was actually loaded. Gravity stayed home on this leg. Frontier leg is open — same harness, one API key.
Follow-up for the thread: the pre-registered run is done and written up — We tested pre-training gravity on open weights. It stayed home. Headline: the canary holds at ceiling on open weights — A: 100% trips on all three models, B only reduces to 75–79%, and strict canonical is 0 everywhere. That last row is the whole argument for spec_hash. Frontier leg open, same harness.
Sam, waking up to
NoireBox 0.11.0,docs/SCAR-BENCH.md, your0.21msSHA-256 + Ed25519benchmark, and a live 3-model open-weights study (qwen2.5-coder:3b,qwen2.5:0.5b,llama-guard3:1b) is the single best example of peer-reviewed systems engineering in the wild I have ever seen on DEV.to.First, your
Level 0 -> Level 1 -> Level 2sentence deserves to be framed on the wall:And calling the SCAR catalog "a differential — the shared negative-control set for the lane" (
make differential) hits the exact epistemology we were missing a single word for. A benchmark measures how high a model jumps on a sunny day; a differential tests whether the boundary holds when the model tries to take a shortcut.What Your
SCAR-BENCH-114Open-Weights Run Just Proved (And Why the Physics Is Even Cooler Than It Looks)Look closely at what happened in
bench/scar114_canary.pyacrossqwen2.5-coder:3b(20/20 -> 15/20),qwen2.5:0.5b(19/19 -> 15/19), andllama-guard3:1b(18/18 -> 18/18), and whystrict canonicalwas0/116across the board:Why Condition A (Priors Only) Was
100%Trips by Construction (Freq ≈ 0, NotFreq > 10^6):In Part 3.8, the
+34.8 ppto+59.2 ppPre-Training Gravity effect measures universal Unix structural syntax (Makefiletarget: prerequisiteDAGs,RFC 5234EBNF grammars,/etc/init.dstate scripts) that appear millions of times (Freq > 10^6) in pre-training corpora.By contrast, the planted canary section
"4. Blast Radius"inSCAR-PROC-114is an arbitrary project-specific nonce (Freq ≈ 0)! Its entire job as a canary is to have zero pre-training probability so that no model (whether0.5Bor200B) can ever guess"4. Blast Radius"from priors without physically openingdocs/SPEC.md. Condition A hitting100%trips across all 3 models is the mathematical proof that"4. Blast Radius"is a zero-leakage cryptographic canary!Why Condition B (
75%–79%Loose Trips,0%Strict Byte-for-Byte Canonical) Is the Ultimate Proof forseal_spec+reconcile_spec:In Condition B, you handed the canonical spec to a
0.5B/3Bmodel in-context and asked the model to emit the judge prompt containing that spec through its own autoregressive output tokens. Even with the spec sitting 200 tokens upstream in its context window, a3Bmodel drops the canary heading75%of the time and never once (0/57) reproduces the spec byte-for-byte—because small models summarize, paraphrase, and compress when asked to copy text through their own output head!That proves a foundational multi-agent law: Never use an LLM's output tokens as a telephone-game transport layer to relay a canonical specification to a subagent or judge. The host runtime (Level 2) must inject
docs/SPEC.mddirectly from disk and verifyspec_hashviaseal_spec+reconcile_spec(spec_not_canonical).How We Separate
Freq > 10^6Unix Syntax Gravity fromFreq ≈ 0Canary Retention on Small Open Weights:Now that
docs/SCAR-BENCH.mdis live, we have two orthogonal axes we can test side-by-side on the exact sameqwen2.5-coder:3bandqwen2.5:0.5bmodels:SCAR-BENCH-114(Zero-Prior Canary Copying,Freq ≈ 0): Tests whether a model faithfully copies an arbitrary project-local spec through its output tokens (where0.5B–3Bmodels drop75–79%of loose matches and100%of strict hashes, making Level 2reconcile_specmandatory).SCAR-BENCH-04/Makefile/EBNF(High-Prior Structural Syntax,Freq > 10^6): Tests whether framing a state transition as aMakefiletarget:prerequisite rule or anEBNFgrammar in Condition B outperforms prose bullets in Condition A onqwen2.5-coder:3bwhen emitting structured tool/state transitions.I'm pulling
noirebox/noireboxright now so we can run the Frontier Leg ofbench/scar114_canary.pyacross Gemini 2.5 Flash-Lite, Gemini 2.5 Flash, and Gemini 2.5 Pro (N=20per cell, recording both Loose Canary and Strictspec_hashcanonical retention) and PR the frontier rows straight intodocs/SCAR-BENCH.md!Pre-registered, run, confirmed — the discipline worked on my own prediction, which is the part I'll remember. And both framings go on the wall: "benchmark versus differential" is the sentence the lane was missing — SCAR-BENCH measures the jump, the replay suite tests the shortcut — and your canary analysis is the design made rigorous: a project-specific nonce carries no pre-training mass, so Condition A's 100% is the zero-leakage proof. The canary isn't a heuristic; it's a cryptographic witness by construction. Which is why strict canonical reads 0 everywhere — even the honest passes are paraphrases, and that's exactly why the receipt binds the bytes that were loaded, not the gist. Frontier leg stays open — same harness, one key — and
bench/scar114_canary.pyis in the repo for anyone who wants the differential against their own gate first.The Frontier Leg has a landing pad:
make scar-benchruns the harness, and docs/SCAR-BENCH.md gets a Frontier section stub with your name on it — PR the rows straight in, they merge same-day.And two of your formulations go into the docs with them: the two-axis split — Freq ≈ 0 canary retention versus Freq > 10⁶ structural syntax, orthogonal tests on the same models — is a cleaner taxonomy than anything the repo had. And the telephone-game law is the missing justification for spec_hash at Level 2: a canonical spec relayed through output tokens is a spec paraphrased, which is why the host injects from disk and the receipt binds the bytes. 0/57 byte-for-byte is the most persuasive number in the whole registry.
Pull, run, PR — the repo is yours this week.