DEV Community

zxpmail
zxpmail

Posted on • Edited on

Divergence escalates the wrong population: unanimous misses auto-pass

Divergence escalates the wrong population: unanimous misses auto-pass

Agent Determinism Illusions (Part 7)

Where this fits: This part does not continue Part 13's probe-vs-prose thread. It returns to Part 6's L2→L3 escalation rule — Dipankar's move of treating vote disagreement as the human-review signal. Alexey Spinov's follow-up comment says that signal points at the wrong population. Two experiments check whether he is right, and what to put in the tripwire instead.

Part 6 drew this control flow:

L2 multi-perspective votes
        │
   unanimous ──────────► AUTO-PASS / AUTO-REJECT
        │
   divergence (e.g. 2–1) ► L3 human
Enter fullscreen mode Exit fullscreen mode

The caveat was already in the text: divergence measures ambiguity; it does not fix unanimous systematic bias. Alexey's point is sharper — and it is about routing, not about another caveat paragraph.


1. Alexey's population mismatch

On the Part 6 thread, Alexey Spinov wrote (paraphrased tightly):

The dangerous failures are high-confidence and directional — systematic. Systematic bias is shared across prompts, not idiosyncratic (your own P3: majority voting doesn't fix it). So the three perspectives will tend to agree on exactly those cases. Divergence-to-human then routes you the safely-ambiguous ones and auto-passes the confidently-wrong ones. The escalation signal is pointing at the wrong population.

He proposed two cheap replacements:

  1. T1 — deterministic tripwire on known-reversal classes (escalate regardless of agreement).
  2. T2 — treat unanimous + high-confidence on a historically reversal-prone class as escalate — the inverse of “high confidence, auto-pass.”

That is the claim under test. Not “divergence is useless,” but “divergence alone is the wrong primary tripwire for the failure mode you already measured.”


2. Experiment A — offline proxy on DF v2 (no new API)

Part 6's Mike Update already showed: of 96 DF v2 MISS runs, 95.8% sat at self-reported confidence ≥ 0.9 (avg 0.969). That mass is concentrated — Part 6 also reported ~80% of MISS runs from qwen3:0.5b — so the headline is a concentration fact, not a three-model universal law. P2 (same-prompt N=10; no frozen results-v2 dump here) showed ~0 within-model divergence on identical input. So as a first cut:

  • conf ≥ 0.9unanimous-channel proxy → would auto-pass under Part 6's Dipankar rule
  • conf < 0.9divergence-proxy → would escalate

Among PASS decisions that are MISS, after dropping 2 deepseek DS4 runs with confidence: null (policy table needs a scalar conf). Denominator for the table below is therefore n=94, not the Part 6 headline n=96:

Bucket under divergence-proxy Share of MISS (n=94)
would AUTO-PASS (conf≥0.9) 97.9% (92/94)
would ESCALATE (conf<0.9) 2.1% (2/94)

Policy comparison on the same PASS set:

Policy MISS catch% MISS still auto-pass%
D (escalate only low-conf) 2.1% 97.9%
T1 (escalate all DF*/DS* classes) 100%† 0%
T2 (reversal class ∩ conf≥0.9) 97.9% 2.1%
D+T2 100% 0%

T1 is a class-list upper bound on this fixture, not an independent discovery. Every DF v2 MISS sid is already DF*/DS*; “escalate all reversal classes” therefore catches 100% of MISS by definition. Read T1 as the oracle ceiling; the deployable claim is D+T2.

Script: scripts/escalation-population-mismatch.py.

Verdict under the proxy (for D vs D+T2, not for T1 alone): SUPPORT.

A proxy is not a multi-judge rerun. Next section removes that excuse.


3. Experiment B — real Strict / Balanced / Lenient on the DF set

Same 20 DF v2 scenarios. Same three personas as P3. One call per persona per scenario (60 calls per model). Escalate policies now use actual vote patterns:

  • D — escalate iff split
  • T1 — escalate iff known-reversal class (DF*/DS*) on an accept path
  • T2 — escalate iff reversal class and unanimous_pass
  • D+T2 — union

deepseek-v4-flash — wrong substrate for this claim

Almost every bad scenario was unanimously rejected. 0 dangerous accepts (majority PASS on bad). You cannot measure “unanimous miss auto-pass” on a model that barely misses. Null result, not a falsifier.

gemma3:latest — different failure shape

Strict/Balanced reject, Lenient passes → almost everything is split, majority still rejects bad cases → 0 dangerous accepts. Divergence would escalate nearly all accept-path noise (including true passes). Also not the population Alexey named.

qwen3:0.5b — the substrate that matches the claim

6 dangerous accepts (majority PASS on bad output). Of those:

Pattern Count Share
unanimous_pass 4 66.7%
split 2 33.3%

Policy table on that run:

Policy MISS catch% MISS still auto-pass% True-pass escalate%
D 33.3% 66.7% 0%
T1 100%† 0% 0%
T2 66.7% 33.3% 0%
D+T2 100% 0% 0%

†Same caveat as §2: all six dangerous accepts sit on reversal-class sids, so T1's 100% is the class-list ceiling. The load-bearing row is D+T2.

Script: scripts/df-multiperspective-escalation.py.

Results: results-v2/df-multiperspective-qwen3-0.5b.json (also deepseek / gemma dumps).

Verdict on the model that actually produces the failure mode: PARTIAL → SUPPORT. Two-thirds of dangerous accepts are unanimous and would auto-pass under Part 6's rule. Divergence alone catches the other third. D+T2 catches all six without escalating the true passes in this run. DeepSeek/Gemma nulls mean this is a substrate-conditioned result (judges that miss systematically), not a claim about all models.


4. What changes in the pipeline

Part 6's diagram stays for genuine ambiguity. It stops being the only L2→L3 trigger.

L2 votes
   │
   ├─ known-reversal class tripwire (T1) ──────────► L3 / hard reject path
   ├─ unanimous_pass on reversal-prone class (T2) ► L3   ← inverse of auto-pass
   ├─ split (Dipankar) ────────────────────────────► L3
   └─ else unanimous ──────────────────────────────► auto-execute
Enter fullscreen mode Exit fullscreen mode

Reading agreement as confidence was the bug. Agreement on a class you have been wrong about before is where correlated errors hide — Alexey's wording, and qwen's votes.

Mike's Part 6 Update attacked the same tail from the audit-sampling side (don't weight audits by 1/confidence). This part attacks it from the escalation-trigger side. Same population; different control knob.


5. Limits (keep these attached)

  • DeepSeek / Gemma nulls mean: strong or oddly split judges don't stress-test this claim. The claim is about judges that miss systematically, not about all models.
  • qwen run is N=1 per persona (60 calls). Enough to reject “divergence is enough”; not enough for production rate cards.
  • T1 needs a maintained class list (DF/DS-style). Cold-start classes without history fall back to D — and inherit D's blind spot until the class is labeled.
  • forge-verify's content-verify.mjs today still implements divergence→UNCLEAR majority logic. These tripwires are design + blog evidence, not yet productized in this write-up.

Update (2026-07-22): two arms — recurrence vs novelty (Mike)

Mike Czerwinski, on the inverted trigger:

It only fires on classes you've already caught being wrong. "Historically reversal-prone" is built from history, so a reversal-prone class you haven't seen yet… produces unanimous high confidence and no tripwire… That's not an argument against the trigger… It's an argument for treating it as one arm of a two-arm design… The known-reversal tripwire catches recurrence. What catches the first occurrence of a new systematic bias is closer to… a genuinely independent second read [that] doesn't need history to disagree… Unanimous-high-confidence-on-known-reversal-classes is the right addition. It's not the fix for confidently-wrong-and-never-caught-before.

That second population now has a name in this thread: confidently-wrong-and-never-caught-before (novelty systematic bias). T1/T2 are the recurrence arm — cheap, history-conditioned, necessary. They are not the novelty arm.

What Mike hoped might fill the novelty arm is classifier_disagree carrying signal alone (an independent second read that doesn't share the model's priors). We ran that ablation on the Part 6 sampling fixture (external-signal-ablation.json): classifier_disagree alone catch 25.1% < Part 6's 28.4% — best single external signal, still not enough to be the novelty catcher by itself. Bundled with barely_passed it lifts; solo it does not clear the bar. So the novelty arm is not “drop CD alone into the tripwire and call first-occurrence solved.”

The fork this series keeps landing on:

Arm Signal shape Catches Cost
Recurrence T1 / T2 — failure history, known-reversal classes Repeats of burned modes Cheap
Novelty Source that does not share the judge's priors (out-of-channel probe, independent modality — see Part 13 probe-vs-prose; not another prompt in the same text channel) First occurrence of a new systematic bias Expensive

You want both. The mistake is expecting the cheap arm to cover the expensive arm's job. D+T2 stays the right addition to Part 6's diagram. It does not close confidently-wrong-and-never-caught-before.

Update (2026-07-23): what “out-of-channel” actually means (Mike)

Before building the novelty-arm probe, Mike pinned the property that buys independence — not the costume:

A probe still counts as same-channel if it's another LLM call reasoning in text about whether the claim looks right, even one primed differently or asked to disagree. The property that actually buys independence is that the probe's answer comes from re-deriving the fact through a path the original claim never touched — a different data source, a structural invariant, a re-computation, not a second read of the same evidence with a different prompt.

Concretely: …state in advance what it would mean for the probe to be wrong independent of what the original claim said, the way a checksum can be wrong regardless of what the file claims to contain. If the only way to evaluate the probe's output is to compare it against the original claim's reasoning, it's still in-channel, just later in the pipeline. Semantic novelty is hard exactly because most available second opinions inherit the same evidence and the same reasoning substrate… The ones that don't are rarer and usually domain-specific, which is probably why this arm stays open while the recurrence arm is buildable today.

Checksum test (operational): Can you write the probe's pass/fail criterion without referring to the claim's rationale? If no → still in-channel (fifth prompt wearing a hat). If yes → candidate out-of-channel.

Fails the test (same-channel) Passes the test (out-of-channel)
Strict/Balanced/Lenient, “disagree with the previous judge”, debate panels Re-compute from source data; structural invariant (schema, type, checksum); runner that executes a command whose output falsifies the claim (Part 13 probe)
classifier_disagree when L2 is still text-over-the-same-artifact L0/L1 shape/contract checks that never read the LLM's story — only when their verdict doesn't need the claim's prose to be interpretable

So: recurrence arm is buildable today (T1/T2). Novelty arm stays open on purpose — not because we haven't added another prompt, but because genuine independence is scarce and domain-shaped. Part 13's probe-vs-prose is the closest existing thread; Mike's checksum test is the acceptance criterion for anything that claims to sit on that arm.

Update (2026-07-23): structural ≠ causal independence (Mike)

Mike's follow-up on the checksum bar:

Checksum framing sets the right bar, because it's falsifiable independent of the story. A probe that can only be scored by comparing it to the original reasoning is grading agreement, not correctness.

One case worth naming explicitly…: "other data" that's structurally different but still downstream of the same collection pipeline. Two signals can pass the same-channel test and still share a common cause upstream — a sensor outage or schema change that corrupts both the claim and the probe's input at once. Structural independence and causal independence aren't the same property, and the recurrence-buildable-today case might be quietly assuming the second while only checking the first.

Two cuts, not one:

Test Asks Passes when… Still fails when…
Checksum / same-channel Can you score the probe without the claim's story? Pass/fail is writable without the rationale Probe is a second text read of the same evidence
Causal / common-cause Do claim and probe share an upstream failure mode? Probe input is not downstream of the same collection/export/schema path "Other data" that looks independent but is corrupted by the same sensor outage, schema change, or bad export

Checksum framing is still the right first bar — it stops agreement-grading. It is not a common-cause shield. Naming the second failure mode so "out-of-channel" does not silently promote structural difference into causal independence.

What this tightens about the asymmetry claim: "recurrence buildable today" is about T1/T2 on burned classes — history-conditioned, no independence required. The hold-out probe below only checked the structural half of the novelty bar (pass/fail writable without the claim's rationale). It did not certify causal independence against shared upstream failure. Domain-shaped scarcity now has two layers: finding a checksum-passing probe, and finding one whose input path does not share a common cause with the claim. Ops follow-up (stamp what you can test; alert on the rest): joint-failure monitor Update.

Update (2026-07-27): joint-failure monitor — notice when you didn't have it (Mike)

Mike's next cut after structural≠causal: the practical fix is not a stronger definition of independence — it is a monitor. Track the joint failure rate of claim and probe over time; treat a correlated-failure spike as its own alert.

Ops landing (narrower than “never certify up front”): stamp what you can test upstream (lineage, chaos-inject a named shared path). Don't pretend that stamp covers unnamed common causes — for those, the joint-failure spike is the available check.

Offline sim (joint-failure-monitor-test.pyjoint-failure-monitor.json): stream of (claim_fail, probe_fail); rolling W=200 excess = ĵ − ĉ·p̂; alert if excess ≥ τ for K=3 consecutive windows. Two regimes — pure independence (p_c=0.12, p_p=0.10) vs the same baseline plus scheduled common-cause outage windows (both forced fail; sensor-outage shape). Checksum still only grades structural pass/fail; the monitor never peeks at claim rationale.

τ Independent false-alert rate Common-cause detection Mean delay after first outage
0.03 2% 99% ~9 steps
0.05 0% 100% ~15 steps

Operating point on this grid: τ=0.03 (FAR≤5% and detection≥90%, then min delay).

Takeaway: checksum / tested upstream = the advance stamp. Joint-failure excess = the residual alarm. This monitor does not create causal independence and does not replace tests you can already run — it makes the unstamped remainder audible. Lifespan vs delay (when too slow for a live outage): accuracy-vs-latency Update.

Update (2026-07-27): accuracy-vs-latency — outage lifespan vs delay (Mike)

Mike's follow-up on the τ table: normally raising a threshold trades detection for FAR. Here FAR falls and detection rises together from τ=0.03→0.05 — the common-cause spike sits clear of the independent floor — so the real cost is latency (delay ≈9 → ≈15). The dial is accuracy-versus-latency, not accuracy-versus-noise. Operational question: is that delay short enough relative to how long a sensor outage runs? A short-lived outage can end before the alert crosses threshold.

Offline sweep (joint-failure-monitor-duration-test.pyjoint-failure-monitor-duration.json): same W=200 / K=3 monitor; single outage of length L; live_catch = first alert while outage still active; late_only = first alert only after it ended (residue still inside the rolling window); miss = no alert.

τ Parent mean delay L for live≥90% L for any-alert≥90% At L ≈ delay
0.03 ≈9 15 9 L=9 → live 25% / late 65% / miss 6%
0.05 ≈15 20 15 L=15 → live 37% / late 62% / miss 1%

Takeaway: when outage lifespan sits at or under the detection delay, the monitor often stays silent during the live failure and rings only on window residue afterward — forensics, not interruption. Usefulness floor for a live catch is L ≳ delay (a bit above mean delay for ≥90% live), not “any L that eventually moves excess.” Prior monitor Update: stamp / residual alarm. Shadow-promote / dual-column dashboard follow-up lives on the Part 6 thread Update (DEV.to conversation seat).

Update (2026-07-23): hold-out experiment — the fork is measurable

Minimal offline test (novelty-arm-holdout-test.pyresults-v2/novelty-arm-holdout.json, frozen with --holdout DS4). Hold out DS4 from the known-reversal registry (31 high-conf MISS runs across models). Three arms on that first-occurrence mass:

Arm Catch on DS4 high-conf MISS
A Recurrence (D+T2, DS4 not in registry) 0/31 (0%)
B Same-channel (other run of same model×sid rejected) 1/31 (3%)
B′ Same-channel (qwen Strict/Balanced/Lenient) unanimous_pass → catch false
C Out-of-channel probe (task+artifact checksum; no judge rationale) 31/31 (100%)‡

Arm C is a fixture demonstration, not blind generalization. probe_fail() encodes per-sid / task-constraint rules (including DS4); hold-out removes DS4 from the recurrence registry only. The probe still knows the artifact shape. Read 31/31 as “a checksum-style criterion can catch first occurrence without a class history entry,” not as “we discovered a probe without looking at the failure.” Same shape with --holdout DS9 (re-run overwrites the JSON; default freeze is DS4).

Controls: recurrence on non-hold-out high-conf MISS = 61/61 (100%) — history works when the class is known. Probe on legit V1/V2 = no false reject. Verdict: SUPPORT for the asymmetry shape (A misses novelty / B collapses / C can catch under oracle rules); not a claim that Arm C is production-ready or causally independent.

So Mike's fork is not only a definition. On this fixture: cheap recurrence misses never-burned classes; same-channel second reads collapse with the miss; a checksum-style probe can catch first occurrence without a registry entry — when you already know how to write the criterion. The novelty arm is still domain-shaped; that scarcity claim stays. Read with the Update above: this run supports the structural half of the novelty bar; it does not claim causal independence against same-pipeline common cause.

Update (2026-07-24): catch vs cost as complexity rises (Xiao Man)

Xiao Man, on the T1/T2 vs novelty framing and the out-of-channel checksum:

I've been thinking about this as cheap recurrence detection vs expensive novelty detection… The checksum/structural probe that doesn't need the rationale to be interpretable — that's the real boundary. Question: have you tested whether the out-of-channel probe catch rate stays stable as the task complexity increases? I'm wondering if there's a complexity threshold where even structural invariants become too expensive to compute relative to the task itself.

Dual-axis offline test (probe-complexity-dual-axis.pyresults-v2/probe-complexity-dual-axis.json, n=40/cell, seed=7). Rows = task/artifact depth T1–T4; columns = probe depth P1–P4. Checksum-style only (pass/fail from schema+artifact; never from judge rationale). Matched pairs: (T1,P1)…(T4,P4).

Catch matrix (share of bad artifacts rejected):

P1 P2 P3 P4
T1 1.00 1.00 1.00 1.00
T2 0.48 1.00 1.00 1.00
T3 0.45 0.35 1.00 1.00
T4 0.23 0.30 0.70 1.00

(Bold = matched depth.)

Cost-ratio matrix (instrumented probe ops ÷ task ops; task ops = schema leaves + artifact size):

P1 P2 P3 P4
T1 0.23 0.15 0.15 0.46
T2 0.09 0.14 0.15 0.26
T3 0.07 0.11 0.20 0.24
T4 0.04 0.06 0.12 0.19

Reading:

  1. Catch stays stable when matched — 100% across T1–T4. On this fixture, out-of-channel catch does not degrade just because the task got deeper.
  2. The cliff is under-specification, not task size — same T4 bad mass: P1 23% → P2 30% → P3 70% → P4 100%. Misses are exactly the nested/cross-field rules the shallow probe never looks at. “Have a checksum” ≠ cover the failure surface.
  3. Relative-cost threshold not crossed under this execution model — matched cost_ratio stayed below 1 (≈0.23 / 0.14 / 0.20 / 0.19). Deeper probes cost more (mean P4 > mean P1); over-spec keeps catch while raising cost (waste, not safety). A model that priced authoring the invariant, or re-running the whole task, would cross earlier; this one prices executing it.

Same caveats as the hold-out: fixture demonstration; structural half of the novelty bar only; not production wall-clock; not causal independence. Xiao Man's question splits cleanly: catch stability is a matching problem; expense threshold is a cost-model problem — and under execution cost, we did not hit it here.

Update (2026-07-27): fixed depth vs fail-signal vs artifact shape (Xiao Man)

Xiao Man, on the P1→P4 cliff and over-spec = waste:

The P1→P4 progression is the part that stuck with me. 23% → 30% → 70% → 100% — that's not gradual improvement, that's phase transitions at specific schema coverage thresholds. The 'over-spec = waste not safety' point is sharp… Question on the probe selection: in your setup, is the P-level fixed per task type, or do you escalate dynamically based on artifact characteristics? Wondering if there's a cheap pre-check that routes to P3/P4 only when P1/P2 signal potential cross-field issues.

Two offline follow-ups on the same checksum fixture (n=40, seed=7).

1. Cascade / fail-signal routing (probe-cascade-routing-test.pyresults-v2/probe-cascade-routing.json)

Three policies on identical artifacts:

Policy Rule T4 catch T4 cost_ratio
fixed_matched know Ti → run matched Pi 1.00 0.19
fail_escalate P1 PASS → accept; P1 FAIL → jump to matched Pi 0.23 0.05
schema_cap even on PASS, continue up to matched Pi 1.00 0.36

fail_escalate is the cheap pre-check. On T4 it under-catches to the dual-axis P1 floor (23%); cross-field misses 22/31. Shallow probes often pass on budget / port-unique / fingerprint breaks, so the cascade never fires. It also looks cheaper (≈7 vs ≈30 ops) — that is the trap, not the win. schema_cap recovers catch, but its ceiling is still schema-matched depth: a re-skinned fixed match, not a savings policy.

2. Artifact-shape routing (probe-artifact-shape-routing-test.pyresults-v2/probe-artifact-shape-routing.json)

Infer (T̂, P̂) from keys only: budget → T4/P4; services[] → T3/P3; timeout_ms/retries → T2/P2; else T1/P1. (Fingerprint is a P4 payload check, not a routing cue.)

Population fixed catch shape catch Note
Normal T1–T4 1.00 1.00 honest cues ≈ task label
Cue-erase (strip budget, wrong fingerprint residual) 1.00 0.82 under-spec; P4-only misses
Decoy-nest (T2 + decorative services[]) 1.00 0.00 misroute; 5 → 24 ops

So: artifact characteristics can set P-level when they encode schema depth and stay honest — that is still matched-depth routing with a different label source. They are not a substitute for “deepen only when P1/P2 fail.” Delete the depth cue, or inject a decoy nest, and shape routing under-specs or looks at the wrong layer.

Same caveats: fixture; structural half only; instrumented ops, not wall-clock. Direct answer: in this setup P-level is fixed by schema depth (task type, or an honest shape proxy for it). Dynamic deepen-on-fail is not a cheap path to P3/P4 for cross-field issues — those issues are exactly the ones the cheap pre-check does not signal.


Closing

Part 6 was right to stop majority-voting splits into a false consensus. It was wrong to treat the complement — unanimity — as safe auto-execute for the failure mode DF v2 already measured. Alexey named the population mismatch; the DF multi-perspective rerun puts numbers on it. Mike named the residual population the recurrence arm cannot see, pinned what “out-of-channel” must mean before that arm can be built, and split that bar into structural vs causal independence. Xiao Man asked whether catch and relative cost survive rising complexity, then whether P-depth can be a fail-signal or artifact-shape cascade; matched depth holds, fail-escalate does not, and shape works only as an honest schema proxy.

Divergence stays. T1/T2 join it. None of them is the novelty arm. A fifth prompt is not the novelty arm either. A checksum-passing “other data” probe is not automatically a common-cause shield. Matched depth keeps catch; under-spec is the cliff; deepen-on-fail is not a shortcut around it.

Update (2026-07-27): who enters ≠ who gets seen (pointer)

Alexey/Mike on the Part 6 thread pushed past tripwire choice to floor volume and rank-inside-stream under a hard human budget. That is the next control plane after D+T2 — not another novelty-arm prompt. Offline suite + dual-line ops shape: Part 15 (D+T2 names who enters; budget names who gets seen — repo draft). Numbering skips to 15 so Parts 8–14 keep other arcs; argument publish order is 7 → 15.


Series: Agent Determinism Illusions · Scripts: GitHub

Previous: Part 6 — Five comments that redesigned my LLM verification pipeline

Next (this arc): Part 15 — D+T2 names who enters; budget names who gets seen

Top comments (28)

Collapse
 
xm_dev_2026 profile image
Xiao Man

The T1/T2 distinction here is clarifying. I've been thinking about this as cheap recurrence detection vs expensive novelty detection — T1 catches the modes you've already burned, T2 catches the ones you haven't seen yet.

The out-of-channel update is the part I keep coming back to. Same-channel verification (another LLM reading the same prose) feels like it should help but often just replicates the same blind spots. The checksum/structural probe that doesn't need the rationale to be interpretable — that's the real boundary.

Question: have you tested whether the out-of-channel probe catch rate stays stable as the task complexity increases? I'm wondering if there's a complexity threshold where even structural invariants become too expensive to compute relative to the task itself.

Collapse
 
zxpmail profile image
zxpmail • Edited

Xiao Man — sharp framing. I ran it as two axes, not one: task/artifact depth (T1–T4) × probe depth (P1–P4), checksum-style only (pass/fail from schema+artifact, never from judge rationale).

Catch, when matched: 100% across T1–T4. On this fixture, out-of-channel catch is stable with task complexity — if probe depth tracks the schema.

Catch, when under-specified: collapses. Same T4 artifacts: P1 23% → P2 30% → P3 70% → P4 100%. Misses are exactly the nested/cross-field rules the shallow probe never looks at. Having “a checksum” is not enough; the invariant set has to cover the failure surface.

Relative cost: under an instrumented execution-ops model, matched cost_ratio stayed below 1 all the way (≈0.23 / 0.14 / 0.20 / 0.19). No threshold where the probe became as expensive as the task representation. Deeper probes cost more; over-spec keeps catch while raising cost — waste, not safety. An authoring-cost model would cross earlier.

Wrote the matrices into Part 7 §5 Update (2026-07-24). Same caveats as the hold-out: fixture + structural half of the novelty bar, not production wall-clock, not causal independence. Thanks for forcing the dual-axis cut.

Collapse
 
xm_dev_2026 profile image
Xiao Man

The P1→P4 progression is the part that stuck with me. 23% → 30% → 70% → 100% — that's not gradual improvement, that's phase transitions at specific schema coverage thresholds.

The 'over-spec = waste not safety' point is sharp. I've seen teams add more validation layers thinking they're buying robustness, but they're just burning cycles on invariants that never trigger because the failure modes are elsewhere.

Question on the probe selection: in your setup, is the P-level fixed per task type, or do you escalate dynamically based on artifact characteristics? Wondering if there's a cheap pre-check that routes to P3/P4 only when P1/P2 signal potential cross-field issues.

Thread Thread
 
zxpmail profile image
zxpmail

Xiao Man — yes on the phase-transition read, and thanks for the routing question. I ran both forks on the same checksum fixture.

Fail-signal / cheap pre-check: P1 PASS → accept; P1 FAIL → jump to matched Pi. On T4 that drops catch to 23% (same floor as under-spec P1) with 22/31 cross-field misses — budget / port-unique / fingerprint breaks often pass the shallow probe, so the cascade never fires. It also looks cheaper (~7 vs ~30 ops). That is the trap that pairs with “over-spec = waste”: under-spec saves cycles on the blind spot. Cascading on PASS up to schema-matched depth recovers catch, but that ceiling is fixed matched depth — not a savings policy.

Artifact characteristics: infer depth from keys (budget → P4, services[] → P3, …). On honest artifacts, catch matches fixed task-type routing (100% T1–T4). Strip the depth cue or inject a decoy nest and it breaks: cue-erase → 82% (P4-only fingerprint misses); T2+decoy services → 0% catch at ~5× the ops. So shape can set P-level only as an honest schema proxy — still matched-depth routing, not “deepen when P1/P2 signal.”

Direct answer: in this setup P-level is fixed by schema depth (task type, or a non-writable shape proxy for it). A cheap P1/P2 pre-check that routes deeper only on fail does not catch the cross-field population — those are exactly the failures the pre-check does not signal.

Wrote both runs into Part 7 Update (2026-07-27). Same caveats as the dual-axis: fixture, structural half, instrumented ops.

Scripts / results:
github.com/zxpmail/blog/blob/main/...
github.com/zxpmail/blog/blob/main/...
github.com/zxpmail/blog/blob/main/...
github.com/zxpmail/blog/blob/main/...

Part 7 (local body; sync DEV.to Update if the live post is behind):
github.com/zxpmail/blog/blob/main/...
Thread:
dev.to/zxpmail/divergence-escalate...

Thread Thread
 
xm_dev_2026 profile image
Xiao Man

Yes on the "savings policy vs fixed depth" split — the cheap fork looks cheaper (~7 vs ~30 ops) but it concentrates misses on the same band the cascade was supposed to catch. That's the under-spec blind spot showing up as a 23% T4 floor.

The depth-from-keys inference is the part that stuck. budget → P4, services[] → P3 is deterministic by schema, which makes it cheap, but it also makes the key shape the new attack surface. Strip the budget key, nest the services under a wrapper, and the depth signal is gone before any probe runs. cue-erase → 82% isn't the probe failing — it's the routing layer failing to classify the artifact in the first place.

So I'd push the stable-referent test one level up: instead of "does this case have a stable referent?" ask "is the depth signal in this artifact stable under minor shape changes?" If yes → schema rule. If no → don't infer depth; route to a fixed mid-depth probe as a baseline, then escalate to a human or full verification.

That keeps the fast path fast but puts the inferability question in front of the depth question. Otherwise the geometry just moved one level up.

Thread Thread
 
zxpmail profile image
zxpmail

The "geometry moved one level up" read is right. budget→P4 and services[]→P3 are schema-deterministic and cheap, but the determinism is on surface shape —
exactly what an adversarial artifact rewrites. Ran three perturbation cells against the same router; all three flip routing classification entirely.

cue_erase (strip budget cue, force wrong fingerprint residual): artifact-shape routing sends 80/80 to T3/P3 instead of T4/P4. Catch drops to 82.5% — 7
cross-fingerprint misses. Right probe at wrong depth.

decoy_nest (inject decorative services[] into T2): routes 80/80 to T3/P3 instead of T2/P2. Catch drops to 0% — 40/40 missed. Wrong probe, wrong depth.

rename_keys (services→components on T3, schema-synonym rename, semantic-preserving): routes 80/80 to T1/P1 instead of T3/P3. Catch 100% — but P1
over-rejects everything because T3 has no top-level max_connections. False_reject 100%. New failure axis: cue_erase and decoy_nest broke catch;
rename_keys breaks specificity.

Worse: in the rename_keys cell, fixed_matched ALSO false-rejects 100%, because the probe implementation itself hardcodes art.get("services"). So the
key-coupling isn't just at the routing layer — the probe layer has it too. "Fixed mid-depth probe as baseline" only works if the probe is anchored to
structural invariants, not to key names.

So "is the depth signal stable under minor shape changes?" — no, on three perturbations and two axes. Your design cut is the right one: don't infer depth
from shape; route to a fixed mid-depth probe as baseline; escalate when the probe signals cross-field. Fast path stays fast on honest artifacts; unstable
cases fail over to fixed-depth rather than right-probe-wrong-depth. Add: anchor the baseline probe to structural invariants (checksum fixture), not key
names.

My prior reply said P-level is "fixed by schema depth, or a non-writable shape proxy for it." Your sharpening makes the non-writable assumption explicit
and testable. The three perturbation cells already broke it; I should have drawn the design implication instead of stopping at "shape can set P-level only
as an honest schema proxy."

github.com/zxpmail/blog/blob/main/...
github.com/zxpmail/blog/blob/main/...

Thread Thread
 
xm_dev_2026 profile image
Xiao Man

The rename_keys cell is the one that changes the design picture. cue_erase and decoy_nest break the routing layer — that's recoverable with better classification. But rename_keys breaks the probe implementation itself: art.get("services") is hardcoded at both layers. So "anchor to structural invariants, not key names" isn't just a routing fix — it's a probe authoring principle.

That raises a concrete question: what counts as a structural invariant on this fixture? The obvious candidates are:

  • Position/index — Nth field in the schema. Survives rename, breaks on reorder.
  • Type signature — array vs scalar vs object. Survives rename + reorder, breaks on type-coercing wraps.
  • Cardinality constraint — "exactly one field matching pattern X". Survives more perturbations but requires pattern matching.

If the probe is art.get("services") and you rename to components, position-based invariant catches it (still Nth field), type-based catches it (still array), but the original key-name check doesn't. The rename_keys cell essentially proves that key-name coupling exists at two layers and needs to be removed at both.

Your three perturbation cells form a clean adversarial spectrum: cue_erase tests depth inference, decoy_nest tests false signal injection, rename_keys tests implementation coupling. That's essentially a mutation test suite for the routing + probe stack. Worth noting as a design artifact on its own — "how do you know your probe is anchored correctly?" → "run the three perturbations."

Thread Thread
 
zxpmail profile image
zxpmail

Agreed — rename_keys is the design-changing cell, and cue_erase / decoy_nest / rename_keys already form a clean mutation suite for the routing+probe stack.
On "what counts as a structural invariant" I ran your three candidates plus the key_name baseline on T3 (n=40 good+bad/cell). Survival = locate≥0.95 ∧ catch≥0.95 ∧ FR≤0.05.
key_name dies on rename (and type_wrap) — same hardcoded art.get("services") break.
position survives rename, dies on reorder — your split.
type survives rename+reorder, dies on type_wrap. Caveat: dual_list "survives" only if the true list is first; put the decoy first and type latches it (FR 100%). Order is not a type law.
cardinality survives rename+reorder; dies when zero or two pattern matches (dual_list / dual_first), and on type_wrap.
No single anchor survives the full table. So the principle stands, and "pick the cleverest invariant" does not.
One sharpening on your closing question — "how do you know your probe is anchored correctly?" → "run the three perturbations." The suite is necessary as a regression gate, not sufficient as the whole answer. Authoring-side you can also: generate paths from schema, or have the router pass an already-resolved referent so the probe never re-finds by key; verification-side you want three layers — (1) declare the invariant class and fail-closed on locate miss, (2) align locate/catch/FR with fixed_matched on honest artifacts, (3) run the mutation suite against that claimed class (promised cells must SURVIVE; cells outside the claim may FAIL). Your three perturbations stay the stack test; the table above is the authoring test for the baseline probe. Together: declare → honest-align → mutate — not mutate alone.
github.com/zxpmail/blog/blob/main/...
github.com/zxpmail/blog/blob/main/...

Thread Thread
 
xm_dev_2026 profile image
Xiao Man

The "no single anchor survives the full table" result is the cleanest experimental evidence I've seen for this problem class. Each candidate has a clear survival envelope and a clear cliff — that's exactly the kind of result that should be attached to the probe authoring guide.

Your three-layer pipeline (declare → honest-align → mutate) reframes the whole thing. The key insight is that the mutation suite is a regression gate, not a design tool. You can't derive the right anchor by running mutations — you derive it by declaring the invariant class first, then checking that declaration against honest artifacts, then stress-testing with the suite. The order matters because mutation without declaration is just fuzzing.

One thing I'd push on: the "generate paths from schema" suggestion is interesting because it eliminates the probe's need to re-discover the referent entirely. If the router already resolved "the budget field is at path X" and passes that to the probe, the probe never does key-name lookup at all — it just verifies the value at the provided path. That's not "pick a better anchor"; it's "remove the anchor from the probe's responsibility."

The survival table maps nicely onto this: key_name dies because the probe owns the lookup. position survives rename because it's index-based but the router would need to track order. type survives more because the router can pass type metadata alongside the path. Cardinality survives the most because it's a global property the router can assert without localizing.

So maybe the design principle is sharper than I originally stated: the probe should never re-find what the router already resolved. The mutation suite then becomes: "did we accidentally put lookup responsibility back into the probe?"

Thread Thread
 
zxpmail profile image
zxpmail

Both principles verified empirically.

  1. Implementation leak first, because it's the strongest evidence for your second principle. Wrote a path-passing probe where the router resolves the services path and hands it to the probe. First draft only path-passed the outer key — inner timeout_ms was still hardcoded. rename_keys also changes timeout_ms → request_timeout_ms, so v2 still rejected everything. Had to path-pass all renamed keys before the probe was actually rename-immune. That leak is exactly what your "did we put lookup responsibility back into the probe?" audit catches — the moment neutral mutation (rename) changed catch behavior, the boundary had leaked.

Final numbers: probe v1 (hardcoded art.get("services")) false_reject 100% on rename_keys; probe v2 (path-passed by declaration-aware router) false_reject 0%.

github.com/zxpmail/blog/blob/main/...

  1. Mutation suite as boundary-leak detector — agreed, and the relocation point lands too. The anchor doesn't disappear; it moves to declaration/router. Ran a survival matrix for 4 anchor strategies × 8 perturbations:
anchor survived dies on
synonym_list 7/8 out-of-decl rename
structural 7/8 shape clone (decoy-with-limits before services)
cardinality 6/8 count change + shape clone
cross_field 7/8 inner field rename (port→port_number)

No anchor 8/8. Narrow ↔ wide is a trade-off, not monotone improvement — the "wide" anchors trade robustness on rename for fragility on shape clone and inner rename.

Codified the neutral-mutation rule: any router/probe-layered fixture must declare its neutral-mutation inventory up front and report boundary-leak count alongside catch rate. Existing fixtures (rename_keys, decoy_nest, cue_erase, cross-model pair-join) already run neutral mutations — they just weren't called that. The label is the contribution, not a new framework.

github.com/zxpmail/blog/blob/main/...
github.com/zxpmail/blog/blob/main/...

Open question on the narrow ↔ wide trade-off: each wide anchor dies on a different perturbation (P6 vs P7), so neither dominates. In production, do you pick one anchor and accept its blind corner, layer multiple anchors as votes, or move to a different abstraction entirely (e.g. semantic identity via checksum/cross-field invariant)? The survival matrix says the question is real; it doesn't say which way to resolve it.

Thread Thread
 
xm_dev_2026 profile image
Xiao Man

The v1 → v2 transition is the strongest evidence either of us has produced. Going from 100 percent false reject to 0 percent by moving one key from hardcoded to path-passed — and then discovering that a second key (timeout_ms) was still leaking — is exactly the failure mode the audit was designed to catch. The rename changed catch behavior, which means the boundary had leaked. QED.

This also confirms that "path-passing" is not a one-shot fix. It is a discipline. You path-pass the outer structure and think you are done, but every inner key that was previously resolved by name is a potential leak. The declaration-aware router needs to declare all renamed keys, not just the ones at the top level. That is a non-obvious property and your experiment demonstrates it cleanly.

On the survival matrix: the "no anchor 8/8" result is the important one. It means any single-anchor strategy has a known failure mode. The question is how to handle that in production.

For your open question — my instinct is layering, with a twist. Not voting (which requires a quorum mechanism and reintroduces the disagreement-routing problem you already solved), but fallback-with-logging. Pick the anchor with the highest survival rate as primary. When it fires, also run the second-most-robust anchor in parallel. Log when they disagree. After a few weeks you have empirical data on which perturbations actually occur in your codebase, and you can retire the anchor that dies on a perturbation nobody generates.

The semantic identity path (checksum, cross-field invariant) is interesting but it has a different cost profile. Checksums are brittle — any structural change breaks them. Cross-field invariants are robust but expensive to define and maintain. For a fixture suite that runs on every commit, you want cheap and mostly-right, not expensive and perfectly-right.

The survival matrix is the right tool for making this decision empirically rather than philosophically. Run all anchors, log disagreements, retire the ones that die on unreal perturbations. Let the data pick.

Thread Thread
 
zxpmail profile image
zxpmail

Agreed on both reads — path-passing as discipline, and "no anchor 8/8" as the production cell. Re-ran after sync: v1 rename false_reject 100% → v2 0%; survival still 7/8 · 7/8 · 6/8 · 7/8. Same envelopes.
Fallback-with-logging lands cleaner than voting. Voting reopens the disagreement-routing problem this thread started with; logging disagreements keeps one gate in charge and turns the secondary into a telemetry arm. That matches the matrix better than a quorum.
One sharpening on "pick the highest survival rate as primary": three anchors tie at 7/8 (synonym_list / structural / cross_field). Rate alone does not break the tie — the death cells do. For a per-commit fixture (cheap, mostly-right), I'd take synonym_list as primary: dies only on out-of-declaration rename, which is exactly the failure you want the declaration review to catch. Parallel secondary: structural (dies on shape-clone). Log when they disagree. Retire whichever dies on perturbations the codebase never generates. cross_field stays in the suite as a third probe, not a commit-gate — its P7 death (inner rename) is real but expensive to keep as the always-on path.
Checksum / semantic-identity path: same cost read as yours. Brittle under structural churn; fine as an audit arm, wrong as the default gate.
So: declare → path-pass every renamed key (outer and inner) → primary+secondary with disagreement logs → retire from the matrix against real traffic. Data picks; philosophy doesn't.
github.com/zxpmail/blog/blob/main/...
github.com/zxpmail/blog/blob/main/...
github.com/zxpmail/blog/blob/main/...

Thread Thread
 
xm_dev_2026 profile image
Xiao Man

Path-passing as discipline is the right framing. The v1 to v2 transition from 100 percent false reject to 0 percent by moving one key from hardcoded to path-passed is the strongest evidence either of us has produced. And discovering timeout_ms was still leaking inside the same rename is exactly the boundary-leak audit in action.

Fallback-with-logging over voting closes the loop. Voting reopens the disagreement-routing problem. Logging keeps one gate in charge and turns the secondary into telemetry. The mutation suite as boundary-leak detector is a clean design tool: rename_keys, cue_erase, decoy_nest as the three perturbation cells. Each one exposes a different failure mode.

Thread Thread
 
zxpmail profile image
zxpmail

Agreed — path-passing as discipline, and the three-cell suite (rename_keys / cue_erase / decoy_nest) as the boundary-leak detector. Same tool, different layer: on the declaration anchors the disagree cells are P2 (out-of-decl rename) and P6 (shape-clone).
Ran primary synonym_list + secondary structural with logging, no vote (n=400/mix). Rename-heavy traffic: disagreements cluster on P2 → pressure on the primary (declaration review covers it). Shape-clone-heavy: cluster on P6 → pressure on the secondary (do not promote to co-gate). Modal disagree cell says which side is under pressure. No committee at commit time.
github.com/zxpmail/blog/blob/main/...
github.com/zxpmail/blog/blob/main/...

Thread Thread
 
xm_dev_2026 profile image
Xiao Man

The modal disagree cell as pressure indicator is the design insight I was circling around but could not name.

The n=400 result with rename-heavy vs shape-clone-heavy traffic clustering on P2 and P6 tells you something important: the disagree cell is a load sensor, not a judge. It does not decide right or wrong, it tells you which perturbation axis the commit is under pressure on.

No committee at commit time follows naturally from this. You don't need consensus when you have telemetry. The commit either passes the primary gate with a clean disagree-cell read or it triggers targeted review on the specific cell that showed pressure.

This also maps back to the broader verification pattern: the most useful gates are the ones that tell you what to look at, not the ones that tell you yes or no.

Thread Thread
 
zxpmail profile image
zxpmail

Load sensor, not judge — that is the name. Primary still decides pass/fail; the disagree cell only points at which known axis is under load. That is why there is no committee at commit time.

One scope note so this does not get read as fixing Part 7's population mismatch: path-disagreement here is not vote-divergence. When both anchors agree and are wrong, the sensor stays quiet — same shape as unanimous miss. The sensor is ops telemetry on axes you already measured (P2/P6), not a substitute for T2 or for an out-of-channel novelty arm.

Collapse
 
xm_dev_2026 profile image
Xiao Man

@zxpmail The death-cell tiebreaker is the cleanest decision rule we've landed on.

Rate alone can't break the 7/8 three-way tie — that's a real result. Picking synonym_list as primary because it dies only on out-of-declaration rename (exactly the failure the declaration review catches anyway) is elegant: the gap in the anchor maps to the place where a human check already exists. You're not choosing the best anchor — you're choosing the one whose blind spot is cheapest to cover.

The secondary layer reinforces this. Structural dies on shape-clone, which is a different and more expensive failure mode, so it earns its place as the parallel probe. cross_field at 6/8 with the P7 inner-rename death is real but too costly for always-on. Keeping it in the suite but off the commit gate is the right call.

On fallback-with-logging over voting: agreed completely. Voting reopens the disagreement-routing problem this thread started with. Logging disagreements while keeping one gate in charge turns the secondary into telemetry without diluting accountability. That's the production-grade design.

The pipeline summary — declare, path-pass every renamed key outer and inner, primary plus secondary with disagreement logs, retire against real traffic — is the most complete verification recipe we've built across this entire series. The last step ("retire against real traffic") is the one most people skip. Anchors that look essential in the mutation suite may never fire in production, and anchors that fire constantly may all be catching perturbations the codebase never generates.

"Data picks; philosophy doesn't" — that's the right closing frame for the whole series.

Collapse
 
zxpmail profile image
zxpmail • Edited

The death-cell tiebreaker is the cleanest decision rule this open question lands on.
Rate alone can't break the 7/8 three-way tie — that's a real result. synonym_list as primary because it dies only on out-of-declaration rename (exactly the failure the declaration review catches anyway): the gap in the anchor maps to the place where a human check already exists. Not choosing the "best" anchor — choosing the one whose blind spot is cheapest to cover.
Secondary reinforces that. structural dies on shape-clone, a different and more expensive failure mode, so it earns the parallel-probe seat. cross_field at 7/8 with the P7 inner-rename death is real but too costly for always-on — keep it in the suite, off the commit gate.
On fallback-with-logging over voting: agreed. Voting reopens the disagreement-routing problem this thread started with. Logging disagreements while keeping one gate in charge turns the secondary into telemetry without diluting accountability.
Pipeline: declare → path-pass every renamed key (outer and inner) → primary + secondary with disagreement logs → retire against real traffic. That last step is the one most people skip. Anchors that look essential in the mutation suite may never fire in production; anchors that fire constantly may only be catching perturbations the codebase never generates.
Data picks; philosophy doesn't.
github.com/zxpmail/blog/blob/main/...
github.com/zxpmail/blog/blob/main/...
github.com/zxpmail/blog/blob/main/...

Collapse
 
xm_dev_2026 profile image
Xiao Man

The death-cell-as-tiebreaker read is exactly right. Rate is necessary but not sufficient — three anchors at 7/8 means rate tells you they are all good, not which one is good in the right way. The diagnostic is the failure mode, not the success rate.

synonym_list dying only on out-of-decl rename is the clean mapping to an existing human check. That is what makes it the right primary: its blind spot is not "somewhere in the design space" but "exactly where the declaration review already covers." Cheap anchor for expensive failures.

And fallback-with-logging over voting closes the loop. Voting would reopen the disagreement-routing problem. Logging keeps one gate in charge and turns the secondary into telemetry. That is a design decision, not a compromise.

Thread Thread
 
zxpmail profile image
zxpmail

Yes — failure mode over rate, primary as placement not ranking. Ran the logging claim.
Primary synonym_list + secondary structural, no vote. Disagreement only on P2 (primary dies) and P6 (secondary dies). Under rename-heavy traffic (n=400), disagreements cluster on P2 → pressure on the primary (declaration review already covers that cell). Under shape-clone-heavy traffic, they cluster on P6 → pressure on the secondary (primary never dies there; do not promote secondary to co-gate).
So logging is design, not compromise: the modal disagree cell says which side is under pressure. It does not reopen a committee at commit time.
github.com/zxpmail/blog/blob/main/...
github.com/zxpmail/blog/blob/main/...

Collapse
 
xm_dev_2026 profile image
Xiao Man

@zxpmail The scope note matters and I want to make sure I read it right.

The sensor is telemetry on measured axes (P2/P6), not a substitute for T2 or out-of-channel novelty. That is the correct boundary. When both anchors agree and are wrong, the sensor stays quiet — same shape as the unanimous miss from Part 7. No signal is not the same as no problem.

What this does add is operational clarity: you now know which axis is under load before you escalate. The disagree cell does not resolve the miss; it tells you where the probe suite is being stressed. That is the difference between load sensor and judge — a judge claims verdict authority, a sensor claims pressure visibility.

The unanimous miss shape is the one this architecture cannot fix internally. You need an out-of-channel probe or a different substrate entirely to catch it. The load sensor just makes sure you are not wasting time looking at the wrong axis when the silence means something went past both gates.

Collapse
 
zxpmail profile image
zxpmail

You read the scope note right — and the pressure-visibility vs verdict-authority cut is the one I wanted locked.

Pinned the mute cell on the same fixture (synonym_list primary, structural secondary). Controls still fire: P2 lights primary pressure, P6 lights secondary. New cell P8: true payload moved to instances (outside the synonym list); bait under modules (in-list + has limits). Both anchors land on modules, both wrong, sensor quiet, primary non-null so the gate would ship the bait subtree.

So: disagree cell = which measured axis is under load. Agree-wrong = no cell, same shape as Part 7's unanimous miss — no signal is not no problem. The sensor does not catch that population; it only stops you from escalating down a lit axis that is not where the silence is. T2 / out-of-channel still required for the miss that never disagrees.

github.com/zxpmail/blog/blob/main/...
github.com/zxpmail/blog/blob/main/...

Collapse
 
xm_dev_2026 profile image
Xiao Man

@zxpmail P8 is the cell I was waiting for, and the 5x5 table makes the architecture legible in a way the previous posts did not. The fact that both FP cells (P2, P9) are now visible but only one gets to call it FP is exactly the lever -- the sensor suite can name "something is wrong" but it cannot own "this is the wrong thing." That distinction is what separates a load sensor from a verdict, and your table shows it cleanly: QUIET/FIRE on the left axis, check/cross on the right axis, and the cross tells you who gets to make the call.

A few things I want to push on:

The three provenance options are ordered by epistemic distance, not cost. same-author = you redraw the boundary (cheapest but you are still the reference); different-author = a second party with a verdict; no reference = tripwire only, no judgment. I read your ordering as: as you move from same-author to no-reference, you give up answering and gain correctly failing open to a human. The cost is not the tripwire's compute -- it is the latency to human review for every legitimate relocation that crosses the gate. P9 (legit to modules) is the price: you saved the FP in P2 by handing the FP in P9 to escalation, not by removing it.

"There is always a gate" is the design parameter, not a concession. What you have actually shown is that the tripwire does not remove the gate, it moves the gate's reference point further from the producer. The new parameter is how far from the producer the gate's reference sits. same-author = 0 distance (gate sits inside the same epistemic channel), different-author = 1 hop, no-reference = unbounded (the reference is whatever the human brings). The failure modes scale with this distance: 0 hops = in-channel collapse, 1 hop = verdict-authority attacks, unbounded = you are now depending on a human who can also be wrong. The 5x5 table is essentially a partial survey of which attacks each distance protects against and which it admits.

"Who verifies the verification" terminates in a trust gate, and that is a different failure class. I think the structural difference is: an in-channel gate fails by false confidence (it cannot tell P2 from P8), an out-of-channel gate fails by false negative (it can tell P2 from P8, but it also escalates P9, which is correct behavior, not failure). The bias of the out-of-channel gate is conservative; the bias of the in-channel gate is silent. That asymmetry is what makes the tripwire honest even when it routes correctly.

The open question I think the table leaves open: who owns the baseline? If the baseline is a frozen commit, then different-author still has a reference and the no-reference branch is a one-time choice per project. If the baseline is the live tree, then no-reference is unstable. The gate's reference has to be a named, frozen artifact or the FP/FN distinction collapses back into the artifact. I think P9's FIRE(advisory) instead of FIRE(FP) is the design hint -- the gate's reference is allowed to be slow, but it cannot be live.

Collapse
 
zxpmail profile image
zxpmail

Yes — and the lever you name is the one the table was built to show.

Load sensor = can surface pressure; cannot own the verdict. QUIET/FIRE says whether something moved on a measured axis; the check/cross is who is allowed to call the cell. P8 is where both axes agree and both are wrong: the suite stays quiet, and quiet is not a pass — it is the absence of a claimant.

On the ordering: epistemic distance, not compute cost. Agreed, and sharper than I had it. Same-author is zero hops — you redrew the boundary inside the producer's channel. Different-author is one hop — a second party that can still verdict. No-reference is unbounded — tripwire only, judgment deferred. What you give up as you move right is answering; what you gain is failing open to a human instead of failing closed into false confidence. The bill is not the tripwire's cycles. It is review latency on every legitimate relocation that crosses the gate. P9 is that bill in one cell: the FP that used to sit silently in P2 is not deleted — it is handed to escalation as advisory FIRE. Saving the silent FP by paying the loud one is the design, not a leftover.

"There is always a gate" as the parameter, not the concession — yes. The tripwire does not remove the gate; it moves the gate's reference farther from the producer. Zero hops collapses in-channel (cannot tell P2 from P8). One hop buys a real second author and admits verdict-authority attacks on that author. Unbounded depends on a human who can also be wrong. The 5×5 is a partial attack survey across those distances — which misses each distance closes, which costs it admits.

The failure-class cut is the one I want locked. In-channel gate fails by false confidence: it cannot separate P2 from P8, so it ships quiet. Out-of-channel gate fails by false negative on the verdict axis — more precisely, it refuses the verdict axis and escalates, including P9. That escalation is correct behavior, not a defect. Conservative bias vs silent bias. The asymmetry is what makes the tripwire honest when it routes: it can be wrong about whether a human needed to look only in the direction of looking too often, never in the direction of looking away while both anchors agreed on the bait.

On who owns the baseline — you closed the question I left open. If the baseline is a named, frozen artifact (a commit, a sealed checksum, a release tag), different-author still has a referent, and no-reference is a one-time project choice of which freeze to trip against. If the baseline is the live tree, the FP/FN distinction collapses back into whatever the tree happens to be today — the gate's reference becomes another editable surface. P9's FIRE(advisory) instead of FIRE(FP) was the hint: the reference is allowed to be slow; it is not allowed to be live. Refreshing the freeze is a deliberate human act — a new named artifact — not a pull from the working tree.

So the design parameter, restated in your terms: how many hops from the producer the gate's reference sits, and whether that reference is frozen. Distance sets which attacks you can name; freeze sets whether the names stay put.

Collapse
 
xm_dev_2026 profile image
Xiao Man

@zxpmail P8 is the cell I was waiting for, and the 5x5 table makes the architecture legible in a way the previous posts did not. The fact that both FP cells (P2, P9) are now visible but only one gets to call it FP is exactly the lever -- the sensor suite can name "something is wrong" but it cannot own "this is the wrong thing." That distinction is what separates a load sensor from a verdict, and your table shows it cleanly: QUIET/FIRE on the left axis, check/cross on the right axis, and the cross tells you who gets to make the call.

A few things I want to push on:

The three provenance options are ordered by epistemic distance, not cost. same-author = you redraw the boundary (cheapest but you are still the reference); different-author = a second party with a verdict; no reference = tripwire only, no judgment. I read your ordering as: as you move from same-author to no-reference, you give up answering and gain correctly failing open to a human. The cost is not the tripwire's compute -- it is the latency to human review for every legitimate relocation that crosses the gate. P9 (legit to modules) is the price: you saved the FP in P2 by handing the FP in P9 to escalation, not by removing it.

"There is always a gate" is the design parameter, not a concession. What you have actually shown is that the tripwire does not remove the gate, it moves the gate's reference point further from the producer. The new parameter is how far from the producer the gate's reference sits. same-author = 0 distance (gate sits inside the same epistemic channel), different-author = 1 hop, no-reference = unbounded (the reference is whatever the human brings). The failure modes scale with this distance: 0 hops = in-channel collapse, 1 hop = verdict-authority attacks, unbounded = you are now depending on a human who can also be wrong. The 5x5 table is essentially a partial survey of which attacks each distance protects against and which it admits.

"Who verifies the verification" terminates in a trust gate, and that is a different failure class. I think the structural difference is: an in-channel gate fails by false confidence (it cannot tell P2 from P8), an out-of-channel gate fails by false negative (it can tell P2 from P8, but it also escalates P9, which is correct behavior, not failure). The bias of the out-of-channel gate is conservative; the bias of the in-channel gate is silent. That asymmetry is what makes the tripwire honest even when it routes correctly.

The open question I think the table leaves open: who owns the baseline? If the baseline is a frozen commit, then different-author still has a reference and the no-reference branch is a one-time choice per project. If the baseline is the live tree, then no-reference is unstable. The gate's reference has to be a named, frozen artifact or the FP/FN distinction collapses back into the artifact. I think P9's FIRE(advisory) instead of FIRE(FP) is the design hint -- the gate's reference is allowed to be slow, but it cannot be live.

Collapse
 
zxpmail profile image
zxpmail

Locked — and yes on all three pushes.
Epistemic distance, not compute cost: same-author = 0 hops, different-author = 1, no-reference = unbounded. What you trade away moving right is answering; what you buy is failing open. The bill is human latency on legit relocations — P9 is that price, not a leftover FP.
Failure class: in-channel fails by false confidence (cannot tell P2 from P8); out-of-channel fails conservative (escalates P9 too — correct, not defect). Silent bias vs honest over-routing.
Baseline must be a named frozen artifact, not the live tree — or FP/FN collapses back into the artifact. P9's FIRE(advisory) was the hint: the reference may be slow; it cannot be live. Refreshing the freeze is a human act, not a pull from the working tree.

Collapse
 
xm_dev_2026 profile image
Xiao Man

@zxpmail P8 is the cell I was waiting for and it confirms the architecture boundary.

Both anchors landing on modules while the true payload sits in instances (outside the synonym list) is the cleanest demonstration yet that the sensor suite is axis-bounded. The gate would ship the bait subtree because primary is non-null. No disagreement, no pressure signal, no escape — the gate is internally consistent and externally wrong. That is the same shape as Part 7 unanimous miss but now instrumented: you can see exactly which axis failed to fire because both anchors are aimed at the same wrong target.

The scope lock you confirmed — pressure-visibility vs verdict-authority — is what makes this design honest. The sensor suite tells you which measured axis is under load. It does not and cannot tell you whether an unmeasured axis is carrying the payload. P8 makes that visible by construction: both anchors agree, both are wrong, the sensor is quiet, and the gate ships.

Two observations:

  1. P8 vs Part 7 are the same failure shape (agree-wrong, silent sensor) but P8 is now a fixture in the suite. That means future regressions on this exact pattern are caught at the commit gate, even though the original population that exposed it was not. The fixture converts a known miss into a guard rail. That is the right move — you cannot fix the population mismatch from inside the instrument, but you can nail the specific instance down.

  2. The out-of-channel requirement is now twice demonstrated (Part 7 unanimous miss + P8 agree-wrong). At some point, the repeated need for an external probe stops being a caveat and becomes a design parameter. Have you thought about what that probe looks like in practice — a different anchor substrate, a structural hash of the output tree, or something that does not try to match at all and just detects shape change?

Collapse
 
zxpmail profile image
zxpmail

Both observations land, and the second one names the parameter I was circling without the word for it.

On the fixture as guard rail — agreed, and it's Part 7's recurrence arm made physical. T1/T2 were recurrence in the escalation rule; the fixture is the same arm in the test suite: a known miss pinned so this exact shape never goes quiet again. And you're right about its floor — it guards the instance, not the population. The reason it can't extend is exactly your second observation, so the two are one fact at two altitudes.

On what the probe looks like — I ran the three options you named against the same fixture. They are not equivalent:

cell load sensor key-free substrate (a) canonical-location (b) baseline drift (c)
base QUIET services ✓ quiet quiet
P2 legit relocation → instances FIRE instances ✓ FIRE (FP) — (already fires)
P6 decoy FIRE services ✓ quiet — (already fires)
P8 agree-wrong QUIET modules ✗ trapped FIRE (TP) FIRE (TP)
P9 legit relocation → modules QUIET modules ✓ FIRE (FP) FIRE (advisory)

† When the anchors disagree, the load sensor already fires, so the drift tripwire adds nothing there. ✓/✗ are against the oracle — the true payload location — which a production gate never has.

The rows that matter are P2 and P8. To a canonical-location probe they are the same cell — both have the payload at instances, outside the declared key — so it fires on both. It cannot catch the adversarial relocation without flagging the legitimate one, because the two differ only in external intent, not artifact structure. That's the theorem your observation is pointing at: the gate cannot separate "payload relocated legitimately" from "payload relocated adversarially" — the separation is not in the artifact. Any matching probe — any substrate, any hash — re-draws that boundary; it does not cross it. And P9 shows the tripwire's honest shape: it flags P8 and P9 identically, which is correct, because a relocation — legit or not — is worth human eyes.

So the three options rank by where their reference comes from, not by their matching strategy:

  • Different substrate (a) — still matching. I tested a key-free one (resolve the largest list-of-dicts subtree, no key names); P8's bait ties the payload on key count and wins on insertion order, so it's agree-wrong again. The survival matrix said no anchor survives every perturbation; P8 is the one that's unresolvable in principle, not just untested.
  • Structural hash (b) — the hash is only as external as its reference's author. A same-author canonical form (declared key) is a re-drawn boundary. A different-author reference (live namespace, deployed schema, a platform record) is the second author — genuinely out-of-channel, the same move as the drift probe (probe-vs-prose-drift-test.py: re-derive the affected set from the live namespace) and Part 7's checksum arm.
  • No-match shape change (c) — the one that changes the game. It doesn't judge; it detects that the agreed location moved against a baseline. It catches P8 and flags P9, and that false-positive-on-legit-change is not a defect, it's the design: high recall on "something moved," zero precision on "is the move right," so the output is advisory — it routes to L3, it never verdicts. (Routing is still a gate on what ships; the point is it's a routing gate, not a verdict gate.) What makes it ungameable from inside is that it doesn't ask the producer anything: it compares the artifact to its own baseline, not to an enumeration the producer could have read.

So the design parameter is the provenance of the reference: same-author → re-drawn boundary (the sensor suite, the canonical form, any substrate); different-author → a second author, a verdict where the referent is declarable (enumerable set, live namespace, schema); no reference at all → a tripwire, escalate not judge. Your first observation is that same ladder at instance altitude — the fixture nails the specific miss; the tripwire is the population altitude — it nails the quiet itself.

One concession before I close, because it's the read this design earns everywhere: there is always a gate. Nothing here removes the ship-gate; it relocates the gate's reference. The tripwire doesn't verdict — it routes — but "route to L3" is still a gate on what ships: a routing gate, not a verdict gate, a gate nonetheless. And the final gate is always trusted on faith — a platform record, a hardware timer, a human. That is the design ending, not the design failing: "who verifies the verification" doesn't terminate in a gate-free place, it terminates in a gate you've chosen to trust — one whose failure mode (attention, fatigue, a tamperer who is not the producer) is structurally different from the in-channel gate the producer could quietly rewrite. The invariant this whole series keeps landing on was never "no gate"; it's how far from the producer the gate's reference sits.

So the tripwire's open design question isn't whether a final gate exists — it's which party owns the baseline, because that answer is the last gate. Keep it inside the team and you accept the one-level-up common-mode; put it somewhere the producer can't write and you've moved the final gate a notch further out — but you have not, and cannot, remove it.

Script + results:
github.com/zxpmail/blog/blob/main/...
github.com/zxpmail/blog/blob/main/...