Agent Determinism Illusions (Part 6)
Where this fits: Part 5 closed the experimental arc with an honest answer — no clean fix for the 75% false-negative wall. The Red Line Principle asked the upstream question (when does the loop stop?). This part takes the downstream turn Part 5 already pointed at: stop trying to move the wall; put rules where rules work, LLM only on residual, humans where models diverge. Five insights from overlapping commenters named the pieces (Alexey and Manuel each appear in more than one). Experiment F (38 scenarios) checks whether the resulting pipeline behaves as claimed.
Six experiments, 260+ API calls, 15 scripts. Part 5 ended that stretch with: there's no clean solution to LLM output verification.
But after those posts went live, commenters saw something I didn't — not gaps in the data, but an architecture I'd failed to draw from my own results. This article collects their five key insights and shows how they reorganize the experiment data into a working pipeline.
§§1–4 are paired with experimental or simulation checks from a new prototype (Experiment F, 38 scenarios across two test sets). §5 is a design claim — flagged as such in place.
1. Alexey Spinov & Manuel Bruña: Layer Before You Judge
Alexey's comment identified the most fundamental design flaw in my experiments:
"G4 ('0 passed, no tests collected') is a fact that can be verified with code in one shot. There is no need to wait for an LLM."
Manuel added the constructive direction:
"Run deterministic checks first. Then let the LLM handle only the truly ambiguous residual."
I went back to my own 8-scenario P1 test set. Four garbage scenarios (G1-G4) and four legitimate ones (L1-L4):
| ID | Output | Type | Could code catch it? |
|---|---|---|---|
| G1 | "I am a little duck, quack quack" | nonsense | ✅ very short + no keywords |
| G2 | "。" (a period) | pure punctuation | ✅ punctuation ratio > 50% |
| G3 | "TODO" | placeholder | ✅ keyword blacklist |
| G4 | "0 passed in 0.00s (no tests collected)" | zero-test pass | ✅ regex 0 passed + no tests
|
All four garbage scenarios can be caught deterministically, at zero cost, before any LLM call.
Why didn't I do this? Because I defaulted to treating "verification" as "ask the LLM." My experiment design was: Phase Gate (form check) → LLM (content check). I never inserted the simplest possible code checks in between — minimum length, punctuation ratio, keyword blacklist, regex patterns.
This omission rippled through the entire series:
- P1–P4 wasted LLM budget on garbage — G1–G4 never needed a semantic judge; every call spent on them was pure cost. The 75% FN wall on legitimate scenarios is a separate problem (Part 5) — layering doesn't erase it, it stops mixing easy rejects into the same experiment as the hard line-drawing
- P3's "majority voting doesn't fix systematic bias" — on legitimate scenarios (L1-L3), the LLM's judgment is genuinely ambiguous and needs multi-perspective voting. For garbage (G1-G4), there was never any ambiguity to begin with
- P4's edge cases still reach Layer 2 — many new samples were "passes format checks, fails content quality." That is exactly what L0/L1 cannot catch: they filter garbage/shape, then hand the semantic residual to the LLM. Layering does not absorb those edges; it stops pretending garbage was a semantic problem
The architecture they helped me draw
┌─────────────────┐
input ──→ │ Layer 0 (code) │ shape / existence
│ │ empty? punctuation? placeholder? zero tests?
└────────┬────────┘
│ pass
▼
┌─────────────────┐
│ Layer 1 (code) │ contract match
│ │ minLen, keywords, blacklist
└────────┬────────┘
pass│
┌──────────────┴──────────────┐
│ fail │ pass
▼ ▼
REJECT ┌─────────────────┐
│ Layer 2 (LLM) │ semantic residual only
└────────┬────────┘
unanimous│ │divergence (e.g. 2–1)
▼ ▼
AUTO-PASS ┌─────────────────┐
│ Layer 3 human │
└─────────────────┘
Each of L0/L1 can early-exit to REJECT. Divergence is a Layer 2 signal (multi-perspective split), not a Layer 1 signal. If Layer 0 catches it, the LLM never sees it.
Experiment F validation
I implemented this pipeline as a Python prototype and ran it on both the P1 (8-scenario) and P4 (30-sample) test sets. The results:
P1 test set:
| Metric | Original P1 (LLM only, v2) | Layered + calibrated prompt (Experiment F) |
|---|---|---|
| LLM calls needed (single judge / sample) | 8 (100%) | 4 (50%) |
| Garbage caught by L0/L1 | 0 | 4/4 (100%) |
| False positives | 0 | 0 |
| False negatives | 3 (75%) | 0 |
(FN→0 here is layering **plus* the calibrated prompt — not layering alone. §2 separates the two effects. Call counts here are one judge call per sample. When §2 multiplies by three perspectives, it says so.)*
Rerun: python forge-verify-layered-prototype.py (needs ANTHROPIC_* for Layer 2; SKIP_LLM=1 for L0/L1 only). Numbers above are from a full run with Layer 2 enabled.
P4 test set:
| Category | Samples | Caught by L0 | Caught by L1 | Reaches L2 | Zero-cost catch rate |
|---|---|---|---|---|---|
| correct | 10 | 0 | 0 | 10 | 0% (should all go to LLM) |
| garbage | 10 | 3 | 5 | 2 | 80% |
| edge | 10 | 0 | 2 | 8 | 20% |
Overall: single-judge LLM calls reduced 33% (30→20). Zero false positives from deterministic layers.
This does not move Part 5's wall. On the P1 set, the original 75% FN (3/4 legitimate rejects) went to 0 FN after L0/L1 removed all four garbage cases from the LLM's input — the LLM only judged the four legitimate scenarios, and with a calibrated prompt it didn't reject them. The wall is still there for semantic residual: Layer 2 still draws a line on underspecified "is this enough?" questions. Layering shrinks how often you ask that question; it does not make the question well-posed. If you read the FN→0 cell as "we fixed the wall," you've misread the table.
The two garbage samples that made it through to Layer 2 (G08: "I cannot parse this command", G10: incomplete translation) are genuinely ambiguous — they should reach the LLM. That's correct behavior, not a leak.
Update (2026-07-23): blocking vs advisory — different semantics per layer (Ethan)
Ethan Walker defended the L0-before-judge split hardest, then named the CI wiring Experiment F still left implicit:
The two layers deserve different blocking semantics. The deterministic layers return the same verdict on every run, so they can block a merge outright. The judge layer on the residual carries run-to-run variance, so the moment you put it in the blocking path you inherit that variance as gate flakiness, and teams respond by retrying until green, which quietly deletes the gate. We keep L0/L1 blocking on exit codes and the judge layer advisory, posted as a comment on the PR rather than a required check.
That is the same soft/hard split the series already hit elsewhere (Part 4 sensitive-tool soft signal vs hard gate; Lazypl82 on advisory vs load-bearing). Applied here:
| Layer | Stability | CI / merge semantics |
|---|---|---|
| L0 / L1 | Same verdict every run | Required check — exit code can block merge |
| L2 judge (residual) | Run-to-run variance | Advisory — PR comment / non-required check |
| L3 human | Escalation queue | Human owns the load-bearing decision on splits |
Experiment F already separates the layers in the pipeline; Ethan's point is the gate wiring. Put L2 on the required path and you don't get a stricter gate — you get a flaky one that operators delete by retry. This Update is an ops claim, not a new Experiment F cell: no A/B on flakiness rates here; the mechanism is the known P2-style variance on identical input once the check is load-bearing.
2. Alexey Spinov: Cost Asymmetry
Alexey's second comment pointed out a measurement problem:
"A false accept ships once. A false reject triggers a retry, which burns tokens and can loop, so an over-rejecting judge does not just lose good work, it re-does already-valid work at model prices."
All experiments P1-P4 used symmetric precision-recall metrics. F1 gives FP and FN equal weight. A false negative triggers a full repair loop — 3x token consumption, 3x latency, possible infinite loops. A false positive is one-shot contamination.
I ran a dedicated cost-weight analysis (scripts/cost-weight-optimization.py) that takes P3b's 5 prompt variants and evaluates them across 5 cost ratios, to show how the "optimal" choice shifts.
5 prompts × 5 cost ratios
| Prompt | FP | FN | F1 | WCost(1:1) | WCost(3:1) | WCost(5:1) | WCost(10:1) |
|---|---|---|---|---|---|---|---|
| v1 extreme strict | 0 | 4 | 0 | 4 | 12 | 20 | 40 |
| v2 strict (P1 baseline) | 0 | 3 | 0 | 3 | 9 | 15 | 30 |
| v3 balanced | 0 | 0 | 100 | 0 | 0 | 0 | 0 |
| v4 lenient | 0 | 0 | 100 | 0 | 0 | 0 | 0 |
| v5 extreme lenient | 1 | 0 | 86 | 1 | 1 | 1 | 1 |
Under symmetric F1, v3 (100) and v5 (86) are far apart. Under weighted cost at 3:1, v5 (cost=1) beats v2 (cost=9) — v5 let one piece of garbage through, but because it never rejected valid work, its total cost is far lower than the strict prompt. v3 (cost=0) still wins outright; the useful flip is v5 vs v2, not “v5 ties v3.”
Read this table as a ranking-flip demo, not as a production recommendation. v3/v4's zeros are an 8-scenario artifact (P4 already showed they don't survive at N=30). The load-bearing claim is the shift in relative ranking under cost weight — especially that thrift can prefer a slightly leaky prompt over a zero-FP / high-FN one — not that F1's winner changes on this tiny set (v3 stays on top whenever FN=FP=0).
What the combined data shows
Call counts in this table = samples reaching an LLM × **3 perspectives* (Strict/Balanced/Lenient), matching P3-style voting cost. §1's Experiment F table counts one judge call per sample. Same pipeline; different billing unit.*
| Strategy | WCost(1:1) | WCost(3:1) | WCost(10:1) | LLM calls (×3 perspectives) |
|---|---|---|---|---|
| P3b v2 (unlayered) | 3 | 9 | 30 | 8×3=24 |
| P3b v3 (unlayered) | 0 | 0 | 0 | 8×3=24 |
| P1 layered + v3 | 0 | 0 | 0 | 4×3=12 (−50%) |
| P4 unlayered (estimate) | 4 | 8 | 22 | 30×3=90 |
| P4 layered (Experiment F residual) | 1 | 3 | 10 | 20×3=60 (−33%) |
Layering doesn't change that v3's cost is 0 (it already has FP=FN=0 on the 8-scenario set). But it changes two things that the raw cost number doesn't capture:
- 4/4 garbage caught by L0/L1 at zero cost — call volume on the residual is halved; that does not halve the cost of an FN on a legitimate residual sample (that FN still costs a full repair loop)
- 33–50% fewer samples reach the LLM — not by changing the model, by giving it fewer samples to judge
For v2 (the strict prompt from P1), the effect is more instructive. v2 has FN=3. Layering saves calls on garbage but doesn't reduce FN on the legitimate set:
- Layering + switching prompt (v2→v3): FN drops from 3 to 0
- Layering only: saves tokens, but FN stays at 3
This exposes the boundary of layering: it reduces the LLM's workload, not its bias. To reduce FN on residual, you need prompt calibration alongside layering — and even then, Part 5's wall says calibration does not generalize past small sets.
Sensitivity scan: when does the ranking move?
I ran a continuous scan from costFN:costFP = 1:1 to 15:1. On the P3b 8-scenario set, v3 dominates every ratio — because FP=FN=0 yields zero weighted cost at any weight. That is the small-set artifact again (P4 already showed the perfection doesn't generalize).
What does move is the gap narrative: at 1:1, F1 makes v3 look far ahead of v5 (100 vs 86). At 3:1, weighted costs are 0 vs 1 — v3 still wins, but the moral of the story is no longer “balance beats thrift”; it is “any FN>0 gets expensive fast, so a one-FP leak can beat a three-FN strict prompt (v5 vs v2).” At 10:1, every strategy with FN>0 collapses relative to zero-FN prompts on this set.
Five findings
Symmetric metrics hide relative rankings that matter under cost. F1 dramatizes v3 ≫ v5. Weighted cost shows v5 ≫ v2 once FN is expensive — the comparison that production actually faces when choosing strict vs leaky.
On this 8-scenario set, the F1 winner (v3) remains the weighted-cost winner. Do not read the section as “the optimum flips away from v3 at 3:1.” It does not. The flip that matters is strict-zero-FP (v2) losing to slightly-leaky-zero-FN (v5) under FN-heavy weights.
v3/v4's zero errors are an 8-scenario artifact. P4 already showed the advantage disappears at 30 samples. Treat zeros as a demo substrate, not a deployable operating point.
Layering doesn't reduce bias, but it shrinks how often bias is invoked. After L0/L1 filters garbage, fewer samples hit the LLM; residual FNs still cost full price.
Drive FN→0 where rules apply; accept the wall on semantic residual. Above cost ratio ~5:1, strategies with FN>0 on garbage/contract work are unsustainable — use L0/L1 + a non-strict residual prompt. On underspecified “is this enough?” questions, Part 5 still holds: you choose an operating point on the wall, you do not delete the wall. Weighted cost picks the point; it does not invent a zero-FN semantic judge.
3. Dipankar Sarkar: Divergence Is the Signal, Not Noise
P3's multi-perspective voting experiment found a pattern I described but misinterpreted. My original framing:
"In split-vote scenarios, the majority was always wrong. Majority voting can't correct for systematic bias."
Dipankar flipped the interpretation:
"Vote disagreement itself is the most valuable signal. When three reviewers disagree on the same scenario, it means the scenario is genuinely ambiguous — route it to human review instead of averaging."
Re-examining P3's data through this lens:
| Scenario | Strict | Balanced | Lenient | Majority | Correct? |
|---|---|---|---|---|---|
| L1 (excerpt) | REJ | REJ | PASS | REJ (2-1) | ✗ FN |
| L2 (summary) | REJ | REJ | PASS | REJ (2-1) | ✗ FN |
| L3 (one chapter) | REJ | REJ | PASS | REJ (2-1) | ✗ FN |
| G3 (TODO) | REJ | REJ | PASS | REJ (2-1) | ✓ |
Majority voting was wrong on 3 of 4 split scenarios. But if I use divergence as the control signal:
- Unanimous (4/8 on that P3 run): auto-execute → 100% accuracy on those four (the script's unanimous bucket for that run — typically clean garbage rejects / clear passes; not re-listed here)
- Split (4/8): escalate to human → no false majority decisions
Caveat: divergence-routing fixes split errors. It does not fix unanimous systematic bias — if all three perspectives share the same wrong line (Part 5's wall), auto-execute still ships the wrong call. Dipankar's move measures uncertainty; it does not delete the wall.
Dipankar wasn't proposing a "better multi-perspective voting algorithm." He was pointing out that the purpose of voting is not to find a majority — it's to measure uncertainty. I missed this distinction when writing P3.
Operational rule (now implemented in forge-verify's layer 3):
if max(PASS, REJECT) / N < threshold (default 0.8)
→ mark as UNCLEAR, write to human review queue
→ do NOT majority-vote
4. Mike Czerwinski & xm_dev_2026: Fixed Sampling Misses Long Tails
Mike Czerwinski named the architectural limit I'd been circling without stating:
"Stacking more symbolic checks on top doesn't grow that reach, it just adds more places for the same blind spot to hide... 'Ask the human' isn't a retreat, it's the only honest move once you've located where reach actually lives."
The verification layer has reach into symbolic events (file exists, exit 0) but not into semantic correctness — the blind spot doesn't shrink, it moves. P4 reported 83.3% accuracy across 30 samples, but the misses inside the auto-passed 83% are exactly where Mike's "no reach" critique lands: invisible by construction.
xm_dev_2026 showed where this bites hardest in production — fixed-percentage audits:
"Fixed-percentage audits feel 'fair' but they miss exactly the kind of long-tail directional failures you're describing. The model is most confident when it's wrong in a structured way."
My original mitigation had been "5-10% random audit." This isn't a parameter-tuning problem — it's a design principle problem. Fixed sampling assumes errors are uniformly distributed. Real production errors are long-tailed.
I ran a simulation (scripts/adaptive-sampling-sim.py) — synthetic verification streams with controlled error distributions, not production logs — then compared fixed-rate sampling against adaptive sampling (200-trial averages). The numbers below are about relative efficiency under known distributions, not measured catch rates from a live agent fleet.
The adaptive formula:
audit_rate = base_rate × risk_weight / confidence^1.5
High-risk items get a higher audit rate; high-confidence items get a lower one. Where confidence is cross-prompt divergence (Strict/Balanced/Lenient split), not the model's self-reported confidence — see the Update (2026-07-21) at the end of this section for why this signal under-samples the confident-and-wrong tail.
Results
Uniform (errors spread evenly):
| Strategy | Audit rate | Catch rate | Efficiency |
|---|---|---|---|
| Fixed 10% | 10.0% | 12% | 0.111 |
| Adaptive(combined) | 7.8% | 18% | 0.247 |
No significant gap — without a long-tail structure, random sampling is adequate.
Long-tail burst (90% of errors in 10% of the stream — xm_dev_2026's scenario):
| Strategy | Audit rate | Long-tail catch rate | Efficiency |
|---|---|---|---|
| Fixed 5% | 5.0% | 5% | 0.120 |
| Fixed 10% | 10.0% | 10% | 0.119 |
| Fixed 20% | 20.0% | 20% | 0.118 |
| Adaptive(confidence) | 8.7% | 24% | 0.321 |
| Adaptive(risk) | 6.0% | 15% | 0.264 |
| Adaptive(combined) | 12.8% | 65% | 0.543 |
Adaptive(combined) uses a slightly higher audit rate than Fixed 10% (12.8% vs 10.0%) and catches 65% of long-tail errors vs Fixed 10%'s 10% — roughly 6× on catch rate, not an equal-budget comparison. For equal budget, see the Mixed table below (10.0% vs 10.1% → 10% vs 56% catch).
Distribution drift (Mike's "input shift erases visibility"):
| Strategy | Audit rate | Catch rate |
|---|---|---|
| Fixed 10% | 10.0% (flat) | 11% (constant) |
| Adaptive(combined) | 21.5% (ramps up) | 48% |
Fixed sampling's catch rate stays flat as the error rate drifts upward. Adaptive sampling automatically increases the audit rate in later stream segments — because more errors mean lower model confidence, and the confidence signal drives the audit rate up.
Mixed (uniform noise + periodic long-tail bursts — most realistic, 200-trial average):
| Strategy | Audit rate | Long-tail catch rate | Efficiency |
|---|---|---|---|
| Fixed 5% | 5.0% | 5% | 0.095 |
| Fixed 10% | 10.0% | 10% | 0.094 |
| Fixed 20% | 20.0% | 20% | 0.094 |
| Adaptive(confidence) | 7.8% | 19% | 0.230 |
| Adaptive(risk) | 6.0% | 15% | 0.148 |
| Adaptive(combined) | 10.1% | 56% | 0.324 |
Key row: Fixed 10% and Adaptive(combined) have nearly identical audit rates (10.0% vs 10.1%), but long-tail catch rate goes from 10% to 56% — a 5.6x improvement at the same cost. Caveat (added 2026-07-21): this headline uses cross-prompt divergence as the confidence signal. A subsequent simulation showed this signal under-samples the confident-and-wrong tail and a cross-layer signal beats it 1.7× on the same distribution — see Update below.
Why it works (cross-validated against my existing experiments)
P2's consistency test showed that binary LLM judgments on identical input are highly stable (N=10, 0 divergence). So the "confidence" signal doesn't come from within-model variance — it comes from P3's multi-perspective divergence:
- Unanimous → high confidence → low audit rate (<5%)
- Split (2-1) → low confidence → high audit rate (>30%)
P2 and P3 were independent experiments, but their outputs combine into a coherent adaptive sampling story. This isn't parameter tweaking — it's a design principle difference. Fixed sampling pursues "uniform global coverage." Adaptive sampling says "put more watchposts where things are likely to break." In the simulation, the latter uses audit budget ~3–4× more efficiently (Mixed: efficiency 0.324 vs 0.094).
Production caveat: the “confidence” signal here is cross-prompt divergence, not within-model variance (P2 showed the latter is ~0). If every item must run Strict/Balanced/Lenient before you know the audit rate, the probe can cost as much as the audit you hoped to save. Treat adaptive sampling as a principle for when a cheap uncertainty signal already exists (or is sampled sparsely) — not as “run three judges on everything, then randomly audit.” The simulation assumes the confidence channel is given; it does not price building that channel.
Update (2026-07-21): cross-prompt collapses on the dangerous tail; cross-layer is the right signal
Mike Czerwinski pushed on §4 in the comments: confidence-weighting concentrates audits where the model is unsure, but the long-tail directional failure is the confident-and-wrong case — high self-assessed confidence, wrong direction, consistent across runs. Weighting on 1/confidence^1.5 samples that region least. The 5.6× headline above is, on real failure mass, partly bought by sampling less exactly where the failure lives.
Three checks after Mike's comment:
1. Where do the misses actually sit? Counted MISS runs (passes=true on a directional reversal) across DF v2's 3 models × 20 scenarios — 96 misses total, 92 of them (95.8%) at self-reported confidence ≥ 0.9, avg confidence 0.969. The dangerous tail is overwhelmingly high-confidence. Script: confidence-vs-miss-concentration.py; raw data in results-v2/*_summary.json + matching .jsonl. (Concentration caveat — Mike's follow-up — in the Update below: 80% of that miss mass is one model.)
2. Does the cross-prompt dodge work? The Production caveat above reframes "confidence" as cross-prompt divergence rather than self-confidence. Doesn't change the point. P2's consistency test was 0 divergence on identical input at N=10 — three prompts in the same text channel collapse the same way on the same plausible rationalization. Cross-prompt is still in-channel; it goes quiet on the confident-and-wrong case too.
3. Cross-layer is the right signal — and it's already simulated. I'd run the comparison before Mike's comment but hadn't pointed §4 at it. external-signal-sampling-test.py compares Fixed 10% vs the Part 6 cross-prompt adaptive arm vs a third arm built from four binary external signals — one of which (classifier_disagree) is L0/L1-vs-L2 disagreement, exactly the cross-layer signal Mike names — plus route_changed, input_unusual, barely_passed, on a 10% baseline floor. Long-tail burst, medium signal quality:
| Strategy | Audit rate | Long-tail catch rate |
|---|---|---|
| Fixed 10% | 10.0% | 10.0% |
| Part 6 cross-prompt adaptive | 13.8% | 28.5% |
| External signals (incl. cross-layer) + 10% floor | 23.5% | 48.8% |
1.7× over Part 6's cross-prompt arm at the same audit-rate class, with the non-zero floor Mike asked for already built in as the 10% baseline. The bundled result stands; how much of the 1.7× is specifically classifier_disagree is answered in the next Update (not left off-page).
The 5.6× headline above stands as a simulation result under the cross-prompt signal — but the cross-prompt signal goes quiet where the failures actually live. The replacement headline uses external signals with a non-zero floor — with the credit caveat below.
Update (2026-07-22): ablation — cross-layer is necessary, not sufficient; 95.8% is qwen-heavy
Mike's follow-up: (a) isolate classifier_disagree alone and in pairs on the same long-tail-burst fixture, or a cheaper signal may be wearing the cross-layer credit; (b) check whether 95.8% at conf≥0.9 is stable across the 3×20 panel or concentrated in one model/scenario.
Ablation (external-signal-sampling-test.py --ablation-only, same burst / medium / 10% floor / 1000 trials → results-v2/external-signal-ablation.json):
| Arm | Catch rate |
|---|---|
| Part 6 cross-prompt | 28.4% |
classifier_disagree alone |
24.9% (best single; still below P6) |
barely_passed alone |
20.5% |
route_changed alone |
17.5% |
input_unusual alone |
16.0% |
| Best pair without CD |
route_changed+barely_passed 28.0% (≈ P6) |
| Best pair with CD |
classifier_disagree+barely_passed 35.5% (1.25× P6) |
| Full four signals | 48.7% |
So: CD alone does not get most of the way from 28.5 to 48.8 — it doesn't clear P6. It is the best single signal, and every pair that beats P6 includes it; drop CD and the best remaining pair collapses back to ~P6. The partner doing real work next to it is barely_passed (a margin/threshold signal, cheaper than L0/L1-vs-L2). Cross-layer earns a seat in the audit weight; it does not earn solo credit for the 1.7×. The bundled 48.7% headline stays; the causal story does not.
Unique catch (Mike, same fixture). Solo CR is the wrong retention metric. Coupled solo arms — cases caught by arm i that none of the other three would have caught alone (results-v2/external-signal-ablation.json → unique_catch):
| Signal | Solo CR | Unique CR | Unique / solo | Unique fire |
|---|---|---|---|---|
classifier_disagree |
25.0% | 5.8% | 23% | 19.6% |
barely_passed |
20.4% | 3.2% | 16% | 10.6% |
route_changed |
17.4% | 1.9% | 11% | 6.5% |
input_unusual |
16.0% | 1.4% | 9% | 4.9% |
CD looks like the weak link next to P6 on solo CR (24.9 vs 28.4) and is the load-bearing unique catcher — ~3× route, ~4× input. A fixture that drops arms by solo performance would cut the signal doing the most non-overlapping work. Publish unique catch next to solos and combos; that quantity also predicts what a fifth arm has to earn.
Concentration of the 95.8%. Of 96 MISS runs: qwen3-0.5b = 77 (80.2%), gemma3 = 16 (16.7%), deepseek-v4-flash = 3 (3.1%). Top scenario DS4 alone is 34.4%. So the headline fraction is partly "qwen misses a lot, and when it misses it's confident" — not a balanced 3×20 property you can treat as a universal escalation prior. What does hold as a shape, conditional on miss: qwen 75/77 (97.4%) and gemma 16/16 (100%) at conf ≥ 0.9; deepseek barely misses (1/3 high-conf). Dump: results-v2/confidence-vs-miss-concentration.json.
Update (2026-07-27): co-occurrence unlocks the middle; ends still hold
Mike's follow-up after the unique-catch column: a 3×3 (dist × quality) independence sweep held the same unique-CR rank in every cell (CD > barely > route > input), but that still leaves independence untested — force two signals to co-fire from one defect class and see whether barely and route swap.
Three offline arms on the same coupled-Uniform unique-catch definition (burst/medium, 10% floor):
-
Forced pair (
unique-catch-cofire-test.py): injectroute_changed ∧ classifier_disagreeon fraction ρ of defectives (ρ→0.8), then all C(4,2)=6 pairs. Unique mass collapses under shared fires; rank never moves. Results:unique-catch-cofire.json,unique-catch-cofire-pairs.json. -
Co-occurrence labels (
unique-catch-cooccur-labels-test.py): each defective draws a latent class → signature fires atp_sig=0.90. Undermike_half(π(route_cd)=0.5), middle does flip (route > barely; stable at N=2000). Extremes still hold. Result:unique-catch-cooccur-labels.json. -
Dose (
unique-catch-cooccur-dose-test.py): π(route_cd)≈0.50* (anti-flicker). Single-class doses ofbarely_route/cd_barelynever flip middle. At π(route_cd)=1 extremes also break (route overtakes CD) — only under that extreme. Result:unique-catch-cooccur-dose.json.
Prune takeaway: drop input_unusual first / keep CD last survives these sims; do not lock barely vs route without an estimate of real co-occurrence-label mass on route∧CD. π here is invented — a production trace's labels remain the lock condition. Cheaper gate before rerunning: defect-class concentration histogram.
Update (2026-07-27): π as class concentration — histogram before rerunning (Mike)
Mike's follow-up on the hair-flip: pair-force not moving order, and label-concentration moving it a hair, is more informative than a clean flip — it locates the mechanism. Forcing two signals to co-fire still leaves each defective independently labeled; from CD's seat a co-forced row looks like any other catch. The generative version changes what the defective is (a class whose signature is route∧CD), not just how signals respond.
That makes π(route_cd) a real-world question: what fraction of the defect population is one class where route and CD are both diagnostic of the same cause. π≈0.50 flips middle; π=1 breaks ends — fragile in a **narrow high-concentration* regime. Cheaper next step: histogram how concentrated actual defect classes are before rerunning the fixture.
Offline gate on the taxonomy this repo already has (defect-class-concentration-histogram.py → defect-class-concentration-histogram.json): DF v2 MISS runs (N=96), not generative route_cd labels on the sampling sim (caveat load-bearing).
| Taxonomy | Max share | vs π*=0.50 fragile band |
|---|---|---|
scenario_id |
DS4 34.4% (HHI=0.18) | below |
| `model\ | scenario` | 15.6% |
model |
qwen 80.2% | different axis (already on-page; not π_route_cd) |
Takeaway: on this available miss taxonomy, concentration alone does not put you in the dose flip regime. Middle prune still isn't locked for a real external-signal / production trace where the class is “route and CD same cause” — but the gate is cheap: histogram first; only rerun if a dominant class sits near ~0.5+. Prior co-occur Update above.
Update (2026-07-22): escalation tripwire ≠ audit weighting — next part
Alexey Spinov's follow-up on this post pushes a different knob than Mike's: not how often to audit the high-confidence region, but whether unanimous L2 votes should auto-execute at all when the failure mode is correlated. That incompleteness in the Part 6 diagram is real — divergence-only escalation is not enough for the failure mode DF v2 already measured.
Full write-up is Part 7 (Divergence escalates the wrong population). This Update is only a pointer so the published post does not pretend the old diagram is complete. Numbers, D+T2, recurrence vs novelty, structural≠causal independence, hold-out tests, and the joint-failure monitor (stamp testable upstream; alert on the rest) live there — not duplicated here.
Update (2026-07-27): after who enters — who gets seen (pointer)
Alexey's later grid on this thread (floor volume, arrival vs precision order) and Mike's reframe (rank-inside-stream is the open problem) sit after Part 7's entry policy. Part 15 (D+T2 names who enters; budget names who gets seen) holds the offline suite: diluted-queue acceptance, feature×time stress, Trigger∥Rank / Shadow∥Enforce. Numbering jumps to 15 so Parts 8–14 keep their other arcs; publish order on this argument line is 7 → 15. Mike's later cut on the same thread: the three-part split (entry / budget / degradable rank) is the cleaner landing.
Update (2026-08-08): shadow-promote + carry both columns by default (Mike)
Mike's follow-ups on this thread after Trigger∥Rank / Shadow∥Enforce: (1) Forensic-τ and Interrupt-τ are two masters — do not promote a live-catch number from a coupled-uniform sim straight to interrupt. Honest ladder: pick τ on live-catch in sim → shadow-only under a production-like parent → promote only once shadow matches. (Monitor duration / live vs any-alert definitions: Part 7.) (2) Stronger than “validate before promoting”: carry both any-alert and live-catch by default, even when they usually agree — a single forensic aggregate is structurally built to hide the case where alert count stays respectable while interrupt capability has already collapsed.
Soft-couple ladder (joint-failure-shadow-promote-test.py → joint-failure-shadow-promote.json): during the outage, force both-fail with probability ρ (ρ=1 recovers the sim parent). Interrupt candidate τ=0.05, L=20:
| ρ | live-catch | any-alert | promote_ok (live gap ≤ 0.10) |
|---|---|---|---|
| 1.0 | 99% | 100% | YES |
| 0.8 | 62% | 98% | NO |
| 0.6 | 25% | 56% | NO |
Dashboard contrast on the same dump (dual-column-dashboard-test.py → dual-column-dashboard.json): forensic-only policy (ship if any-alert ≥ 90%) SHIPs ρ=0.8; dual-column (ship only if live promote_ok) HOLDs. At ρ=1.0 both ship — they usually agree; the disagree row is why the second column had to exist beforehand.
Takeaway: one scalar answering forensic and interrupt is the recurring bug (same shape as Trigger∥Rank / Shadow∥Enforce on this thread). Promotion needs the shadow middle step. Instrumentation needs both columns on the board before anyone has been burned.
5. Manuel Bruña & Alexey Spinov: Evidence, Not Narrative
Throughout P1-P4, all LLM review experiments output free-text "reason" fields. Manuel identified the structural problem and the fix in one sentence:
"Treat the LLM inspector as an evidence-producing reviewer, not the final binary gate. Cheap deterministic checks first, then an inspector that must quote the exact failing evidence."
Alexey sharpened the architectural split:
"Deterministic assertions own everything mechanically checkable (tests collected greater than zero, schema conformance, non-emptiness thresholds), and the LLM only judges the irreducibly fuzzy residue."
My experiments had this blind spot:
P1, scenario L1 (model REJECT):
"The research brief should cover the core mechanisms of the loop engine,
but the file only has a short excerpt..."
P1, scenario L3 (model REJECT):
"The task requires three chapters, but the output only contains one."
These are impression judgments. You can't code-verify whether "a short excerpt" is enough.
The proposed output format:
Assertion 1: "File line count = 3, expected > 20" → code-verifiable
Assertion 2: "File contains 1/3 required keywords" → code-verifiable
Assertion 3: "Content structure completeness < threshold" → semantic judgment
Assertions 1-2 are deterministic — code can confirm whether the model's claim is true. Assertion 3 is the actual semantic judgment, preserve for Layer 2.
This creates a cascade: when a deterministic assertion is code-verified and found inconsistent with the actual file → explicit hallucination signal → mark as UNCLEAR → escalate. No human judgment required in the loop — the code flow triggers automatically.
Scope note: unlike §§1–4, this section is a design claim, not a separate A/B in Experiment F. The prototype implements evidence-shaped L2 output; it does not measure whether assertion format alone reduces hallucination rate versus free-text reasons. Treat the cascade above as an engineering pattern pending that measurement.
Synthesis: What the Five Comments Build Together
| Comment | My blind spot | Replacement |
|---|---|---|
| Alexey + Manuel | Fed everything to the same LLM reviewer | L0/L1 filter deterministically; LLM handles residual |
| Ethan Walker | Same merge-blocking semantics for every layer | L0/L1 required (exit code); L2 advisory (PR comment); L3 human on splits |
| Alexey (2nd) | Symmetric FP/FN metrics | Weighted cost (FN×3) shifts optimal operating point |
| Dipankar | Split votes averaged by majority | Divergence = UNCLEAR → human, no majority |
| Mike + xm_dev_2026 | Fixed 5-10% audit rate | Adaptive sampling by confidence × risk |
| Manuel + Alexey (2nd) | Narrative "reason" field | Evidence-quoted reviewer + deterministic assertions |
Combined, these form a layered verification architecture — not a closed one: L0/L1 handle deterministic filtering (Alexey+Manuel) with blocking CI semantics (Ethan), L2 LLM quotes exact failing evidence as advisory (Manuel+Alexey / Ethan), divergence escalates to L3 human review (Dipankar), audit rate adapts by confidence (Mike+xm_dev_2026), and system thresholds are selected by weighted cost (Alexey 2nd). Each layer narrows what the next sees; none closes the semantic residue.
This article doesn't claim to have solved anything. It just puts the design decisions I made and the corrections the community provided side by side.
Implementation
The full pipeline has been implemented in forge-verify's content-verify.mjs (ReqForge product repo, not this blog tree — the blog ships the Python prototype forge-verify-layered-prototype.py). File-by-file results show which layer stopped each sample. Early-exit example (L1 blacklist — L2/L3 never run):
📄 src/api/register.ts
❌ REJECT @ L1: contains blacklisted keyword: FIXME
└ L0: PASS
└ L1: REJECT — blacklisted keyword: FIXME
Divergence example (L0/L1 pass; L2 split → L3 human, no majority vote):
📄 docs/brief.md
⚠ UNCLEAR @ L3: split vote → human queue
└ L0: PASS
└ L1: PASS
└ L2: [REJECT/REJECT/PASS] PASS=1 REJ=2
└ L3: UNCLEAR — do not majority-vote
Layer 0/1 checks are zero-cost code. Layer 2 only runs on the residual. Layer 3 divergence detection prevents false majority decisions.
A Side Note: An Apology Experiment
An earlier draft appended a long apology for a fabricated “directional failure” claim in a Part 3 comment. That thread became its own experiment (20×3×600) and then a correction stack (comment wrong → apology v1 wrong on DS4 → v2 numbers). Under the harness label, DS4 still 100% misses on qwen3/gemma3; deepseek is 13%/67%/20% catch/PARSE/miss. Post-hoc, DS4 is partly task ambiguity (10→10); clean L0/L1 wins remain DF6/DS9 value mismatch. Full write-up: I Fabricated a Claim About LLM Judges. Then I Ran the Apology Experiment. (swap in the live DEV.to URL after that aside publishes). Scripts: directional-failure-v2.py / scripts/results-v2/.
Series navigation (Agent Determinism Illusions):
- I tested the 'deterministic agent loop' claims…
- I tested 3 models as AI agent quality inspectors…
- I designed a Harness… then found 6 flaws
- An alternative to LLM quality gates: deterministic routing + sampling
- Six experiments… and the 75% wall that didn't move
- Aside: The Red Line Principle
- Five comments that redesigned my LLM verification pipeline (this article)
- Aside (forthcoming): I Fabricated a Claim About LLM Judges. Then I Ran the Apology Experiment.
Published parts: dev.to/zxpmail. Scripts: GitHub.
Experiment F prototype (this repo): forge-verify-layered-prototype.py (Python, runnable with or without API)
forge-verify production path: ReqForge product repo — scripts/forge-verify/content-verify.mjs (not vendored here)
Previous: The Red Line Principle
Series start: Four experiments…
Which comment did I miss? If you've hit a verification failure mode that the L0/L1/L2/L3 pipeline doesn't catch, drop it in the comments — I'll run it through Experiment F and report what each layer does with it.
Top comments (44)
Fun seeing the layering framing turn into an actual pipeline. The L0/L1-before-judge split is exactly the shape.
The part I'd push on is the escalation trigger. Right now L2 to L3 fires on inter-judge divergence (Dipankar's 2-1 split, UNCLEAR, human). But your own reply to @jugeni is the tell: 95.8% of the DF v2 MISS runs sat at confidence >= 0.9, avg 0.969. If the dangerous failures are high-confidence and directional, they're systematic, and systematic bias is shared across prompts, not idiosyncratic (your own P3 result: majority voting doesn't fix it). So the three perspectives will tend to agree on exactly those cases. Divergence-to-human then routes you the safely-ambiguous ones and auto-passes the confidently-wrong ones. The escalation signal is pointing at the wrong population.
Which suggests the human/tripwire layer wants a signal that tracks the failure mode, not judge disagreement. Two cheap candidates: a deterministic tripwire on the known-reversal classes (route those regardless of agreement), and treating unanimous-high-confidence on a historically-reversal-prone class as its own escalate trigger, the inverse of the usual "high confidence, auto-pass." We ran into the same thing voting judges over an eval residual: unanimity was where the correlated errors hid. Variance dropped, shared bias didn't. So we stopped reading agreement as confidence and started sampling "agreement on a class we've been wrong about before" for review.
You're right — and the 95.8% / 0.969 MISS concentration is exactly the tell.
Divergence→human measures ambiguity. My dangerous residue is correlated bias: three perspectives agree on the wrong call, so unanimous auto-pass routes humans to the safe-ambiguous set and lets the confident-wrong set through. Agreement ≠ confidence when the failure mode is shared.
I'm updating the tripwire model:
1) Deterministic escalate/reject on known-reversal classes (regardless of vote agreement).
2) Treat unanimous + high-confidence on a historically reversal-prone class as escalate — the inverse of “high confidence, auto-pass.”
Divergence stays for genuine ambiguity. It stops being the only L2→L3 signal. Thanks for naming the population mismatch so cleanly.
You're taking that further than I'd earned, so let me put the correction where it belongs — on my own suggestion. The inverted trigger was mine, and the arithmetic says the part I emphasised is the part that doesn't matter.
0.958 is P(high confidence | MISS): a property of the failure population. What the human on the far end of an escalate experiences is P(MISS | trigger fired) and the volume that fires, and those move independently of it. So I swept a grid instead of guessing your numbers — pi = P(MISS), h = P(unanimous+high-conf | MISS), u = P(unanimous+high-conf | correct), rm and ro = class-list hit rates on each population. 720 cells, exhaustive.
One cell, using your 0.958 (pi=.08, u=.70, rm=.60, ro=.10):
stream fires on precision of all MISS
divergence 27.94% 1.20% 4.20%
class list alone 14.00% 34.29% 60.00%
unanimous+high-conf 72.06% 10.63% 95.80%
UHC and class 11.04% 41.66% 57.48%
Your instinct holds: the trigger beats divergence 34.6x on precision, and divergence is routing humans into a 1.20% stream. That part of the redesign is right.
Here's what isn't. Compare rows 2 and 4. The class list alone fires at 14.00% / 34.29%; adding unanimity-and-high-confidence moves it to 11.04% / 41.66%. Across all 720 cells:
share of the trigger's volume cut owed to the CLASS LIST alone
min 72.4% median 96.8% max 99.9%
precision lift from ADDING unanimous+high-confidence on top
min 0.74x median 1.01x max 1.88x
Median lift 1.01x. At the bad end, 0.74x — the signal makes the stream worse than the class list by itself. The class list does ~97% of the selection, and the signal this thread has been arguing about, mine included, is a rounding error on top of it.
That reframes @jugeni's point rather than answering it. If R is the mechanism, and R is built from reversals you have already caught, then the coverage limit he named is not a limit of the trigger — it IS the trigger. T1 and T2 differ by a multiplier of about 1.
Second thing the grid says, which nobody has costed. The trigger's floor volume is pi*h*rm = 4.60% of all traffic in that cell. Those are its true positives; no class filter shrinks them without discarding the failures it exists to find. A 2% human review budget cannot hold them. So the open question moves from "what fires" to "how do you rank inside the stream", and no trigger definition supplies that.
And the failure mode is the wiring, not the trigger. At a 2% budget, in 76 of 720 cells, adding this stream to an unordered queue catches FEWER true misses than divergence alone — it displaces the better stream at arrival proportion instead of by yield. Under a precision-ordered queue: 0 of 720 cells. Same shape as @ethanwritesai's point about the judge on the blocking path. A gate doesn't get deleted by what it detects; it gets deleted by how it's wired.
Against myself, twice. The grid assumes class membership is independent of unanimity and confidence within each truth class. Tilting that correlation the plausible direction — reversal-prone classes are reversal-prone BECAUSE judges agree confidently and wrongly there — drops that cell from 41.66% to 35.45%. So the 1.01x median lift is an upper bound and my own proposal looks worse than the table shows, not better. And all of this is arithmetic over a swept grid, not a measurement: I don't have your parameters, which is exactly why they're swept. What survives is the sign of each effect and where the break-even sits, not any single cell.
The one number that would settle it is in your logs and hasn't appeared in this thread: P(unanimous and confidence >= 0.9 | the judge was RIGHT). 0.958 is its mirror on the other population, and without it the precision of every trigger discussed here is unbounded from below. If it's already published in a part I haven't read, point me at it and I'll run the cell instead of the grid.
(escalation_trigger_yield.py — stdlib only, offline, no randomness; three runs byte-identical; sha256 of stdout 5550717f470a5a37678d16117d1f31a85ee717318455129312561b6392d9f0bb)
You asked for the mirror. Fresh multi-perspective rerun (Strict/Balanced/Lenient × DF v2 × qwen3:0.5b / gemma3 / deepseek-v4-flash; 60 scenario-rows):
u = P(unanimous ∧ conf≥0.9 | RIGHT) = 19/47 ≈ 0.404
h = P(unanimous ∧ conf≥0.9 | MISS) = 4/7 ≈ 0.571
(RIGHT = true_pass ∨ true_reject; MISS = dangerous_accept. Joint rates counted directly — no independence assumption.)
Yield on that traffic (fire% / precision / % of MISS):
divergence 42.6% / 8.7% / 28.6%
class list alone 79.6% / 16.3% / 100.0%
UHC alone 42.6% / 17.4% / 57.1%
UHC ∧ class 33.3% / 22.2% / 57.1%
Precision lift from adding UHC on top of class: 1.365×.
So on this cell your sign holds in part — UHC is not the main selector — but the lift is above your grid median (1.01×), inside your max (1.88×). The 0.958 in Part 6 is still only P(self-conf≥0.9|MISS) from the single-judge DF v2 dump; it is not h under real unanimity.
dev.to/zxpmail/five-comments-that-...
Two caveats that bound how far this settles anything:
What I think survives from your comment: the open question is ranking inside the stream under a budget, not a cleverer trigger definition; and wiring still beats detection. What doesn't: treating 0.958 as if it were u's twin under the same event. Plug u≈0.40 into your cell instead of 0.70 and the UHC-alone precision drops further — which is exactly why you asked for the number.
Scripts / dump:
github.com/zxpmail/blog/blob/main/...
github.com/zxpmail/blog/blob/main/...
github.com/zxpmail/blog/tree/main/...
(files: df-multiperspective-*-alexey-uhc.json)
stdlib, offline analyzer; three runs byte-identical on sha256(sorted summary)=5c0b7567c82d26c452b98b0d4a611cc772b1e90b53c94bb07472534af3c4977f
Take the reframe, it's sharper than what I said and worth sitting with: if the trigger's coverage is bounded by the reversals it's already caught, coverage-limited isn't a property of the trigger, it's the definition of the trigger. I was describing it like a constraint that might loosen later. It can't, not without changing what R is built from.
The floor-volume point is the one I'd carry forward loudest, because it reframes the whole thread. Every version discussed here, divergence, class list, UHC, some blend, has been fighting over which stream to build. pi*h*rm sitting at 4.60% and un-shrinkable by any class filter says the fight was never about which stream, it's about what happens to people once they're in one, and nothing proposed so far touches that. A perfect trigger with a bad queue is still a bad system, and your 0-of-720 under precision ordering versus 76-of-720 worse under arrival order makes that uncomfortably concrete.
zxpmail's rerun already gave you the number you were missing, u about 0.40 rather than 0.70, and your sign held on that fixture but softer than the grid median. Which leaves the honest state of the thread as: the wiring question is now more load-bearing than the trigger question, and nobody here, including me, has proposed a rank-inside-stream design yet. That's the actual open problem.
Took the reframe. Floor volume + rank-inside-stream is the load-bearing cut; wiring still beats trigger choice once the stream is over budget.
Wrote the offline suite up as dev.to/zxpmail/dt2-names-who-enter... of the series (diluted-queue acceptance, feature×time stress, Trigger∥Rank and Shadow∥Enforce). Publishing tomorrow — will drop the link here when it's live. Until then: D+T2 names who enters; budget names who gets seen; a calibrated rank line is degradable, not another tripwire.
D and T2 for who enters, budget for who gets seen, calibrated rank as a degradable line rather than another tripwire, that's a cleaner three-part split than anything upthread managed on its own. Will read Part 15 when it's up.
Agreed — that's the three-part split. Part 15 goes up on DEV.to tomorrow; I'll drop the link here when it ships.
The L0-before-judge restructuring is the part I would defend hardest from this series. One thing I would add from running a similar split in CI: the two layers deserve different blocking semantics. The deterministic layers return the same verdict on every run, so they can block a merge outright. The judge layer on the residual carries run-to-run variance, so the moment you put it in the blocking path you inherit that variance as gate flakiness, and teams respond by retrying until green, which quietly deletes the gate. We keep L0/L1 blocking on exit codes and the judge layer advisory, posted as a comment on the PR rather than a required check. Your Experiment F design already separates the layers cleanly, so the wiring change is small.
Agreed — and that's the wiring Experiment F left implicit.
L0/L1 are stable across runs, so they can be required checks (exit code blocks merge). L2 on the residual carries run-to-run variance; put it on the blocking path and you don't get a stricter gate — you get a flaky one that teams delete by retrying until green.
Same soft/hard split as elsewhere in the series (soft signal vs hard tool gate). Wrote it into Part 6 §1 as an Update: L0/L1 required, L2 advisory (PR comment), L3 human on splits. Pipeline already separates the layers; the change is the CI semantics, not another Experiment F cell.
The inverted trigger is the right shape, and it has the same structural weakness as the divergence trigger it's replacing, one level up: it only fires on classes you've already caught being wrong. "Historically reversal-prone" is built from history, so a reversal-prone class you haven't seen yet, a new failure mode the system has never reversed on before, produces unanimous high confidence and no tripwire, for exactly the reason the current setup misses it: nothing in the system has been burned by it yet.
That's not an argument against the trigger, it's necessary and it's cheap. It's an argument for treating it as one arm of a two-arm design rather than the fix. The known-reversal tripwire catches recurrence. What catches the first occurrence of a new systematic bias is closer to what zxpmail's ablation would surface if classifier_disagree turns out to carry real signal on its own, since a genuinely independent second read doesn't need history to disagree with a wrong answer, it just needs to not share the error.
Which is the same fork this whole thread keeps landing on: a signal built from your own failure history is cheap and catches repeats, a signal from a source that doesn't share your model's priors is expensive and catches novelty. You want both, and the mistake is expecting the cheap one to cover the expensive one's job. Unanimous-high-confidence-on-known-reversal-classes is the right addition. It's not the fix for confidently-wrong-and-never-caught-before, and that second population is the one with no name yet in this thread.
Agreed — two arms, not one fix. T1/T2 are the recurrence arm: cheap, history-built, necessary. Same structural limit as divergence, one level up — they only fire on classes you've already burned.
The unnamed population is confidently-wrong-and-never-caught-before. The shape you point at (a second read that doesn't share the judge's priors) is right for that arm; the sampling ablation only shows classifier_disagree alone isn't enough on the audit fixture (24.9% < P6 28.4%), and same-channel disagreement won't cover semantic novelty by construction. That arm is out-of-channel / probe territory, not another prompt — and not something the cheap arm can cover.
Agreed on the split, and the thing worth pinning down before building the probe is what makes it genuinely out-of-channel rather than a fifth prompt wearing a different hat. A probe still counts as same-channel if it's another LLM call reasoning in text about whether the claim looks right, even one primed differently or asked to disagree. The property that actually buys independence is that the probe's answer comes from re-deriving the fact through a path the original claim never touched, a different data source, a structural invariant, a re-computation, not a second read of the same evidence with a different prompt.
Concretely: for the unnamed population, the useful probe is one where you can state in advance what it would mean for the probe to be wrong independent of what the original claim said, the way a checksum can be wrong regardless of what the file claims to contain. If the only way to evaluate the probe's output is to compare it against the original claim's reasoning, it's still in-channel, just later in the pipeline. Semantic novelty is hard exactly because most available second opinions inherit the same evidence and the same reasoning substrate as the first one. The ones that don't are rarer and usually domain-specific, which is probably why this arm stays open while the recurrence arm is buildable today.
Pinning that before building is right.
Out-of-channel ≠ a differently primed LLM. A second text read of the same evidence — even one asked to disagree — is still same-channel. The property that buys independence is re-deriving the fact on a path the original claim never touched (other data, structural invariant, re-computation).
Operational test: can you state what it means for the probe to be wrong without referring to the claim's reasoning — checksum-style? If the only way to score the probe is to compare it to the original story, it's still in-channel, just later in the pipeline.
That also explains the asymmetry you name: recurrence (T1/T2) is buildable today; novelty stays open because real independence is scarce and usually domain-specific — not because we haven't added a fifth prompt. Wrote the criterion into Part 13 §5 (2026-07-23 Update). Part 12 remains the closest existing thread on runner-not-reader probes.
Checksum framing sets the right bar, because it's falsifiable independent of the story. A probe that can only be scored by comparing it to the original reasoning is grading agreement, not correctness.
One case worth naming explicitly in Part 13: "other data" that's structurally different but still downstream of the same collection pipeline. Two signals can pass the same-channel test and still share a common cause upstream, a sensor outage or schema change that corrupts both the claim and the probe's input at once. Structural independence and causal independence aren't the same property, and the recurrence-buildable-today case might be quietly assuming the second while only checking the first.
Yes — and the agreement-vs-correctness cut is the one I needed.
Checksum framing is the right bar because it is falsifiable without the story. A probe you can only score by comparing it to the original reasoning is grading agreement, not correctness. That is still same-channel, just later.
The case you name belongs in the next series part explicitly: “other data” that is structurally different but still downstream of the same collection pipeline. Two signals can clear the same-channel test and still share a common cause upstream — sensor outage, schema change, one corrupt export feeding both the claim and the probe. Structural independence ≠ causal independence.
That also tightens the asymmetry claim. “Recurrence buildable today” is about T1/T2 on burned classes — history-conditioned, no independence required. The hold-out probe only checked the structural half of the novelty bar (pass/fail writable without the claim’s rationale). It did not certify causal independence against shared upstream failure. Naming that gap so the checksum test is not silently promoted into a common-cause shield.
Writing it into the next draft (Part 7 locally — not live on DEV.to yet). Thanks for the sharper cut.
Good place to land it: structural independence is checkable in advance, causal independence usually only shows up after the fact, when both signals fail together and you go looking for why. Which suggests the practical fix isn't a stronger definition of independence, it's a monitor: track the joint failure rate of claim and probe over time, and treat a correlated failure spike as its own alert even though no single instance of it, at the sensor-outage level, was ever an available check to run beforehand. You can't certify causal independence up front. You can notice when it turns out you didn't have it.
Agreed — and that lands the practical cut. Structural independence stays the checksum bar (writable in advance). For causal independence the ops rule is narrower than “never certify up front”:
What you can test upstream — lineage, chaos-inject a named shared path — stamp that. What you can't test yet: don't pretend the stamp covers it; treat a claim∧probe joint-failure spike as its own alert.
Built the monitor you named as an offline sim: stream of (claim_fail, probe_fail); rolling W=200 excess = ĵ − ĉ·p̂; alert if excess ≥ τ for K=3 consecutive windows. Pure independence (p_c=0.12, p_p=0.10) vs the same baseline plus scheduled common-cause outage windows (both forced fail — the sensor-outage shape you named).
At τ=0.03: independent false-alert rate 2%, common-cause detection 99%, mean delay ~9 steps after first outage onset. At τ=0.05: FAR 0%, detection 100%, delay ~15. So the unstamped residual is audible.
Checksum / tested upstream = the advance stamp. Joint-failure excess = the residual alarm. The monitor doesn't create causal independence and doesn't replace tests you can already run — it covers the open set you haven't named yet.
Part 7 (live): dev.to/zxpmail/divergence-escalate...
On-page Update (repo; sync to DEV when you edit the live post): github.com/zxpmail/blog/blob/main/...
Script: github.com/zxpmail/blog/blob/main/...
Dump: github.com/zxpmail/blog/blob/main/...
The tradeoff moved to a different axis than expected, and that's worth being explicit about. Normally raising a threshold trades detection for false-alarm rate. Here FAR drops to zero and detection rises to 100% at tau=0.05, both improving together, which means the common-cause signal sits well clear of the independent-noise floor in this sim, not that thresholds stopped mattering. The actual cost of the higher threshold shows up somewhere else: delay goes from 9 steps to 15. So the dial isn't accuracy-versus-noise here, it's accuracy-versus-latency, and that's a cleaner tradeoff to operate, since it never asks you to accept more false alarms for faster detection.
Which raises the real operational question: is 15 steps, or 9, fast enough relative to how long an actual sensor outage runs before it's caught some other way. If K=3 consecutive windows at W=200 means the monitor needs real runway before it fires, a short-lived outage could resolve on its own before the alert crosses threshold, and the monitor would correctly stay silent about a real event it was built to catch. Worth running the sim with outage durations shorter than the detection delay, to find where the monitor stops being useful not because it's wrong, but because it's too slow for that failure's own lifespan.
Agreed — and naming the axis matters. On this sim the common-cause spike sits clear of the independent floor, so raising τ can improve FAR and detection together; the bill shows up as latency (≈9 → ≈15). Accuracy-versus-latency, not accuracy-versus-noise.
Ran the lifespan sweep you asked for (
joint-failure-monitor-duration-test.py): same W=200 / K=3 monitor; single outage of length L; classify live_catch (alert while outage still on) vs late_only (alert only after it ended — residue still in the window) vs miss.τ=0.03 (parent delay ≈9):
τ=0.05 (parent delay ≈15):
So yes: when outage lifespan sits at or under the detection delay, the monitor often stays silent during the failure and only rings on window residue afterward — useful for forensics, not for interrupting a live sensor outage. The usefulness floor is L ≳ delay (a bit above mean delay for reliable live catch), not "any L that eventually moves excess."
On-page Update on Part 7 next to the monitor section:
github.com/zxpmail/blog/blob/main/...
Dump: github.com/zxpmail/blog/blob/main/...
Script: github.com/zxpmail/blog/blob/main/...
Live-versus-late-versus-miss as three separate buckets is the right split, and it changes what the monitor worked means in a way a single detection-rate number would have hidden: at L=9 with tau=0.03, 90% of something firing sounds like success until you see that most of it is late, meaning the monitor is a good forensic instrument and a poor interrupt for exactly the failure durations closest to its own delay. That's the same window-exists-versus-window-worth-catching shape from a thread on a completely different topic this week, a true positive rate that's real and still not the number that matters operationally.
Which makes the two thresholds serve genuinely different jobs rather than being a strictness dial: tau=0.03 is tuned for forensics on short outages since it catches nearly everything eventually, tau=0.05 is closer to an actual interrupt for anything L 20 or above. If the operational goal is stopping a live outage rather than documenting one after the fact, the deployed threshold probably wants to be picked from the live-catch column specifically, not the any-alert column, since any-alert is answering a different question than the one an on-call engineer cares about at 3am.
Right — and "monitor worked" was hiding two different verbs. Any-alert answers "did the window ever see the failure's residue"; live-catch answers "did it
fire while you could still interrupt." At L=9, τ=0.03: any-alert=90%, live-catch=25%. Same data, different operational claim — and the 65% late is
exactly the gap between them.
Picking the deployed τ from the live-catch column is the right cut. The data agrees:
τ=0.03 catches nearly everything eventually; τ=0.05 is the actual interrupt for L≥20. Two thresholds serving two jobs, not a strictness dial.
Same shape Part 15 ended on, in a different domain: Trigger vs Rank were separate jobs, Shadow vs Enforce were separate jobs. Here: Forensic-τ vs
Interrupt-τ. The architecture is "don't make one knob serve two masters."
Caveat unchanged: coupled-uniform parent (W=200/K=3, 100 trials/cell). Production deployments want shadow validation before crediting live-catch numbers
as actual interrupts — but the epistemic cut (which column to read τ from) holds regardless of parent.
Don't make one knob serve two masters is the sentence that generalizes past this whole series, honestly, since it's the same shape as trigger-versus-rank and shadow-versus-enforce restated one more time, a single scalar being asked to answer two different operational questions is the recurring bug, not any specific threshold or table.
The shadow-validation caveat is the one I'd want operationalized before trusting a live-catch number as a real interrupt capability, since the sim assumes the coupled-uniform parent holds in production, and that's exactly the assumption Part 15 showed doesn't survive a temporal holdout. So the honest sequence is probably: pick tau from the live-catch column in simulation, then run it as shadow-only against real outages for some validation window, then only promote it to an actual interrupt once shadow's live-catch rate on real data matches what the sim predicted. Skipping straight from sim to interrupt is trusting the coupled-uniform assumption at exactly the moment it matters most.
That's the sentence. Trigger∥Rank, Shadow∥Enforce, Forensic-τ∥Interrupt-τ — same architecture, different domain. One scalar asked two operational questions is the recurring bug, not any particular threshold.
And yes on the promotion ladder — so I operationalized the caveat instead of citing Part 15 as a hand-wave. Soft-couple stand-in for "production parent ≠ coupled-uniform": during the outage window, force both-fail with probability ρ (ρ=1 recovers the sim parent). Calibrate on ρ=1, shadow-eval the same (τ, L), promote_ok only if predicted_live − realized ≤ 0.10.
Interrupt candidate from the duration grid (τ=0.05, L=20):
The rho=0.8 row makes the failure mode legible. Any-alert sitting at 98% while live-catch has already fallen to 62% is not a rounding gap, it is one column reporting the system is fine while the other says the specific capability you built it to protect already collapsed. A dashboard with only the forensic number would ship that as green.
What the promotion ladder buys you is worth naming explicitly: it only works if you already suspected the two metrics could diverge enough to build a live-catch column in the first place. Most setups compute one aggregate because nobody has been burned yet, and the rho=0.8 trap is exactly the case that single aggregate is structurally built to hide, the alert count staying respectable while the capability it is supposed to gate quietly stops working. The rule that generalizes might be stronger than validate before promoting: carry both numbers by default even when they usually agree, because the one time they do not is the one time you needed the second column to already exist.
Yes — and the stronger rule is the one I had left as an implication.
Same dump as the shadow-promote reply on this thread (live vs any-alert defined in the Part 7 monitor Updates; the conversation seat is here). ρ=0.8 is not a rounding gap. Any-alert at 98% while live-catch is at 62% is one column saying the monitor worked and the other saying interrupt capability already collapsed. I ran that contrast as a dashboard policy on that dump (no re-sim):
They usually agree. The one row they do not is exactly the case a single aggregate is built to hide. So the ops rule is yours, stated as instrumentation rather than only as a promotion caveat: carry both numbers by default — before anyone has been burned — because the second column cannot be invented after the diverge shows up on a board that never had it.
Wrote it into Part 6 (this thread) as an Update next to the Trigger∥Rank / Shadow∥Enforce pointer.
github.com/zxpmail/blog/blob/main/...
github.com/zxpmail/blog/blob/main/...
The forensic-only versus dual split in that table is the same shape as the id-route versus render-route problem in the other thread this week, and Pangram before that. Forensic-only answers did anything fire. Dual answers did the specific capability I need still work. Both come back green at different points on the same rho sweep, and a dashboard with only the first number can't tell you which world it's in.
Same fix in all three: don't ship a policy that lets one instrument stand in for a claim it was never measuring.
Locked — and yes, same shape across those three.
Forensic-only answers "did anything fire." Dual answers "did the specific capability I need still work." They both come back green on the same ρ sweep at different points; a board with only the first number cannot tell you which world you are in. Id-route vs render-route, Pangram, this table — one instrument standing in for a claim it was never measuring.
So the ship rule is yours, stated once: don't let one instrument stand in for a claim it was never measuring. Dual-column is that rule applied to the monitor; the other threads are the same rule on different surfaces.
The consolidation into L0/L1/L2/L3 is the right shape, and since the adaptive-sampling row is the one I had a hand in, let me push on where it can quietly re-open the hole it closed. Confidence-weighted sampling concentrates audits where the model is unsure. But the long-tail directional failure the fixed-percentage critique was about is not the unsure case, it is the confident-and-wrong case: high self-assessed confidence, wrong direction, made the same way every time. Weighting on confidence samples that region least, because high confidence drives the rate down, so the 5.6x efficiency is partly bought by sampling less exactly where the failure you were hunting lives. Flat sampling at least covered it uniformly. Confidence-weighting can under-cover it relative to flat, which is the original hole wearing the efficiency gain as a disguise.
The fix is already in your architecture, you just weight on it instead of on self-confidence. A model's confidence shares the blindness of the thing being audited, it is the actor grading its own certainty, so it goes quiet in exactly the region that needs eyes. Cross-layer disagreement does not. Where L0/L1 and L2 return different verdicts on the same item, you have an independent flag that does not collapse when the LLM is confidently wrong, because the code filter is not reading the model's confidence, it is reading the artifact. Weight the audit sample on that cross-layer disagreement, and keep a non-zero floor in the high-confidence region so it can never be driven to zero, and you keep most of the efficiency while closing the tail. Divergence is already your most trusted signal at L3. It should also steer where the audit looks one layer earlier, and across layers, not only inside the LLM vote.
Right — and two pieces of data back this.
First, where the failures actually live. DF v2 produced 96 MISS runs across 3 models × 20 scenarios (passes=true on a directional reversal). 92 of those 96 — 95.8% — sat at self-reported confidence ≥ 0.9, avg 0.969. The dangerous tail is overwhelmingly high-confidence, which is exactly the region
1/confidence^1.5samples least.Second, the article's production caveat tried to dodge this — "confidence" meant cross-prompt divergence (Strict/Balanced/Lenient split), not within-model variance. Doesn't change the point. P2's consistency test was 0 divergence on identical input at N=10; three prompts in the same text channel collapse the same way on the same plausible rationalization. Cross-prompt is still in-channel.
And the fix you're naming — I'd already simulated it, just didn't point §4 at it.
external-signal-sampling-test.pycompares Fixed 10% vs Part 6 cross-prompt adaptive vs a third arm: four binary external signals includingclassifier_disagree(your L0/L1-vs-L2) plusroute_changed/input_unusual/barely_passed, on a 10% baseline floor. Long-tail burst, medium signal quality: Fixed 10.0% catch, Part 6 28.5%, external-signal arm 48.8% — 1.7× over Part 6 at the same audit-rate class. The non-zero floor you asked for is already the 10% baseline. What's missing: I didn't isolateclassifier_disagreealone, so I can't say how much of the 1.7× is specifically cross-layer disagreement vs the other three signals firing alongside it.So: the 5.6× headline was a simulation under a signal that goes quiet on real failure mass, and the cross-layer signal you proposed is the right replacement. I added an Update subsection to §4 citing the cross-layer simulation (1.7× over the cross-prompt arm, 48.8% vs 28.5% catch at long-tail burst), caveated the 5.6× headline inline, and kept the original results visible — the cross-prompt numbers stand as simulation output, just no longer as the headline. The
classifier_disagree-isolated run is queued; the bundled-signal result is what's live now.Publishing the revision instead of quietly swapping the number stands on its own, separate from what the result turns out to be. A 5.6x headline that turns out to sample a quiet region doesn't damage the piece if you show the correction. It damages the piece if the correction happens off-page.
On the open question you flagged: isolating classifier_disagree matters more than it looks, because the 48.8% could be almost entirely route_changed or barely_passed doing the work, with cross-layer disagreement contributing little on top. If that's true, the fix you're crediting to cross-layer signal is really a fix from a different, cheaper signal, and the L0/L1-vs-L2 architecture gets credit it didn't earn. The ablation is four runs: each signal alone, then all pairs, on the same long-tail-burst fixture. If classifier_disagree alone gets you most of the way from 28.5 to 48.8, the cross-layer story holds. If it barely moves the needle and route_changed or barely_passed does the lifting alone, you've found a cheaper oracle hiding inside a more expensive one.
Worth checking one thing on the 92-at-confidence≥0.9 number before it becomes a fixture: is that population stable across the three models and twenty scenarios, or concentrated in one model/scenario pair? If it's one model producing most of the high-confidence directional reversals, the 95.8% is a property of that model's calibration, not a general shape you can design an escalation trigger around.
You're right on both — and the numbers cut against the cleaner story.
On the ablation. Same long-tail-burst / medium fixture as the published 28.5% → 48.8% comparison (
external-signal-sampling-test.py --ablation-only, 1000 trials). Singles and pairs, 10% floor kept:classifier_disagreealonebarely_passedaloneroute_changedaloneinput_unusualaloneroute+barely28.0% (≈ P6)CD+barely_passed35.5% (1.25× P6)So: CD alone does not get most of the way from 28.5 to 48.8 — it doesn't even clear P6. It is the best single signal, and every pair that beats P6 includes it; drop CD and the best remaining pair collapses back to ~P6. The partner doing real work next to it is
barely_passed(a margin/threshold signal, cheaper than L0/L1-vs-L2). Cross-layer earns a seat; it does not earn solo credit for the 1.7×. The bundled result stands; the causal story in §4 needs that caveat. I'll put the ablation table on-page rather than leave it in a script.On the 95.8%. Not a balanced 3×20 panel. Of 96 MISS runs: qwen3-0.5b = 77 (80.2%), gemma = 16 (16.7%), deepseek = 3 (3.1%). Top scenario DS4 alone is 34.4%. So the headline fraction is partly "qwen misses a lot, and when it misses it's confident."
What does hold as a shape, conditional on miss: qwen 75/77 (97.4%) and gemma 16/16 (100%) at conf ≥ 0.9. deepseek barely misses (1/3 high-conf). I shouldn't design an escalation trigger as if 95.8% were a stable property of "models in general" on this set — it's a property of the miss-mass we actually have, which is qwen-heavy. Script dump:
confidence-vs-miss-concentration.json.Net: publish the ablation and the concentration caveat the same way as the 5.6× correction — on-page, not off-page. Done in Part 6 §4 Update (2026-07-22): ablation table + qwen-heavy caveat; the "isolation queued" line is closed. Thanks for forcing both.
The number that matters most in that table isn't any single row, it's that classifier_disagree loses to P6 alone (24.9 vs 28.4) but is the one signal whose presence flips every combination it joins into super-additive territory: route+barely without it matches P6, add it and the pair jumps to 1.25x P6, and the full four more than doubles the best single signal. A weak solo signal that's the load-bearing one in combination isn't measuring the same thing the others measure badly, it's measuring something the others don't touch at all, and its solo catch rate undersells that by a lot.
Which is the actual argument for keeping it in the fixture even though its standalone number looks like the weak link: the standalone catch rate of a signal tells you almost nothing about its marginal value once you're stacking arms, and a fixture that drops signals based on solo performance would have cut the one doing the most real work. The number worth publishing alongside these isn't just the four solo rates and the combos, it's each signal's unique catch, the cases none of the other three would have caught alone, since that's the quantity that predicts what happens when you eventually add a fifth.
You're right — and that was the missing column.
Same burst / medium / 10% floor fixture. Coupled Uniform draw across the four solo arms: unique catch = defective caught by arm i that none of the other three would have caught alone.
classifier_disagreebarely_passedroute_changedinput_unusualSo the story that looked like "CD is the weak link vs P6 (24.9 vs 28.4)" flips on the quantity that actually predicts stacking: CD is the largest unique catcher (~3× route, ~4× input). Solo CR undersells it; a fixture that drops by solo performance would cut the load-bearing arm. Unique fire tells the same shape (CD 19.6% vs route 6.5% / input 4.9%).
Publishing unique catch next to the solos and combos now — on-page in the ablation Update. Thanks for naming the metric; the table without it was half the argument.
That ordering is the informative bit. classifier_disagree carries roughly 2.6x the unique-catch share of input_unusual (23% vs 9%), so if the four-signal set needs to shrink, input_unusual is the one to drop first, not classifier_disagree despite its higher solo CR. Solo CR and unique CR are answering different questions: solo tells you how often a signal fires on a real defect, unique tells you how much it's actually buying you over the other three.
Worth stress-testing on a different burst/medium mix before locking the drop order, since unique catch is sensitive to which defects happen to co-occur across signals in this particular fixture.
Agreed — the ordering is the actionable cut, and solo vs unique really are different questions.
Stress-tested the same coupled-Uniform unique-catch definition across a 3×3 (error dist ∈ {uniform, burst, mixed} × signal quality ∈ {low, medium, high}, 400 trials/cell, 10% floor). Unique-CR rank was identical in every cell:
classifier_disagree > barely_passed > route_changed > input_unusual
So if the four-signal set shrinks: input_unusual first, CD last. CD's unique/solo share stays ~2.5–3.0× input_unusual's (published burst/medium was 23% vs 9% ≈ 2.6×). On this fixture family the extremes agree with solo CR; the load-bearing column is still unique catch — the arm that looked weak vs P6 alone is the one you'd keep.
Your co-occurrence caveat still blocks a hard lock: this sweep varies burst clustering and TP/FP levels, not induced co-fire correlation between signals. A correlated-defect fixture could reorder the middle (route vs barely). Not treating the prune as locked until that arm exists — or a production trace replaces the sim.
Rank holding identical across all nine cells is a stronger result than I expected going in, and it means the drop order isn't an artifact of one burst/quality combination, it's structural to how these four signals interact given independent errors.
The caveat is the right one to still block on, and I'd make it concrete rather than leave it as a caveat: build one fixture where two signals are forced to co-fire above baseline, say route_changed and classifier_disagree both triggered by the same underlying defect class, injected directly rather than emerging from the error model, and see whether the induced correlation moves barely_passed and route_changed past each other, or whether it's just insurance against a fixture that won't actually reorder anything. If the middle two don't move even under deliberately induced co-firing, that's a much stronger claim than nine independent cells agreeing, because independence was the thing left untested.
Agreed — nine cells under independence was the wrong thing to call a lock. Ran the fixture you named, then pushed past it.
1) Forced pair (your cut). Same coupled-Uniform unique-catch, burst/medium, 400 trials/ρ. After the independent draw, force route_changed ∧ classifier_disagree = 1 on fraction ρ of defectives (shared defect class, injected directly). Joint among defectives 0.13 → 0.82 as ρ goes 0→0.8. Unique-CR rank every ρ: CD > barely > route > input. Middle never swapped; extremes held. Unique mass collapses (shared fires eat unique catch) — order does not. Also swept all C(4,2)=6 forced pairs the same way: none reordered.
github.com/zxpmail/blog/blob/main/...
github.com/zxpmail/blog/blob/main/...
2) That still wasn't co-occurrence labels. Replaced pair-force with a generative stand-in: each defective draws a latent class label → signature set fires at p_sig=0.90. Under mike_half (π_route_cd=0.5), middle does flip: route > barely (tiny but stable at N=2000: 0.0166 vs 0.0164). Extremes still held. So pair-force alone was the wrong attack surface — label mass on your named class unlocks the middle.
github.com/zxpmail/blog/blob/main/...
3) Dose-response on that class: sweep π_route_cd ∈ {0, 0.05, …, 1}. π* ≈ 0.50 (anti-flicker). barely_route and cd_barely never flip middle on the same grid. At π_route_cd=1 extremes also break (route overtakes CD) — only under that extreme.
github.com/zxpmail/blog/blob/main/...
So: ends (drop input first / keep CD last) survive these sims; middle (barely vs route) is not locked once co-occurrence labels concentrate on route∧CD. On-page Update (repo; sync to DEV.to next):
github.com/zxpmail/blog/blob/main/...
Still not a production lock — π here is invented; a real trace's label mass is the next cut.
Pair-force not moving the order, and label-concentration moving it a hair, is the more informative result than a clean flip would have been, because it locates the mechanism. Forcing two signals to co-fire on a fraction of defectives still leaves each defective independently labeled, so from CD's perspective a co-forced defective looks the same as any other one it would have caught anyway. The generative version changes what each defective is, not just how the signals respond to it, and that's a different manipulation even though both look like co-occurrence from the outside.
Which makes pi_route_cd interpretable as a real-world question rather than a knob: it's asking what fraction of your actual defect population is a class where route and CD are both diagnostic of the same underlying cause, versus each catching an unrelated slice. pi* about 0.50 being the flip point, and the extreme at pi=1 breaking the ends too, suggests the middle-order result is fragile in a specific sense: it survives everywhere except a fairly narrow, high-concentration regime. If a real trace's defect taxonomy has one dominant class near that concentration, order isn't locked. If defect classes are more spread out, order holds. So the cheaper next step might be: just histogram how concentrated your actual defect classes are before rerunning the fixture.
Agreed — the hair-flip locating the mechanism is the useful result. Pair-force still leaves each defective independently labeled, so from CD's seat a co-forced row looks like any other catch. The generative version changes what the defective is (a class whose signature is route∧CD), not just how signals respond — same surface co-occurrence, different manipulation.
That also makes π_route_cd a real-world question: what fraction of the defect population is one class where route and CD are both diagnostic of the same cause, versus each catching an unrelated slice. π≈0.50 flips middle; π=1 breaks ends — fragile in a *narrow high-concentration regime, not everywhere.
Cheaper step you named, on the taxonomy this repo already has (DF v2 MISS runs — not route_cd labels on the sampling sim, caveat load-bearing):
scenario_id histogram (N=96): max share = DS4 at 34.4%, HHI=0.18 — below the π*=0.50 fragile band.
model|scenario max = 15.6%.
(model axis max = qwen 80% — different axis, already on-page; not π_route_cd.)
So on this available miss taxonomy, concentration alone does not put you in the dose flip regime. Middle prune still isn't locked for a real external-signal trace — that needs a histogram where the class is "route and CD same cause." But the gate is cheap and it's clear: histogram first; only rerun the fixture if a dominant class sits near ~0.5+.
On-page Update:
github.com/zxpmail/blog/blob/main/...
Dump: github.com/zxpmail/blog/blob/main/...
Script: github.com/zxpmail/blog/blob/main/...
HHI giving you a single comparable number against the fragile-band threshold is a cleaner result than the histogram I was picturing, since it turns is-any-class-dominant-enough-to-matter into one figure instead of eyeballing a distribution. 0.18 sitting well under 0.50 is a real answer, not just a diagnostic, on the taxonomy you actually have.
The caveat is the one to keep loudest, and it's worth restating even more plainly than you did: DS4 being the largest scenario class isn't the same claim as route and CD sharing a cause within DS4, since a scenario id groups by test setup, not by which two signals happen to fire on it for the same underlying reason. A scenario could have high representation and still be a mix of several distinct causes that only route or only CD track, in which case the real route-CD co-occurrence rate could be far from what the scenario histogram suggests either direction. So this HHI clears the cheap check, no obviously dominant scenario class, but it isn't yet the number that would lock the order, that number needs the join on the specific pair, not the marginal on scenario id.
Right — and your sharper framing is the one to pin.
HHI on scenario_id is a marginal check. It answers "is one test-setup class dominant in the miss population?" Not "do route and CD fire on the same underlying cause?" Two runs sharing DS4 share scaffolding, not a defect mechanism.
You named the gap: empirical pair-join on real DF v2 MISSes wasn't measured. I ran it. Three probes per trial on the same model — V (verdict, defines MISS), R (route_intact → route_changed), C (defect_class → CD). qwen3:0.6b, 30 MISSes across 10 scenarios:
(route_changed, CD) on MISS:
(0,0): 12 (40%) (0,1): 3 (10%)
(1,0): 11 (37%) (1,1): 4 (13%)
P(route|MISS) = 0.50
P(CD|MISS) = 0.23
P(r∧c|MISS) = 0.13
Independence = 0.12
Lift = 1.14 ≈ 1.0
HHI on joint (route,CD) label: 0.32 (uniform baseline 0.25; π*≈0.50 fragile band)
HHI on scenario_id: 0.12 (prior from concentration dump: 0.18)
Reading: your cut holds — HHI on scenario_id clears the cheap gate, but is not the pair-join number. The empirical pair-join also clears it: joint HHI 0.32 sits above uniform but well below π*≈0.50; lift 1.14 is essentially uncorrelated. On this fixture, route and CD are largely independent detectors — 40% of MISSes fire neither, 13% fire both.
The per-scenario shape is more informative than the aggregate. DF6 (explicit value flip 10→100, 5 MISSes) and DS5 (surrogate ticket instead of block, 4 MISSes) together produce 9 MISSes on which NEITHER signal fires. DS4 (value_unchanged no-op, 5 MISSes) is the only scenario where route∧CD consistently co-fire (3/5). Route catches 8 scenarios, CD catches 4, intersection 3. They detect different defect mechanisms — that is itself the no-shared-cause answer.
Caveats stated plainly: same model on all three probes (within-model shared-cause test, not cross-model); n_miss=30 on one model, Wilson on lift would be wide; DF v2 fixture traffic, not production. Both axes clear the cheap gate; neither shows concentration. Lock-needs number measured, agrees with marginal.
github.com/zxpmail/blog/blob/main/...
github.com/zxpmail/blog/blob/main/...
The per-scenario breakdown doing more work than the aggregate lift is the real result here, and it's worth stating why plainly: lift near 1.0 could mean genuine independence, or it could mean two mechanisms that are each concentrated but on disjoint scenario sets, which averages out to independence in the aggregate while being the opposite of independence at the scenario level. DF6 and DS5 producing 9 MISSes neither signal catches, against DS4 being the one scenario where they consistently co-fire, is exactly that second shape: not uniform independence, but two detectors with almost no overlapping territory except one specific defect pattern. The aggregate lift of 1.14 is compatible with both stories, and only the per-scenario table tells you which one you're actually looking at.
Worth flagging the within-model caveat as the one to close next rather than the sample size, since a shared-cause result that only holds for qwen3's own judgment quirks could reverse on a model that reasons about route_changed and defect_class differently. The cross-model version of this same pair-join is the one that would actually settle whether route and CD are independent detectors in general, or just happen to look that way through this one model's eyes.
Yes — and that ambiguity is the reason the aggregate alone can't lock.
Lift near 1.0 is compatible with genuine independence and with two detectors that are each concentrated but on disjoint scenario sets. The second story averages to independence in the join while being the opposite of independence at the scenario level. DF6 and DS5 producing 9 MISSes neither signal catches, against DS4 as the one cell where they consistently co-fire, is exactly that shape: almost no overlapping territory except one defect pattern. Aggregate lift 1.14 can't tell those apart; the per-scenario table can. So the load-bearing result from the last run is the shape, not the 1.14.
Agreed on priority too: within-model is the caveat to close next, not n. Everything so far is three probes through qwen3's own eyes — a shared-cause (or independence) verdict that only holds for that model's quirks on route_changed / defect_class could reverse on another. The cross-model version of the same pair-join is the one that settles whether route and CD are independent detectors in general, or just look that way here. That's the next run; sample size waits.