Agent Determinism Illusions (Part 6)
Where this fits: Part 5 closed the experimental arc with an honest answer — no clean fix for the 75% false-negative wall. The Red Line Principle asked the upstream question (when does the loop stop?). This part takes the downstream turn Part 5 already pointed at: stop trying to move the wall; put rules where rules work, LLM only on residual, humans where models diverge. Five insights from overlapping commenters named the pieces (Alexey and Manuel each appear in more than one). Experiment F (38 scenarios) checks whether the resulting pipeline behaves as claimed.
Six experiments, 260+ API calls, 15 scripts. Part 5 ended that stretch with: there's no clean solution to LLM output verification.
But after those posts went live, commenters saw something I didn't — not gaps in the data, but an architecture I'd failed to draw from my own results. This article collects their five key insights and shows how they reorganize the experiment data into a working pipeline.
§§1–4 are paired with experimental or simulation checks from a new prototype (Experiment F, 38 scenarios across two test sets). §5 is a design claim — flagged as such in place.
1. Alexey Spinov & Manuel Bruña: Layer Before You Judge
Alexey's comment identified the most fundamental design flaw in my experiments:
"G4 ('0 passed, no tests collected') is a fact that can be verified with code in one shot. There is no need to wait for an LLM."
Manuel added the constructive direction:
"Run deterministic checks first. Then let the LLM handle only the truly ambiguous residual."
I went back to my own 8-scenario P1 test set. Four garbage scenarios (G1-G4) and four legitimate ones (L1-L4):
| ID | Output | Type | Could code catch it? |
|---|---|---|---|
| G1 | "I am a little duck, quack quack" | nonsense | ✅ very short + no keywords |
| G2 | "。" (a period) | pure punctuation | ✅ punctuation ratio > 50% |
| G3 | "TODO" | placeholder | ✅ keyword blacklist |
| G4 | "0 passed in 0.00s (no tests collected)" | zero-test pass | ✅ regex 0 passed + no tests
|
All four garbage scenarios can be caught deterministically, at zero cost, before any LLM call.
Why didn't I do this? Because I defaulted to treating "verification" as "ask the LLM." My experiment design was: Phase Gate (form check) → LLM (content check). I never inserted the simplest possible code checks in between — minimum length, punctuation ratio, keyword blacklist, regex patterns.
This omission rippled through the entire series:
- P1–P4 wasted LLM budget on garbage — G1–G4 never needed a semantic judge; every call spent on them was pure cost. The 75% FN wall on legitimate scenarios is a separate problem (Part 5) — layering doesn't erase it, it stops mixing easy rejects into the same experiment as the hard line-drawing
- P3's "majority voting doesn't fix systematic bias" — on legitimate scenarios (L1-L3), the LLM's judgment is genuinely ambiguous and needs multi-perspective voting. For garbage (G1-G4), there was never any ambiguity to begin with
- P4's edge cases still reach Layer 2 — many new samples were "passes format checks, fails content quality." That is exactly what L0/L1 cannot catch: they filter garbage/shape, then hand the semantic residual to the LLM. Layering does not absorb those edges; it stops pretending garbage was a semantic problem
The architecture they helped me draw
┌─────────────────┐
input ──→ │ Layer 0 (code) │ shape / existence
│ │ empty? punctuation? placeholder? zero tests?
└────────┬────────┘
│ pass
▼
┌─────────────────┐
│ Layer 1 (code) │ contract match
│ │ minLen, keywords, blacklist
└────────┬────────┘
pass│
┌──────────────┴──────────────┐
│ fail │ pass
▼ ▼
REJECT ┌─────────────────┐
│ Layer 2 (LLM) │ semantic residual only
└────────┬────────┘
unanimous│ │divergence (e.g. 2–1)
▼ ▼
AUTO-PASS ┌─────────────────┐
│ Layer 3 human │
└─────────────────┘
Each of L0/L1 can early-exit to REJECT. Divergence is a Layer 2 signal (multi-perspective split), not a Layer 1 signal. If Layer 0 catches it, the LLM never sees it.
Experiment F validation
I implemented this pipeline as a Python prototype and ran it on both the P1 (8-scenario) and P4 (30-sample) test sets. The results:
P1 test set:
| Metric | Original P1 (LLM only, v2) | Layered + calibrated prompt (Experiment F) |
|---|---|---|
| LLM calls needed (single judge / sample) | 8 (100%) | 4 (50%) |
| Garbage caught by L0/L1 | 0 | 4/4 (100%) |
| False positives | 0 | 0 |
| False negatives | 3 (75%) | 0 |
(FN→0 here is layering **plus* the calibrated prompt — not layering alone. §2 separates the two effects. Call counts here are one judge call per sample. When §2 multiplies by three perspectives, it says so.)*
Rerun: python forge-verify-layered-prototype.py (needs ANTHROPIC_* for Layer 2; SKIP_LLM=1 for L0/L1 only). Numbers above are from a full run with Layer 2 enabled.
P4 test set:
| Category | Samples | Caught by L0 | Caught by L1 | Reaches L2 | Zero-cost catch rate |
|---|---|---|---|---|---|
| correct | 10 | 0 | 0 | 10 | 0% (should all go to LLM) |
| garbage | 10 | 3 | 5 | 2 | 80% |
| edge | 10 | 0 | 2 | 8 | 20% |
Overall: single-judge LLM calls reduced 33% (30→20). Zero false positives from deterministic layers.
This does not move Part 5's wall. On the P1 set, the original 75% FN (3/4 legitimate rejects) went to 0 FN after L0/L1 removed all four garbage cases from the LLM's input — the LLM only judged the four legitimate scenarios, and with a calibrated prompt it didn't reject them. The wall is still there for semantic residual: Layer 2 still draws a line on underspecified "is this enough?" questions. Layering shrinks how often you ask that question; it does not make the question well-posed. If you read the FN→0 cell as "we fixed the wall," you've misread the table.
The two garbage samples that made it through to Layer 2 (G08: "I cannot parse this command", G10: incomplete translation) are genuinely ambiguous — they should reach the LLM. That's correct behavior, not a leak.
2. Alexey Spinov: Cost Asymmetry
Alexey's second comment pointed out a measurement problem:
"A false accept ships once. A false reject triggers a retry, which burns tokens and can loop, so an over-rejecting judge does not just lose good work, it re-does already-valid work at model prices."
All experiments P1-P4 used symmetric precision-recall metrics. F1 gives FP and FN equal weight. A false negative triggers a full repair loop — 3x token consumption, 3x latency, possible infinite loops. A false positive is one-shot contamination.
I ran a dedicated cost-weight analysis (scripts/cost-weight-optimization.py) that takes P3b's 5 prompt variants and evaluates them across 5 cost ratios, to show how the "optimal" choice shifts.
5 prompts × 5 cost ratios
| Prompt | FP | FN | F1 | WCost(1:1) | WCost(3:1) | WCost(5:1) | WCost(10:1) |
|---|---|---|---|---|---|---|---|
| v1 extreme strict | 0 | 4 | 0 | 4 | 12 | 20 | 40 |
| v2 strict (P1 baseline) | 0 | 3 | 0 | 3 | 9 | 15 | 30 |
| v3 balanced | 0 | 0 | 100 | 0 | 0 | 0 | 0 |
| v4 lenient | 0 | 0 | 100 | 0 | 0 | 0 | 0 |
| v5 extreme lenient | 1 | 0 | 86 | 1 | 1 | 1 | 1 |
Under symmetric F1, v3 (100) and v5 (86) are far apart. Under weighted cost at 3:1, v5 (cost=1) beats v2 (cost=9) — v5 let one piece of garbage through, but because it never rejected valid work, its total cost is far lower than the strict prompt. v3 (cost=0) still wins outright; the useful flip is v5 vs v2, not “v5 ties v3.”
Read this table as a ranking-flip demo, not as a production recommendation. v3/v4's zeros are an 8-scenario artifact (P4 already showed they don't survive at N=30). The load-bearing claim is the shift in relative ranking under cost weight — especially that thrift can prefer a slightly leaky prompt over a zero-FP / high-FN one — not that F1's winner changes on this tiny set (v3 stays on top whenever FN=FP=0).
What the combined data shows
Call counts in this table = samples reaching an LLM × **3 perspectives* (Strict/Balanced/Lenient), matching P3-style voting cost. §1's Experiment F table counts one judge call per sample. Same pipeline; different billing unit.*
| Strategy | WCost(1:1) | WCost(3:1) | WCost(10:1) | LLM calls (×3 perspectives) |
|---|---|---|---|---|
| P3b v2 (unlayered) | 3 | 9 | 30 | 8×3=24 |
| P3b v3 (unlayered) | 0 | 0 | 0 | 8×3=24 |
| P1 layered + v3 | 0 | 0 | 0 | 4×3=12 (−50%) |
| P4 unlayered (estimate) | 4 | 8 | 22 | 30×3=90 |
| P4 layered (Experiment F residual) | 1 | 3 | 10 | 20×3=60 (−33%) |
Layering doesn't change that v3's cost is 0 (it already has FP=FN=0 on the 8-scenario set). But it changes two things that the raw cost number doesn't capture:
- 4/4 garbage caught by L0/L1 at zero cost — call volume on the residual is halved; that does not halve the cost of an FN on a legitimate residual sample (that FN still costs a full repair loop)
- 33–50% fewer samples reach the LLM — not by changing the model, by giving it fewer samples to judge
For v2 (the strict prompt from P1), the effect is more instructive. v2 has FN=3. Layering saves calls on garbage but doesn't reduce FN on the legitimate set:
- Layering + switching prompt (v2→v3): FN drops from 3 to 0
- Layering only: saves tokens, but FN stays at 3
This exposes the boundary of layering: it reduces the LLM's workload, not its bias. To reduce FN on residual, you need prompt calibration alongside layering — and even then, Part 5's wall says calibration does not generalize past small sets.
Sensitivity scan: when does the ranking move?
I ran a continuous scan from costFN:costFP = 1:1 to 15:1. On the P3b 8-scenario set, v3 dominates every ratio — because FP=FN=0 yields zero weighted cost at any weight. That is the small-set artifact again (P4 already showed the perfection doesn't generalize).
What does move is the gap narrative: at 1:1, F1 makes v3 look far ahead of v5 (100 vs 86). At 3:1, weighted costs are 0 vs 1 — v3 still wins, but the moral of the story is no longer “balance beats thrift”; it is “any FN>0 gets expensive fast, so a one-FP leak can beat a three-FN strict prompt (v5 vs v2).” At 10:1, every strategy with FN>0 collapses relative to zero-FN prompts on this set.
Five findings
Symmetric metrics hide relative rankings that matter under cost. F1 dramatizes v3 ≫ v5. Weighted cost shows v5 ≫ v2 once FN is expensive — the comparison that production actually faces when choosing strict vs leaky.
On this 8-scenario set, the F1 winner (v3) remains the weighted-cost winner. Do not read the section as “the optimum flips away from v3 at 3:1.” It does not. The flip that matters is strict-zero-FP (v2) losing to slightly-leaky-zero-FN (v5) under FN-heavy weights.
v3/v4's zero errors are an 8-scenario artifact. P4 already showed the advantage disappears at 30 samples. Treat zeros as a demo substrate, not a deployable operating point.
Layering doesn't reduce bias, but it shrinks how often bias is invoked. After L0/L1 filters garbage, fewer samples hit the LLM; residual FNs still cost full price.
Drive FN→0 where rules apply; accept the wall on semantic residual. Above cost ratio ~5:1, strategies with FN>0 on garbage/contract work are unsustainable — use L0/L1 + a non-strict residual prompt. On underspecified “is this enough?” questions, Part 5 still holds: you choose an operating point on the wall, you do not delete the wall. Weighted cost picks the point; it does not invent a zero-FN semantic judge.
3. Dipankar Sarkar: Divergence Is the Signal, Not Noise
P3's multi-perspective voting experiment found a pattern I described but misinterpreted. My original framing:
"In split-vote scenarios, the majority was always wrong. Majority voting can't correct for systematic bias."
Dipankar flipped the interpretation:
"Vote disagreement itself is the most valuable signal. When three reviewers disagree on the same scenario, it means the scenario is genuinely ambiguous — route it to human review instead of averaging."
Re-examining P3's data through this lens:
| Scenario | Strict | Balanced | Lenient | Majority | Correct? |
|---|---|---|---|---|---|
| L1 (excerpt) | REJ | REJ | PASS | REJ (2-1) | ✗ FN |
| L2 (summary) | REJ | REJ | PASS | REJ (2-1) | ✗ FN |
| L3 (one chapter) | REJ | REJ | PASS | REJ (2-1) | ✗ FN |
| G3 (TODO) | REJ | REJ | PASS | REJ (2-1) | ✓ |
Majority voting was wrong on 3 of 4 split scenarios. But if I use divergence as the control signal:
- Unanimous (4/8 on that P3 run): auto-execute → 100% accuracy on those four (the script's unanimous bucket for that run — typically clean garbage rejects / clear passes; not re-listed here)
- Split (4/8): escalate to human → no false majority decisions
Caveat: divergence-routing fixes split errors. It does not fix unanimous systematic bias — if all three perspectives share the same wrong line (Part 5's wall), auto-execute still ships the wrong call. Dipankar's move measures uncertainty; it does not delete the wall.
Dipankar wasn't proposing a "better multi-perspective voting algorithm." He was pointing out that the purpose of voting is not to find a majority — it's to measure uncertainty. I missed this distinction when writing P3.
Operational rule (now implemented in forge-verify's layer 3):
if max(PASS, REJECT) / N < threshold (default 0.8)
→ mark as UNCLEAR, write to human review queue
→ do NOT majority-vote
4. Mike Czerwinski & xm_dev_2026: Fixed Sampling Misses Long Tails
Mike Czerwinski named the architectural limit I'd been circling without stating:
"Stacking more symbolic checks on top doesn't grow that reach, it just adds more places for the same blind spot to hide... 'Ask the human' isn't a retreat, it's the only honest move once you've located where reach actually lives."
The verification layer has reach into symbolic events (file exists, exit 0) but not into semantic correctness — the blind spot doesn't shrink, it moves. P4 reported 83.3% accuracy across 30 samples, but the misses inside the auto-passed 83% are exactly where Mike's "no reach" critique lands: invisible by construction.
xm_dev_2026 showed where this bites hardest in production — fixed-percentage audits:
"Fixed-percentage audits feel 'fair' but they miss exactly the kind of long-tail directional failures you're describing. The model is most confident when it's wrong in a structured way."
My original mitigation had been "5-10% random audit." This isn't a parameter-tuning problem — it's a design principle problem. Fixed sampling assumes errors are uniformly distributed. Real production errors are long-tailed.
I ran a simulation (scripts/adaptive-sampling-sim.py) — synthetic verification streams with controlled error distributions, not production logs — then compared fixed-rate sampling against adaptive sampling (200-trial averages). The numbers below are about relative efficiency under known distributions, not measured catch rates from a live agent fleet.
The adaptive formula:
audit_rate = base_rate × risk_weight / confidence^1.5
High-risk items get a higher audit rate; high-confidence items get a lower one. Where confidence is cross-prompt divergence (Strict/Balanced/Lenient split), not the model's self-reported confidence — see the Update (2026-07-21) at the end of this section for why this signal under-samples the confident-and-wrong tail.
Results
Uniform (errors spread evenly):
| Strategy | Audit rate | Catch rate | Efficiency |
|---|---|---|---|
| Fixed 10% | 10.0% | 12% | 0.111 |
| Adaptive(combined) | 7.8% | 18% | 0.247 |
No significant gap — without a long-tail structure, random sampling is adequate.
Long-tail burst (90% of errors in 10% of the stream — xm_dev_2026's scenario):
| Strategy | Audit rate | Long-tail catch rate | Efficiency |
|---|---|---|---|
| Fixed 5% | 5.0% | 5% | 0.120 |
| Fixed 10% | 10.0% | 10% | 0.119 |
| Fixed 20% | 20.0% | 20% | 0.118 |
| Adaptive(confidence) | 8.7% | 24% | 0.321 |
| Adaptive(risk) | 6.0% | 15% | 0.264 |
| Adaptive(combined) | 12.8% | 65% | 0.543 |
Adaptive(combined) uses a slightly higher audit rate than Fixed 10% (12.8% vs 10.0%) and catches 65% of long-tail errors vs Fixed 10%'s 10% — roughly 6× on catch rate, not an equal-budget comparison. For equal budget, see the Mixed table below (10.0% vs 10.1% → 10% vs 56% catch).
Distribution drift (Mike's "input shift erases visibility"):
| Strategy | Audit rate | Catch rate |
|---|---|---|
| Fixed 10% | 10.0% (flat) | 11% (constant) |
| Adaptive(combined) | 21.5% (ramps up) | 48% |
Fixed sampling's catch rate stays flat as the error rate drifts upward. Adaptive sampling automatically increases the audit rate in later stream segments — because more errors mean lower model confidence, and the confidence signal drives the audit rate up.
Mixed (uniform noise + periodic long-tail bursts — most realistic, 200-trial average):
| Strategy | Audit rate | Long-tail catch rate | Efficiency |
|---|---|---|---|
| Fixed 5% | 5.0% | 5% | 0.095 |
| Fixed 10% | 10.0% | 10% | 0.094 |
| Fixed 20% | 20.0% | 20% | 0.094 |
| Adaptive(confidence) | 7.8% | 19% | 0.230 |
| Adaptive(risk) | 6.0% | 15% | 0.148 |
| Adaptive(combined) | 10.1% | 56% | 0.324 |
Key row: Fixed 10% and Adaptive(combined) have nearly identical audit rates (10.0% vs 10.1%), but long-tail catch rate goes from 10% to 56% — a 5.6x improvement at the same cost. Caveat (added 2026-07-21): this headline uses cross-prompt divergence as the confidence signal. A subsequent simulation showed this signal under-samples the confident-and-wrong tail and a cross-layer signal beats it 1.7× on the same distribution — see Update below.
Why it works (cross-validated against my existing experiments)
P2's consistency test showed that binary LLM judgments on identical input are highly stable (N=10, 0 divergence). So the "confidence" signal doesn't come from within-model variance — it comes from P3's multi-perspective divergence:
- Unanimous → high confidence → low audit rate (<5%)
- Split (2-1) → low confidence → high audit rate (>30%)
P2 and P3 were independent experiments, but their outputs combine into a coherent adaptive sampling story. This isn't parameter tweaking — it's a design principle difference. Fixed sampling pursues "uniform global coverage." Adaptive sampling says "put more watchposts where things are likely to break." In the simulation, the latter uses audit budget ~3–4× more efficiently (Mixed: efficiency 0.324 vs 0.094).
Production caveat: the “confidence” signal here is cross-prompt divergence, not within-model variance (P2 showed the latter is ~0). If every item must run Strict/Balanced/Lenient before you know the audit rate, the probe can cost as much as the audit you hoped to save. Treat adaptive sampling as a principle for when a cheap uncertainty signal already exists (or is sampled sparsely) — not as “run three judges on everything, then randomly audit.” The simulation assumes the confidence channel is given; it does not price building that channel.
Update (2026-07-21): cross-prompt collapses on the dangerous tail; cross-layer is the right signal
Mike Czerwinski pushed on §4 in the comments: confidence-weighting concentrates audits where the model is unsure, but the long-tail directional failure is the confident-and-wrong case — high self-assessed confidence, wrong direction, consistent across runs. Weighting on 1/confidence^1.5 samples that region least. The 5.6× headline above is, on real failure mass, partly bought by sampling less exactly where the failure lives.
Three checks after Mike's comment:
1. Where do the misses actually sit? Counted MISS runs (passes=true on a directional reversal) across DF v2's 3 models × 20 scenarios — 96 misses total, 92 of them (95.8%) at self-reported confidence ≥ 0.9, avg confidence 0.969. The dangerous tail is overwhelmingly high-confidence. Script: confidence-vs-miss-concentration.py; raw data in results-v2/*_summary.json + matching .jsonl.
2. Does the cross-prompt dodge work? The Production caveat above reframes "confidence" as cross-prompt divergence rather than self-confidence. Doesn't change the point. P2's consistency test was 0 divergence on identical input at N=10 — three prompts in the same text channel collapse the same way on the same plausible rationalization. Cross-prompt is still in-channel; it goes quiet on the confident-and-wrong case too.
3. Cross-layer is the right signal — and it's already simulated. I'd run the comparison before Mike's comment but hadn't pointed §4 at it. external-signal-sampling-test.py compares Fixed 10% vs the Part 6 cross-prompt adaptive arm vs a third arm built from four binary external signals — one of which (classifier_disagree) is L0/L1-vs-L2 disagreement, exactly the cross-layer signal Mike names — plus route_changed, input_unusual, barely_passed, on a 10% baseline floor. Long-tail burst, medium signal quality:
| Strategy | Audit rate | Long-tail catch rate |
|---|---|---|
| Fixed 10% | 10.0% | 10.0% |
| Part 6 cross-prompt adaptive | 13.8% | 28.5% |
| External signals (incl. cross-layer) + 10% floor | 23.5% | 48.8% |
1.7× over Part 6's cross-prompt arm at the same audit-rate class, with the non-zero floor Mike asked for already built in as the 10% baseline. What's still missing: the simulation bundles four external signals, so I can't yet isolate how much of the 1.7× comes specifically from classifier_disagree vs the other three firing alongside it. That isolation is queued.
The 5.6× headline above stands as a simulation result under the cross-prompt signal — but the cross-prompt signal goes quiet where the failures actually live. The replacement headline uses cross-layer disagreement, weighted on L0/L1-vs-L2 divergence with a non-zero floor in the high-confidence region.
5. Manuel Bruña & Alexey Spinov: Evidence, Not Narrative
Throughout P1-P4, all LLM review experiments output free-text "reason" fields. Manuel identified the structural problem and the fix in one sentence:
"Treat the LLM inspector as an evidence-producing reviewer, not the final binary gate. Cheap deterministic checks first, then an inspector that must quote the exact failing evidence."
Alexey sharpened the architectural split:
"Deterministic assertions own everything mechanically checkable (tests collected greater than zero, schema conformance, non-emptiness thresholds), and the LLM only judges the irreducibly fuzzy residue."
My experiments had this blind spot:
P1, scenario L1 (model REJECT):
"The research brief should cover the core mechanisms of the loop engine,
but the file only has a short excerpt..."
P1, scenario L3 (model REJECT):
"The task requires three chapters, but the output only contains one."
These are impression judgments. You can't code-verify whether "a short excerpt" is enough.
The proposed output format:
Assertion 1: "File line count = 3, expected > 20" → code-verifiable
Assertion 2: "File contains 1/3 required keywords" → code-verifiable
Assertion 3: "Content structure completeness < threshold" → semantic judgment
Assertions 1-2 are deterministic — code can confirm whether the model's claim is true. Assertion 3 is the actual semantic judgment, preserve for Layer 2.
This creates a cascade: when a deterministic assertion is code-verified and found inconsistent with the actual file → explicit hallucination signal → mark as UNCLEAR → escalate. No human judgment required in the loop — the code flow triggers automatically.
Scope note: unlike §§1–4, this section is a design claim, not a separate A/B in Experiment F. The prototype implements evidence-shaped L2 output; it does not measure whether assertion format alone reduces hallucination rate versus free-text reasons. Treat the cascade above as an engineering pattern pending that measurement.
Synthesis: What the Five Comments Build Together
| Comment | My blind spot | Replacement |
|---|---|---|
| Alexey + Manuel | Fed everything to the same LLM reviewer | L0/L1 filter deterministically; LLM handles residual |
| Alexey (2nd) | Symmetric FP/FN metrics | Weighted cost (FN×3) shifts optimal operating point |
| Dipankar | Split votes averaged by majority | Divergence = UNCLEAR → human, no majority |
| Mike + xm_dev_2026 | Fixed 5-10% audit rate | Adaptive sampling by confidence × risk |
| Manuel + Alexey (2nd) | Narrative "reason" field | Evidence-quoted reviewer + deterministic assertions |
Combined, these form a layered verification architecture — not a closed one: L0/L1 handle deterministic filtering (Alexey+Manuel), L2 LLM quotes exact failing evidence (Manuel+Alexey), divergence escalates to L3 human review (Dipankar), audit rate adapts by confidence (Mike+xm_dev_2026), and system thresholds are selected by weighted cost (Alexey 2nd). Each layer narrows what the next sees; none closes the semantic residue.
This article doesn't claim to have solved anything. It just puts the design decisions I made and the corrections the community provided side by side.
Implementation
The full pipeline has been implemented in forge-verify's content-verify.mjs (ReqForge product repo, not this blog tree — the blog ships the Python prototype forge-verify-layered-prototype.py). File-by-file results show which layer stopped each sample. Early-exit example (L1 blacklist — L2/L3 never run):
📄 src/api/register.ts
❌ REJECT @ L1: contains blacklisted keyword: FIXME
└ L0: PASS
└ L1: REJECT — blacklisted keyword: FIXME
Divergence example (L0/L1 pass; L2 split → L3 human, no majority vote):
📄 docs/brief.md
⚠ UNCLEAR @ L3: split vote → human queue
└ L0: PASS
└ L1: PASS
└ L2: [REJECT/REJECT/PASS] PASS=1 REJ=2
└ L3: UNCLEAR — do not majority-vote
Layer 0/1 checks are zero-cost code. Layer 2 only runs on the residual. Layer 3 divergence detection prevents false majority decisions.
A Side Note: An Apology Experiment
An earlier draft appended a long apology for a fabricated “directional failure” claim in a Part 3 comment. That thread became its own experiment (20×3×600) and then a correction stack (comment wrong → apology v1 wrong on DS4 → v2 numbers). Under the harness label, DS4 still 100% misses on qwen3/gemma3; deepseek is 13%/67%/20% catch/PARSE/miss. Post-hoc, DS4 is partly task ambiguity (10→10); clean L0/L1 wins remain DF6/DS9 value mismatch. Full write-up: forthcoming aside. Scripts: directional-failure-v2.py / scripts/results-v2/.
Series navigation (Agent Determinism Illusions):
- I tested the 'deterministic agent loop' claims…
- I tested 3 models as AI agent quality inspectors…
- I designed a Harness… then found 6 flaws
- An alternative to LLM quality gates: deterministic routing + sampling
- Six experiments… and the 75% wall that didn't move
- Aside: The Red Line Principle
- Five comments that redesigned my LLM verification pipeline (this article)
- Aside (forthcoming): I Fabricated a Claim About LLM Judges. Then I Ran the Apology Experiment.
Published parts: dev.to/zxpmail. Scripts: GitHub.
Experiment F prototype (this repo): forge-verify-layered-prototype.py (Python, runnable with or without API)
forge-verify production path: ReqForge product repo — scripts/forge-verify/content-verify.mjs (not vendored here)
Previous: The Red Line Principle
Series start: Four experiments…
Which comment did I miss? If you've hit a verification failure mode that the L0/L1/L2/L3 pipeline doesn't catch, drop it in the comments — I'll run it through Experiment F and report what each layer does with it.
Top comments (2)
The consolidation into L0/L1/L2/L3 is the right shape, and since the adaptive-sampling row is the one I had a hand in, let me push on where it can quietly re-open the hole it closed. Confidence-weighted sampling concentrates audits where the model is unsure. But the long-tail directional failure the fixed-percentage critique was about is not the unsure case, it is the confident-and-wrong case: high self-assessed confidence, wrong direction, made the same way every time. Weighting on confidence samples that region least, because high confidence drives the rate down, so the 5.6x efficiency is partly bought by sampling less exactly where the failure you were hunting lives. Flat sampling at least covered it uniformly. Confidence-weighting can under-cover it relative to flat, which is the original hole wearing the efficiency gain as a disguise.
The fix is already in your architecture, you just weight on it instead of on self-confidence. A model's confidence shares the blindness of the thing being audited, it is the actor grading its own certainty, so it goes quiet in exactly the region that needs eyes. Cross-layer disagreement does not. Where L0/L1 and L2 return different verdicts on the same item, you have an independent flag that does not collapse when the LLM is confidently wrong, because the code filter is not reading the model's confidence, it is reading the artifact. Weight the audit sample on that cross-layer disagreement, and keep a non-zero floor in the high-confidence region so it can never be driven to zero, and you keep most of the efficiency while closing the tail. Divergence is already your most trusted signal at L3. It should also steer where the audit looks one layer earlier, and across layers, not only inside the LLM vote.
Right — and two pieces of data back this.
First, where the failures actually live. DF v2 produced 96 MISS runs across 3 models × 20 scenarios (passes=true on a directional reversal). 92 of those 96 — 95.8% — sat at self-reported confidence ≥ 0.9, avg 0.969. The dangerous tail is overwhelmingly high-confidence, which is exactly the region
1/confidence^1.5samples least.Second, the article's production caveat tried to dodge this — "confidence" meant cross-prompt divergence (Strict/Balanced/Lenient split), not within-model variance. Doesn't change the point. P2's consistency test was 0 divergence on identical input at N=10; three prompts in the same text channel collapse the same way on the same plausible rationalization. Cross-prompt is still in-channel.
And the fix you're naming — I'd already simulated it, just didn't point §4 at it.
external-signal-sampling-test.pycompares Fixed 10% vs Part 6 cross-prompt adaptive vs a third arm: four binary external signals includingclassifier_disagree(your L0/L1-vs-L2) plusroute_changed/input_unusual/barely_passed, on a 10% baseline floor. Long-tail burst, medium signal quality: Fixed 10.0% catch, Part 6 28.5%, external-signal arm 48.8% — 1.7× over Part 6 at the same audit-rate class. The non-zero floor you asked for is already the 10% baseline. What's missing: I didn't isolateclassifier_disagreealone, so I can't say how much of the 1.7× is specifically cross-layer disagreement vs the other three signals firing alongside it.So: the 5.6× headline was a simulation under a signal that goes quiet on real failure mass, and the cross-layer signal you proposed is the right replacement. I added an Update subsection to §4 citing the cross-layer simulation (1.7× over the cross-prompt arm, 48.8% vs 28.5% catch at long-tail burst), caveated the 5.6× headline inline, and kept the original results visible — the cross-prompt numbers stand as simulation output, just no longer as the headline. The
classifier_disagree-isolated run is queued; the bundled-signal result is what's live now.