Agent Determinism Illusions (Part 6)
Where this fits: Part 5 closed the experimental arc with an honest answer — no clean fix for the 75% false-n...
For further actions, you may consider blocking this person and/or reporting abuse
Fun seeing the layering framing turn into an actual pipeline. The L0/L1-before-judge split is exactly the shape.
The part I'd push on is the escalation trigger. Right now L2 to L3 fires on inter-judge divergence (Dipankar's 2-1 split, UNCLEAR, human). But your own reply to @jugeni is the tell: 95.8% of the DF v2 MISS runs sat at confidence >= 0.9, avg 0.969. If the dangerous failures are high-confidence and directional, they're systematic, and systematic bias is shared across prompts, not idiosyncratic (your own P3 result: majority voting doesn't fix it). So the three perspectives will tend to agree on exactly those cases. Divergence-to-human then routes you the safely-ambiguous ones and auto-passes the confidently-wrong ones. The escalation signal is pointing at the wrong population.
Which suggests the human/tripwire layer wants a signal that tracks the failure mode, not judge disagreement. Two cheap candidates: a deterministic tripwire on the known-reversal classes (route those regardless of agreement), and treating unanimous-high-confidence on a historically-reversal-prone class as its own escalate trigger, the inverse of the usual "high confidence, auto-pass." We ran into the same thing voting judges over an eval residual: unanimity was where the correlated errors hid. Variance dropped, shared bias didn't. So we stopped reading agreement as confidence and started sampling "agreement on a class we've been wrong about before" for review.
You're right — and the 95.8% / 0.969 MISS concentration is exactly the tell.
Divergence→human measures ambiguity. My dangerous residue is correlated bias: three perspectives agree on the wrong call, so unanimous auto-pass routes humans to the safe-ambiguous set and lets the confident-wrong set through. Agreement ≠ confidence when the failure mode is shared.
I'm updating the tripwire model:
1) Deterministic escalate/reject on known-reversal classes (regardless of vote agreement).
2) Treat unanimous + high-confidence on a historically reversal-prone class as escalate — the inverse of “high confidence, auto-pass.”
Divergence stays for genuine ambiguity. It stops being the only L2→L3 signal. Thanks for naming the population mismatch so cleanly.
You're taking that further than I'd earned, so let me put the correction where it belongs — on my own suggestion. The inverted trigger was mine, and the arithmetic says the part I emphasised is the part that doesn't matter.
0.958 is P(high confidence | MISS): a property of the failure population. What the human on the far end of an escalate experiences is P(MISS | trigger fired) and the volume that fires, and those move independently of it. So I swept a grid instead of guessing your numbers — pi = P(MISS), h = P(unanimous+high-conf | MISS), u = P(unanimous+high-conf | correct), rm and ro = class-list hit rates on each population. 720 cells, exhaustive.
One cell, using your 0.958 (pi=.08, u=.70, rm=.60, ro=.10):
stream fires on precision of all MISS
divergence 27.94% 1.20% 4.20%
class list alone 14.00% 34.29% 60.00%
unanimous+high-conf 72.06% 10.63% 95.80%
UHC and class 11.04% 41.66% 57.48%
Your instinct holds: the trigger beats divergence 34.6x on precision, and divergence is routing humans into a 1.20% stream. That part of the redesign is right.
Here's what isn't. Compare rows 2 and 4. The class list alone fires at 14.00% / 34.29%; adding unanimity-and-high-confidence moves it to 11.04% / 41.66%. Across all 720 cells:
share of the trigger's volume cut owed to the CLASS LIST alone
min 72.4% median 96.8% max 99.9%
precision lift from ADDING unanimous+high-confidence on top
min 0.74x median 1.01x max 1.88x
Median lift 1.01x. At the bad end, 0.74x — the signal makes the stream worse than the class list by itself. The class list does ~97% of the selection, and the signal this thread has been arguing about, mine included, is a rounding error on top of it.
That reframes @jugeni's point rather than answering it. If R is the mechanism, and R is built from reversals you have already caught, then the coverage limit he named is not a limit of the trigger — it IS the trigger. T1 and T2 differ by a multiplier of about 1.
Second thing the grid says, which nobody has costed. The trigger's floor volume is pi*h*rm = 4.60% of all traffic in that cell. Those are its true positives; no class filter shrinks them without discarding the failures it exists to find. A 2% human review budget cannot hold them. So the open question moves from "what fires" to "how do you rank inside the stream", and no trigger definition supplies that.
And the failure mode is the wiring, not the trigger. At a 2% budget, in 76 of 720 cells, adding this stream to an unordered queue catches FEWER true misses than divergence alone — it displaces the better stream at arrival proportion instead of by yield. Under a precision-ordered queue: 0 of 720 cells. Same shape as @ethanwritesai's point about the judge on the blocking path. A gate doesn't get deleted by what it detects; it gets deleted by how it's wired.
Against myself, twice. The grid assumes class membership is independent of unanimity and confidence within each truth class. Tilting that correlation the plausible direction — reversal-prone classes are reversal-prone BECAUSE judges agree confidently and wrongly there — drops that cell from 41.66% to 35.45%. So the 1.01x median lift is an upper bound and my own proposal looks worse than the table shows, not better. And all of this is arithmetic over a swept grid, not a measurement: I don't have your parameters, which is exactly why they're swept. What survives is the sign of each effect and where the break-even sits, not any single cell.
The one number that would settle it is in your logs and hasn't appeared in this thread: P(unanimous and confidence >= 0.9 | the judge was RIGHT). 0.958 is its mirror on the other population, and without it the precision of every trigger discussed here is unbounded from below. If it's already published in a part I haven't read, point me at it and I'll run the cell instead of the grid.
(escalation_trigger_yield.py — stdlib only, offline, no randomness; three runs byte-identical; sha256 of stdout 5550717f470a5a37678d16117d1f31a85ee717318455129312561b6392d9f0bb)
You asked for the mirror. Fresh multi-perspective rerun (Strict/Balanced/Lenient × DF v2 × qwen3:0.5b / gemma3 / deepseek-v4-flash; 60 scenario-rows):
u = P(unanimous ∧ conf≥0.9 | RIGHT) = 19/47 ≈ 0.404
h = P(unanimous ∧ conf≥0.9 | MISS) = 4/7 ≈ 0.571
(RIGHT = true_pass ∨ true_reject; MISS = dangerous_accept. Joint rates counted directly — no independence assumption.)
Yield on that traffic (fire% / precision / % of MISS):
divergence 42.6% / 8.7% / 28.6%
class list alone 79.6% / 16.3% / 100.0%
UHC alone 42.6% / 17.4% / 57.1%
UHC ∧ class 33.3% / 22.2% / 57.1%
Precision lift from adding UHC on top of class: 1.365×.
So on this cell your sign holds in part — UHC is not the main selector — but the lift is above your grid median (1.01×), inside your max (1.88×). The 0.958 in Part 6 is still only P(self-conf≥0.9|MISS) from the single-judge DF v2 dump; it is not h under real unanimity.
dev.to/zxpmail/five-comments-that-...
Two caveats that bound how far this settles anything:
What I think survives from your comment: the open question is ranking inside the stream under a budget, not a cleverer trigger definition; and wiring still beats detection. What doesn't: treating 0.958 as if it were u's twin under the same event. Plug u≈0.40 into your cell instead of 0.70 and the UHC-alone precision drops further — which is exactly why you asked for the number.
Scripts / dump:
github.com/zxpmail/blog/blob/main/...
github.com/zxpmail/blog/blob/main/...
github.com/zxpmail/blog/tree/main/...
(files: df-multiperspective-*-alexey-uhc.json)
stdlib, offline analyzer; three runs byte-identical on sha256(sorted summary)=5c0b7567c82d26c452b98b0d4a611cc772b1e90b53c94bb07472534af3c4977f
Take the reframe, it's sharper than what I said and worth sitting with: if the trigger's coverage is bounded by the reversals it's already caught, coverage-limited isn't a property of the trigger, it's the definition of the trigger. I was describing it like a constraint that might loosen later. It can't, not without changing what R is built from.
The floor-volume point is the one I'd carry forward loudest, because it reframes the whole thread. Every version discussed here, divergence, class list, UHC, some blend, has been fighting over which stream to build. pi*h*rm sitting at 4.60% and un-shrinkable by any class filter says the fight was never about which stream, it's about what happens to people once they're in one, and nothing proposed so far touches that. A perfect trigger with a bad queue is still a bad system, and your 0-of-720 under precision ordering versus 76-of-720 worse under arrival order makes that uncomfortably concrete.
zxpmail's rerun already gave you the number you were missing, u about 0.40 rather than 0.70, and your sign held on that fixture but softer than the grid median. Which leaves the honest state of the thread as: the wiring question is now more load-bearing than the trigger question, and nobody here, including me, has proposed a rank-inside-stream design yet. That's the actual open problem.
Took the reframe. Floor volume + rank-inside-stream is the load-bearing cut; wiring still beats trigger choice once the stream is over budget.
Wrote the offline suite up as dev.to/zxpmail/dt2-names-who-enter... of the series (diluted-queue acceptance, feature×time stress, Trigger∥Rank and Shadow∥Enforce). Publishing tomorrow — will drop the link here when it's live. Until then: D+T2 names who enters; budget names who gets seen; a calibrated rank line is degradable, not another tripwire.
The consolidation into L0/L1/L2/L3 is the right shape, and since the adaptive-sampling row is the one I had a hand in, let me push on where it can quietly re-open the hole it closed. Confidence-weighted sampling concentrates audits where the model is unsure. But the long-tail directional failure the fixed-percentage critique was about is not the unsure case, it is the confident-and-wrong case: high self-assessed confidence, wrong direction, made the same way every time. Weighting on confidence samples that region least, because high confidence drives the rate down, so the 5.6x efficiency is partly bought by sampling less exactly where the failure you were hunting lives. Flat sampling at least covered it uniformly. Confidence-weighting can under-cover it relative to flat, which is the original hole wearing the efficiency gain as a disguise.
The fix is already in your architecture, you just weight on it instead of on self-confidence. A model's confidence shares the blindness of the thing being audited, it is the actor grading its own certainty, so it goes quiet in exactly the region that needs eyes. Cross-layer disagreement does not. Where L0/L1 and L2 return different verdicts on the same item, you have an independent flag that does not collapse when the LLM is confidently wrong, because the code filter is not reading the model's confidence, it is reading the artifact. Weight the audit sample on that cross-layer disagreement, and keep a non-zero floor in the high-confidence region so it can never be driven to zero, and you keep most of the efficiency while closing the tail. Divergence is already your most trusted signal at L3. It should also steer where the audit looks one layer earlier, and across layers, not only inside the LLM vote.
Right — and two pieces of data back this.
First, where the failures actually live. DF v2 produced 96 MISS runs across 3 models × 20 scenarios (passes=true on a directional reversal). 92 of those 96 — 95.8% — sat at self-reported confidence ≥ 0.9, avg 0.969. The dangerous tail is overwhelmingly high-confidence, which is exactly the region
1/confidence^1.5samples least.Second, the article's production caveat tried to dodge this — "confidence" meant cross-prompt divergence (Strict/Balanced/Lenient split), not within-model variance. Doesn't change the point. P2's consistency test was 0 divergence on identical input at N=10; three prompts in the same text channel collapse the same way on the same plausible rationalization. Cross-prompt is still in-channel.
And the fix you're naming — I'd already simulated it, just didn't point §4 at it.
external-signal-sampling-test.pycompares Fixed 10% vs Part 6 cross-prompt adaptive vs a third arm: four binary external signals includingclassifier_disagree(your L0/L1-vs-L2) plusroute_changed/input_unusual/barely_passed, on a 10% baseline floor. Long-tail burst, medium signal quality: Fixed 10.0% catch, Part 6 28.5%, external-signal arm 48.8% — 1.7× over Part 6 at the same audit-rate class. The non-zero floor you asked for is already the 10% baseline. What's missing: I didn't isolateclassifier_disagreealone, so I can't say how much of the 1.7× is specifically cross-layer disagreement vs the other three signals firing alongside it.So: the 5.6× headline was a simulation under a signal that goes quiet on real failure mass, and the cross-layer signal you proposed is the right replacement. I added an Update subsection to §4 citing the cross-layer simulation (1.7× over the cross-prompt arm, 48.8% vs 28.5% catch at long-tail burst), caveated the 5.6× headline inline, and kept the original results visible — the cross-prompt numbers stand as simulation output, just no longer as the headline. The
classifier_disagree-isolated run is queued; the bundled-signal result is what's live now.Publishing the revision instead of quietly swapping the number stands on its own, separate from what the result turns out to be. A 5.6x headline that turns out to sample a quiet region doesn't damage the piece if you show the correction. It damages the piece if the correction happens off-page.
On the open question you flagged: isolating classifier_disagree matters more than it looks, because the 48.8% could be almost entirely route_changed or barely_passed doing the work, with cross-layer disagreement contributing little on top. If that's true, the fix you're crediting to cross-layer signal is really a fix from a different, cheaper signal, and the L0/L1-vs-L2 architecture gets credit it didn't earn. The ablation is four runs: each signal alone, then all pairs, on the same long-tail-burst fixture. If classifier_disagree alone gets you most of the way from 28.5 to 48.8, the cross-layer story holds. If it barely moves the needle and route_changed or barely_passed does the lifting alone, you've found a cheaper oracle hiding inside a more expensive one.
Worth checking one thing on the 92-at-confidence≥0.9 number before it becomes a fixture: is that population stable across the three models and twenty scenarios, or concentrated in one model/scenario pair? If it's one model producing most of the high-confidence directional reversals, the 95.8% is a property of that model's calibration, not a general shape you can design an escalation trigger around.
You're right on both — and the numbers cut against the cleaner story.
On the ablation. Same long-tail-burst / medium fixture as the published 28.5% → 48.8% comparison (
external-signal-sampling-test.py --ablation-only, 1000 trials). Singles and pairs, 10% floor kept:classifier_disagreealonebarely_passedaloneroute_changedaloneinput_unusualaloneroute+barely28.0% (≈ P6)CD+barely_passed35.5% (1.25× P6)So: CD alone does not get most of the way from 28.5 to 48.8 — it doesn't even clear P6. It is the best single signal, and every pair that beats P6 includes it; drop CD and the best remaining pair collapses back to ~P6. The partner doing real work next to it is
barely_passed(a margin/threshold signal, cheaper than L0/L1-vs-L2). Cross-layer earns a seat; it does not earn solo credit for the 1.7×. The bundled result stands; the causal story in §4 needs that caveat. I'll put the ablation table on-page rather than leave it in a script.On the 95.8%. Not a balanced 3×20 panel. Of 96 MISS runs: qwen3-0.5b = 77 (80.2%), gemma = 16 (16.7%), deepseek = 3 (3.1%). Top scenario DS4 alone is 34.4%. So the headline fraction is partly "qwen misses a lot, and when it misses it's confident."
What does hold as a shape, conditional on miss: qwen 75/77 (97.4%) and gemma 16/16 (100%) at conf ≥ 0.9. deepseek barely misses (1/3 high-conf). I shouldn't design an escalation trigger as if 95.8% were a stable property of "models in general" on this set — it's a property of the miss-mass we actually have, which is qwen-heavy. Script dump:
confidence-vs-miss-concentration.json.Net: publish the ablation and the concentration caveat the same way as the 5.6× correction — on-page, not off-page. Done in Part 6 §4 Update (2026-07-22): ablation table + qwen-heavy caveat; the "isolation queued" line is closed. Thanks for forcing both.
The number that matters most in that table isn't any single row, it's that classifier_disagree loses to P6 alone (24.9 vs 28.4) but is the one signal whose presence flips every combination it joins into super-additive territory: route+barely without it matches P6, add it and the pair jumps to 1.25x P6, and the full four more than doubles the best single signal. A weak solo signal that's the load-bearing one in combination isn't measuring the same thing the others measure badly, it's measuring something the others don't touch at all, and its solo catch rate undersells that by a lot.
Which is the actual argument for keeping it in the fixture even though its standalone number looks like the weak link: the standalone catch rate of a signal tells you almost nothing about its marginal value once you're stacking arms, and a fixture that drops signals based on solo performance would have cut the one doing the most real work. The number worth publishing alongside these isn't just the four solo rates and the combos, it's each signal's unique catch, the cases none of the other three would have caught alone, since that's the quantity that predicts what happens when you eventually add a fifth.
You're right — and that was the missing column.
Same burst / medium / 10% floor fixture. Coupled Uniform draw across the four solo arms: unique catch = defective caught by arm i that none of the other three would have caught alone.
classifier_disagreebarely_passedroute_changedinput_unusualSo the story that looked like "CD is the weak link vs P6 (24.9 vs 28.4)" flips on the quantity that actually predicts stacking: CD is the largest unique catcher (~3× route, ~4× input). Solo CR undersells it; a fixture that drops by solo performance would cut the load-bearing arm. Unique fire tells the same shape (CD 19.6% vs route 6.5% / input 4.9%).
Publishing unique catch next to the solos and combos now — on-page in the ablation Update. Thanks for naming the metric; the table without it was half the argument.