The Confidence Interval Was [0, 0]. That Was Not Precision.

Originally published on hexisteme notes.

This is the third note I've written about the same measurement pipeline turning up a bootstrap confidence interval of exactly [0, 0] that doesn't mean what it looks like. What's actually new this time: the pipeline has caught up with the first two notes. It now names this exact failure mode in its own documentation — and its most recent weekly snapshot still shipped thirty of these intervals anyway.

The pipeline is PAMSL, the same one behind Your Agent Telemetry Ranks Your Routing Policy, Not Your Models and Sub-Agent Metrics Are Not Comparable to Main-Thread Metrics. The snapshot here is dated 2026-08-31 — seven model epochs crossed with two roles, association-only per its own header, graded no higher than "C (exploratory)." Nothing below ranks a model. It's about what the report's Bootstrap 95% CI table still does, next to what the report's own footnotes now say about it.

Thirty rows out of ninety-five

The Bootstrap 95% CI table compares model-epoch × role cells pairwise, restricted to pairs where both sides clear n=20, resampled 2,000 times from a fixed seed (42), no p-values anywhere by design. A pair filter compares only within the same role — main against main, sidechain against sidechain — and skips any pair where both sides are already flat: median 0, interquartile range 0 to 0.

I counted the rows that survive it. Ninety-five, across six metrics. Thirty of them — just under a third — report a 95% CI of exactly [0, 0]: 12 of 22 tool_error_rate rows, 6 of 10 validation_run_count rows, 12 of 22 error_recovery_seq_count rows. None of same_file_reedit_rate, output_tokens_total, or completion_proxy land on a point.

Three concrete ones:

• tool_error_rate, claude-haiku-4-5-20251001/base×sidechain vs claude-opus-4-8/base×sidechain: Δ = 0, CI = [0, 0]. Haiku sidechain: n=267, median 0, IQR 0–0.0526. Opus-4-8 sidechain: n=2781, median 0, IQR 0–0. The intuitive case — both medians sit at zero, and one side's IQR is a literal point.
• validation_run_count, claude-opus-4-8/base×main vs claude-sonnet-5/base×main: Δ = 0, CI = [0, 0]. The first group: n=211, median 0, IQR 0–4, mean 3.9573. The second: n=61, median 0, IQR 0–0, mean 0.3607. One group runs validation an average of almost four times per thread, with real spread up to its third quartile — and the interval on the difference of medians still reads as a single point.
• error_recovery_seq_count, claude-sonnet-4-6/base×sidechain vs claude-sonnet-5/base×sidechain: Δ = 0, CI = [0, 0]. The second group: n=3478, median 0, IQR 0–1, mean 0.4963 — real dispersion, not a constant — and the paired CI is still a point.

A guard I've already written up twice

Look at the last two examples again. In each pair, one side is exactly flat — claude-sonnet-5/base×main has median 0 and IQR 0–0; claude-sonnet-4-6/base×sidechain has median 0 and IQR 0–0 — and the other side is not, with IQR reaching 4 and 1 respectively and means that are clearly nonzero. The filter's skip rule needs both sides flat to fire. One flat side isn't enough, so neither pair gets skipped. They still collapse to [0, 0], because the non-flat side's median is 0 too: at least half of its own values are exactly zero, which is a narrower condition than "the whole IQR is a point," and it's the one the bootstrap actually runs on.

I've derived this mechanism in detail twice before, on this same pipeline. In July I named it directly: a [0, 0] interval is "tie-degeneracy: so many identical zeros that every resample returns zero". Ten days later, working through this same metric — validation_run_count, means then ranging 0.60 to 4.11 across model pairs, much like the 0.36-to-3.96 pair above — I found the pipeline's existing guard sailing a pair through and degenerating anyway, and called the guard "necessary and insufficient". That's not a new finding here. It's the same mechanism, five weeks later, still doing the same thing to the same metric. An interval that cannot move isn't precise. It's just not moving.

Documented is not fixed

Between the second of those two notes and this snapshot, the pipeline changed — but not by fixing the thing I flagged.

The report's Coverage & Limitations section now carries item F14: for the two binary proxy metrics, completion_proxy and abandonment_proxy, a median-difference CI is explicitly forbidden and replaced with a proportion-difference CI, on the arithmetic grounds that a 0/1 metric's median difference can only ever be -1, 0, or 1. (That it's not much of an interval to build a statistic on is my gloss, not the report's wording.) That is a real fix — narrower than what I'd called for in July, which was checking whether the resampled statistic itself is constant, for any metric. F14 patches the binary metrics specifically. It doesn't reach tool_error_rate, validation_run_count, or error_recovery_seq_count, which is exactly where all thirty zero-width rows above come from.

It also carries item F15, translated from the original: "many cells are nearly constant (mostly 0), so the percentile bootstrap CI collapses to [0, 0] — this is not evidence of precision, but a coverage limitation of the median bootstrap under heavy ties." That is not a fix. It is a sentence in a Limitations section, naming a failure mode, attached to no specific row. The same document that carries it still prints all thirty zero-width rows above with nothing in the table itself distinguishing "genuinely nothing here" from "tie-degenerate, see the footnote."

The self-measurement problem has the same shape. The second of the two July notes found the pipeline measuring its own build-and-audit sessions and promised "the next iteration gets an explicit exclusion stratum." Five weeks later, this snapshot's own Limitations section still marks that stratum as planned for M5 — flagged, not built.

None of this is a complaint that the report is dishonest. It is unusually candid: this section alone also names same_file_reedit_rate conflating iterative editing with rework, a completion_proxy heuristic broken by the exact harness-logging convention I unpacked in the first of these two notes, 37 main and 790 sidechain threads dropped from every comparison for missing model attribution, and historical dispatch-policy versions folded into an explicit UNKNOWN stratum rather than guessed at. Candor at the bottom of a document and a clean number in the middle of a table are two different deliverables. A reader only ever sees one of them without going looking for the other.

What to do the next time you see [0, 0]

Don't read a zero-width CI as "checked, no difference." Check the same cell's IQR and mean first — they sit one column over in the Core Metrics table and cost nothing to look up. If the IQR is a point on both sides, the interval is telling the truth. If the IQR or mean shows real spread on either side while the CI still reads [0, 0] — the validation_run_count case above, median 0 next to a mean of 3.9573 — the interval isn't confirming anything. It's tie-degenerate, and this report, at least, will tell you so if you scroll to the bottom and go looking.

That last part is the part worth fixing, if you own the pipeline rather than just reading its output. A limitation that lives only in prose at the end of the document protects a reader who already knows to look for it, and nobody else. The distance between F15 existing and F15 doing something is the distance between a footnote and a flag on the row: a tied-value share printed next to every median-based interval, or a suppressed cell instead of a manufactured zero — the same gap between citing a number correctly and reading it right, just moved from a model's output to a pipeline's own table. Naming a failure mode is a start. It is not the same work as making the specific row that failed impossible to misread.

Email list for these notes: hexisteme.beehiiv.com — no issue has gone out yet, so you would be on it before the first one. No welcome sequence, no course, no upsell.

More notes at hexisteme.github.io/notes.