Originally published on hexisteme notes.
I had an LLM telemetry report with the usual reassuring furniture: medians, interquartile ranges, sample sizes, and bootstrap intervals. The rows were grouped by model. It looked ready for comparison.
It was actually several different studies sharing a table.
The core process metrics were attributed to model epochs inside threads. The completion proxy existed only at thread level. Main sessions and sidechains had different structures. Mixed-model threads could contribute to one table but fail the purity rule for another. Historical routing policy was often unknown, and task family was not observed at all.
The dangerous mistake was no longer simply calling an association causal. It was treating every n beside a model label as if it counted the same kind of thing.
For readers auditing an agent harness, hexisteme/hard-gate-hooks contains two MIT-licensed Stop-hook examples, their tests, and a read-only scanner. They are adjacent implementation examples, not the telemetry instrument described here.
A model column is not an analysis unit
For the core metrics, attribution happened inside a thread. A multi-model thread could produce separate epoch rows because each assistant turn was assigned to the model epoch that produced it. Tool-error rate, re-edit rate, validation runs, recovery sequences, and output tokens therefore described epoch-attributed portions of work.
The completion proxy had a different contract. It was computed once per thread, only for main threads with model purity at or above 0.9, and censored threads were excluded. The same model label could therefore have one sample size in the core table and a smaller one in the proxy table without either count being wrong.
That distinction changes what a sentence is allowed to say:
- An epoch-attributed error rate describes the observed portion of work assigned to that epoch.
- A thread-level completion proxy describes eligible, sufficiently pure main threads.
- Neither can silently stand in for “tasks completed by this model.”
Putting the values in neighboring sections does not make their denominators interchangeable. Before reading a difference, I now ask what one row represents: a turn-attributed epoch fragment, a whole thread, a delegation edge, or something else.
Role changes the meaning of the metric
The report kept main sessions and sidechains separate. That was necessary because they do not end the same way and do not carry the same amount of interaction.
A main thread can contain repeated reads, edits, recovery, and user turns. A sidechain is often a bounded delegated run. A re-edit rate in those two populations mixes model behavior with session structure. Calling the gap “rework” would add another unsupported interpretation: the metric sees repeated edits to a file, but it cannot tell productive iteration from repair.
The completion proxy made the role mismatch even sharper. One sidechain harness commonly ended on a tool_result line. In one recorded cell, that happened in 92 of 99 threads. A last-line heuristic could read those endings as incomplete even when the delegated work had finished. The report therefore excluded sidechains from completion comparison instead of repairing the headline with a caveat.
That is the right direction for an invalid metric: narrow its valid population. A warning below a cross-role chart does not undo a denominator that never meant the same thing across roles.
Epoch boundaries belong in the key
A model name can survive a relaunch, configuration change, or routing change. The treatment does not.
The report split pre-relaunch and relaunch observations into separate model epochs and refused to pool them. This was more than a naming preference. Sequence metrics were calculated within each epoch fragment, so an edit performed by one model and validated after a model switch was not credited as a within-epoch validation sequence for either side.
That limitation is useful because it is visible. Pooling both epochs under the same display name would create a larger sample by erasing the boundary that explains why the sample is heterogeneous.
The practical key for a row is therefore at least:
model_epoch × role × analysis_unit
Add policy and task family only when those fields are actually observed. A friendly model name is presentation. It is not a stable experimental identity.
Missing strata do not become controls
The report also grouped rows by a source-stamped dispatch-policy hash when that evidence existed. Many historical sessions had no known policy version. Those rows remained UNKNOWN; they were not backfilled from the current router or guessed from which model happened to run.
Task family was stricter still. Every populated row in that stratum was marked NOT_OBSERVED. The correct output was an empty comparison, despite thousands of rows elsewhere in the report.
This is the point where a telemetry system proves whether it is an instrument or a story generator. A missing assignment variable is not a neutral baseline. It is an unmeasured confounder. The table may still describe the sample under the routing policy that produced it, but it cannot claim to have held task family or policy constant.
Bootstrap the metric you actually defined
The interval calculation also had to follow the unit contract. Continuous process metrics used a difference in medians. Binary completion proxies used a difference in proportions. Both cells needed enough observations, and comparisons stayed within the same role.
Several zero-heavy cells still produced a bootstrap interval of [0, 0]. That did not mean the effect was known exactly. When most resamples contain the same tied value, the percentile bootstrap can collapse because the statistic does not move. The interval is describing the resampling behavior of a nearly constant cell, not granting the measurement infinite precision.
An interval inherits every limitation of the metric beneath it. It cannot repair a proxy that changes meaning by role, a pooled epoch, or a policy field that was never observed.
The comparison contract I keep beside every table
Before I let a telemetry row influence routing, I record these fields with it:
- population: which threads or fragments were eligible;
- unit: what one observation represents;
- role: main, sidechain, or another session structure;
- epoch: the version boundary used for attribution;
- assignment evidence: the source-stamped dispatch policy, or
UNKNOWN; - metric semantics: process count, rate, additive total, or outcome proxy;
- exclusion rule: censored, mixed, unattributed, or below the comparison threshold;
- interpretation limit: the condition that would make the apparent difference disappear.
That last field matters most. In routed production telemetry, an observed gap can reverse after task family, project, time, or policy is held fixed. The table is useful for monitoring drift and choosing where a controlled experiment would pay. It is not a model leaderboard.
As of the 2026-09-14 source snapshot, the table remains association-only. The testable prediction is that at least some apparent model gaps will shrink, disappear, or reverse after recorded policy, role, task family, and analysis unit are held fixed. That prediction is invalidated if adequately overlapping matched cohorts preserve the same gaps in direction and practical magnitude. The trigger is the first source-stamped task-family cohort large enough for that comparison. Persistence would justify a randomized dispatch experiment; only randomized assignment could support a causal attribution to the model.
When you look at your own LLM telemetry, does every n count the same kind of thing?
Email list for these notes: hexisteme.beehiiv.com — no issue has gone out yet, so you would be on it before the first one. No welcome sequence, no course, no upsell.
More notes at hexisteme.github.io/notes.
Top comments (3)
We had the same table problem with latency metrics: the p95 column looked like a clean model comparison until we noticed half the rows were cache-warm calls and half were cold, and the split wasn't even across models because routing had changed mid-experiment. The "what does one row in this table represent" question is the one that has since become a standing item in our review checklist before we read any difference between model labels. Your purity filter at 0.9 is a sharper version of what we ended up doing by hand, which was just throwing out any thread that had a routing switch in it and hoping we weren't losing too much data in the process.
"Several different studies sharing a table" is a precise diagnosis and I have not seen it put that way before. The failure is more insidious than a bad causal claim, because everything about the presentation signals rigour — medians, IQRs, bootstrap intervals — while the n beside each row counts a different kind of thing.
Mixing thread-level and epoch-level units is the one I would expect most agent-harness dashboards to have, since the completion proxy almost always lives at a coarser granularity than the process metrics.
The defensible fix is unglamorous: state the unit of observation for every column, and refuse to put columns with different units in one table even when that makes the report uglier.
The [0,0] bootstrap cell and a 13.3% mutation kill rate are the same disease. I had 114 tests passing 98 over real data — looked fine. Mutation testing said the suite was counting "didn't crash," not "verified correct."
Your "narrow its valid population, don't caveat" is the fix I needed. Real data is now a smoke layer; primary evidence is synthetic with hand-derived expected values.
One addition: generating the missing stratum has its own UNKNOWN trap. If the expected value comes from what the code currently outputs, you've locked the bug in as the baseline. The oracle has to be outside the system under test — same discipline, other direction.