Originally published on hexisteme notes.
I run a small fleet of Claude-family agents on one Mac. Every session gets logged — the long interactive ones I drive by hand, and the short subagent runs those main sessions spawn. A while ago I built a measurement pipeline (I call it PAMSL) that reads those transcripts into SQLite and attributes every single turn to the model that produced it. A few thousand real agent threads later, I had the thing people keep asking me for: a table comparing the models on actual workload telemetry. Tool error rate. Re-edit rate. Output tokens. A rough "did the task finish" proxy. Five models, two roles.
The honest deliverable was a stack of refusals to conclude. This note is the walkthrough, because every trap in it applies to anyone reading their own AI usage logs.
The table everyone asks for
Here is the seductive row. On my main interactive sessions, median output tokens per thread:
| model (main sessions) | median output tokens |
|---|---|
| Claude Opus 4.8 | 404,080 |
| Claude Sonnet 5 | 27,616 |
That is nearly a 15x gap. Read naively, it says one model burns fifteen times more tokens than the other for the same work. People see a number like that and immediately want to rewrite their whole setup.
It is not a fact about the models. It is a photograph of my dispatch rules.
It measures my routing, not the models
I deliberately send big, open-ended jobs to Opus main sessions and route quick, mechanical work to cheaper tiers. The models never got assigned to tasks at random. So that 15x is exactly what my own routing policy was built to produce: the expensive tier is where I put the expensive work.
This is what "association-only" means. Every difference in the table is entangled with task difficulty, project, and time-of-week — because those are the very things I use to decide where each job goes. Nothing here is randomized, so every confounder is wide open. The table can describe. It cannot explain. I marked the whole thing Grade C, exploratory, and meant it.
Once you accept that, the job stops being "rank the models" and becomes "list every way this table will lie to you." Here are the ones I had to code around.
Trap 1: main and sidechain are different populations
Main sessions are long, interactive, and I stay in the loop. Sidechain subagent runs are usually one-shot: fire, produce, exit. Those are two different populations with different length and structure. Cross-comparing them measures the harness, not the model, so I keep the two roles strictly separate and never put them in the same comparison.
The sharpest version of this bit me on the completion proxy — a heuristic that guesses whether a thread "finished" from how its last line looks. One model's sidechains scored terribly: 92 of 99 ended on a tool_result line, which the heuristic read as "stopped mid-work." But that is just a harness logging convention for how subagent transcripts terminate. The work was fine. The metric was measuring log formatting. I threw sidechains out of the completion comparison entirely and kept it to main threads only.
Trap 2: a model name is not a stable treatment
In early July I relaunched one of the models after a change to its setup. The name on the label was identical before and after. The behavior was not. If I pool those turns under one name, I am averaging two different treatments and calling the blend "the model."
So I split every model at its version boundaries into epochs — the pre-relaunch epoch and the post-relaunch epoch are separate rows, never summed. Epoch boundaries are calendar dates, fixed in advance. A model name is marketing. A version epoch is the thing you can actually reason about.
Trap 3: zero-inflated cells fake their own precision
Most cells in the table have a median of 0 and an interquartile range of 0 to 0 — tool errors are rare, re-edits in one-shot runs are rare, so the modal thread scores zero on almost everything. When I ran percentile bootstrap confidence intervals on those cells, they came back as [0, 0].
A [0, 0] interval looks like breathtaking precision. It is the opposite. It is tie-degeneracy: so many identical zeros that every resample returns zero. The interval is not telling you the effect is pinned down to nothing; it is telling you the data is almost constant and the method has nothing to chew on. If you report those intervals as confidence, you are laundering a tie into a result.
The one interval worth reading
After discarding the degenerate cells and the cross-role comparisons, plenty of contrasts still qualified for an interval — but every one of them was a process-or-volume measure: token totals, tool-error rates, re-edit rates, recovery sequences. Gaps there mostly restate my dispatch decisions. Exactly one qualifying contrast measured whether the work actually finished rather than how big or bumpy it was: the completion proxy, main sessions, older flagship versus the relaunched model:
- Opus 4.8: 0.8793 completed (n=116)
- the relaunched model: 0.9211 completed (n=38)
- difference: -0.0417, 95% CI [-0.1375, 0.0712]
The interval spans zero. The tempting headline — "the newer model finishes more of its tasks" — dies right there, in the one place I actually had the data to test it. That is the whole essay in three numbers: the cleanest comparison I could build still refused to separate the two.
Two more traps: token math and measuring myself
Only output tokens add up. My pipeline stores input-token and cache-read totals per turn. Summing them across a thread looks reasonable and is wrong: every turn re-sends the same growing context, so those totals are the same tokens counted over and over, duplicates and all. Only output tokens are genuinely additive. Comparing "input token totals" across models compares context re-send patterns, not usage. I let only output tokens into the comparison.
I was inside my own corpus. The sessions where I built and audited this very pipeline are logged like any other work — and they land in the project stratum for the pipeline's own project, distorting exactly the numbers I was most tempted to read. Self-measurement contaminates itself. I flagged it, and the next iteration gets a dedicated exclusion stratum so the observer stops standing in the frame.
Why keep the pipeline if it concludes nothing?
Two reasons, both real.
First, drift monitoring. Even without causal claims, a stable per-turn baseline tells me when something moves — when error rates or completion proxies shift after a relaunch or a harness change. You do not need identified causation to notice a regression.
Second, and more useful: an association table tells you where randomization would pay. The cells that look interesting but are hopelessly confounded are a shopping list for the experiment worth actually running. I pre-registered a separate randomized dispatch experiment aimed at precisely the cells this table lit up. Observational telemetry is a hypothesis generator, not a verdict. Its job is to point at the question, not to answer it.
That is also why I keep the pipeline honest about being a measurement lab, not a scoreboard.
How to read your own AI telemetry without lying to yourself
Five refusals I now apply to any table pulled from usage logs:
- Refuse to cross-compare populations you route differently. Interactive and one-shot, main and sidechain, cheap tier and expensive tier — if a rule sends work there, the split is your policy talking, not the model.
- Refuse to pool a model across version epochs. Same name before and after a change is two treatments. Split at the boundary.
- Refuse to read [0, 0] as precision. On zero-inflated data a collapsed interval is a tie, not an answer.
- Refuse to sum counters that are not additive. Only output tokens add up; input and cache-read totals are the same context re-sent.
- Refuse to call association causal. Nothing was randomized, so the table generates questions and settles none — and throw out any stratum your own tooling sessions polluted.
None of this ranks the models. That was the point. The most honest thing my measurement pipeline produced was a precise account of what it could not yet know.
More notes at hexisteme.github.io/notes.
Top comments (2)
The confounder analysis here is the honest work that most LLM benchmark comparisons skip entirely. The routing policy embeds task difficulty as a hidden variable, so every metric in the table is a joint property of the model and the dispatch rule, and pulling them apart without randomized assignment is exactly as hard as any observational study. The Grade C annotation is the right call and should be the default framing for any telemetry comparison where the task mix wasn't held constant across tiers. The practical thing this kind of analysis does produce, even without causal identification, is a forcing function: if you had to explain every high-variance metric in the table by routing decisions alone, which ones would strain that explanation?
The refusal to conclude is the part I trust most here. Once routing is policy-driven, the useful metric is probably not model A versus model B, but which decision rule sent work there and what artifact came back. I’d almost want every row to carry the router reason next to the token count.