Sub-Agent Metrics Are Not Comparable to Main-Thread Metrics

Originally published on hexisteme notes.

I run a small fleet of coding agents on one machine. Every thread ends up in a log, and a measurement pipeline reads those logs into a database, attributing each turn to the model that produced it. After a few thousand threads I had the table people keep asking for: seven model-versions, five behavioural metrics, real workload rather than a benchmark.

Then I printed one cross-tab I had been skipping, and most of that table stopped meaning what I thought it meant.

The cross-tab was role × model. In this fleet a model runs in one of two roles. It is either the long interactive main thread I drive by hand, or a short one-shot sub-agent that a main thread spawns, runs once, and discards. Same model. Same weights. Two completely different jobs.

Role turned out to move the numbers by up to 135x, and the role mix is wildly different for each model. Those two facts together are enough to make a pooled comparison manufacture a large gap that exists in neither stratum.

A note on labels: model names are replaced with letters on purpose. The point of this note is that these numbers are not a model ranking, and printing the names invites exactly that misreading. Every figure is real, from one snapshot of one fleet.

The role gap is an order-of-magnitude thing

Median output tokens per thread, same model, split by role:

model	main n	main median	sub-agent n	sub-agent median	main ÷ sub
A	200	478,238	2,761	6,212	77x
C	88	427,838	156	8,784	49x
D	11	185,534	122	1,371	135x
G	21	128,415	18	11,358	11x
B	20	16,432	1,338	14,157	1.2x
F	15	4,194	6	51,562	0.08x

The behavioural metrics are worse than lopsided — in one stratum they are flat:

metric	main-thread medians	sub-agent medians
same-file re-edit rate	0.40 / 0.44 / 0.50 / 0.53 (four models)	exactly 0 for six of seven
error-recovery sequences	1 / 2 / 2 / 2 (four models)	exactly 0 for six of seven
validation runs	0 for six of seven	0 for all seven

The "(four models)" qualifier is load-bearing, so let me not hide behind it: the other three model-versions sit at a median of 0 in main too — two genuinely, one because its main cell holds 2 rows and is marked not comparable. Main is where between-model signal can live, not where it always does.

Still, the pattern is not a subtle covariate. A main thread iterates: read, edit, re-edit the same file, hit a failure, recover, run a check. A sub-agent is fire-and-forget — it does its one job and exits, so it rarely touches the same file twice and rarely has a failure to recover from. The metric is structurally near-zero there.

Which means: for re-edit rate and recovery count the sub-agent stratum carries no between-model signal at all — the median is a constant. All the signal lives in the main-thread stratum, 7% of my rows.

The mix is different for every model

Here is the cross-tab I should have printed on day one:

model	main rows	sub-agent rows	main share
E	2	132	1.5%
B	20	1,338	1.5%
A	200	2,761	6.8%
D	11	122	8.3%
C	88	156	36.1%
G	21	18	53.8%
F	15	6	71.4%

Across the whole attributed corpus, 4,533 of 4,890 rows — 92.7% — are sub-agent rows. So a pooled number is mostly a description of sub-agents. But how mostly ranges from 1.5% main to 71.4% main, a 48-fold spread in composition.

The reason is not random sampling — it is the delegation policy. My orchestration rules send mechanical fan-out work to cheaper tiers, so those models accumulate sub-agent rows by the thousand; models I drive by hand accumulate main rows. The edge counts show it directly: model A spawned 1,788 sub-agents that were also A and 730 that were B; model C spawned 393 that were B. Role is assigned by the same policy that assigns the model: the confounder is baked into the architecture, not introduced by chance.

Anything with an orchestration layer has this shape — retry tiers, canary vs steady-state traffic, batch vs interactive queues, free vs paid users. The router picks both which variant handles a request and what kind of request it is.

The pooled number can reverse

Pool A and C across roles, weighted by row counts — exactly what GROUP BY model gives you:

A:  (200 × 684,639  +  2,761 × 13,691) / 2,961  =  59,010
C:  ( 88 × 565,678  +    156 × 13,913) /   244  = 212,910

Pooled, C burns 3.6x the output tokens of A — the headline you would ship.

Now look inside each stratum:

stratum	A mean	C mean	C ÷ A
main	684,639	565,678	0.83x
sub-agent	13,691	13,913	1.02x

Within each stratum the two models are close. C is lower than A on main threads (0.83x) and 1.6% higher on sub-agent runs — a near-tie that happens to lean the same way as the pooled number, just nowhere near its size. Neither ratio comes close to 3.6x. The 3.6x is manufactured by the weights: main threads carry 40–50x the mean output of a sub-agent run, and 36.1% of C's rows are main threads against A's 6.8%, so C's average is dragged toward the expensive stratum and A's is not.

Worth being precise about the name. This is not textbook Simpson's paradox, which needs every stratum to point one way while the pooled number points the other; here one stratum reverses and the other is a near-tie. Arguably that is the more dangerous shape — there is no meaningful within-stratum effect in either direction, and pooling still produced a 3.6x headline out of nothing. Call it an amalgamation effect: a difference in composition, amplified into an apparent difference in the metric.

The reason to walk it out is that it never announces itself. Nobody sets out to pool — pooling is what the obvious query does. The number arrives looking like a model comparison and is in fact a weighted average with different weights per model. What you measured was your dispatcher.

Stratifying is step one, not the answer

Splitting by role fixes the mix problem. It opens two more.

The metric may not mean the same thing in each stratum. My completion proxy — a heuristic that reads whether a thread finished from the shape of its last logged line — collapsed on sub-agent threads, where one model had 92 of 99 threads ending on a tool-result line purely as a harness logging convention. I unpacked that particular failure in an earlier note; what matters here is what it implies for stratification.

It implies stratification is not a repair. A metric whose value is set by logging convention in one stratum is not a noisier measurement there — it is a different measurement, and averaging it separately does not make it comparable. So the question after "did I stratify?" is "does this metric measure the same construct in every stratum?" Construct validity is per-stratum, not per-metric. Where the answer is no, drop that stratum for that metric and say so in the output.

A quieter version of the same problem: the unit can change between metrics. My behavioural metrics are per-turn attribution rows; the completion proxy is thread-level, with a censoring rule that treats threads cut right after a user turn as missing rather than failed. Model A has 200 main attribution rows and 154 usable main threads for the proxy — two "n"s in the same report, meaning different things. Label the unit next to every count, or someone will divide one by the other.

The role gap is not a constant you can adjust away. It is 77x for A, 135x for D, 11x for G, 1.2x for B — and for two thin cells the sign flips outright. If role were a fixed multiplier you could divide it out and carry on. It is not; it interacts with the model. Partly that is because a stratum holding 1.5% of a model's rows is not a random 1.5% — whatever rare circumstance put that model in that role is also selecting the kind of work it did there. Stratification buys you a list of licensed comparisons. It does not buy you a correction factor.

A [0, 0] confidence interval is not precision

Validation-run counts, main threads only. Every model's median is 0. The means are not: A 4.11, C 2.38, G 1.67, B 0.60. I ran percentile bootstrap on all six model pairs. Every one came back with a median difference of 0 and a 95% interval of [0, 0].

A ~7x spread in means, reported as exactly zero with zero width. Both are arithmetically correct: the median really is 0 in each cell and in nearly every resample, so the bootstrap distribution is a spike and the percentile interval collapses onto it. That is tie degeneracy, not precision — and read as confidence it launders a tie into a finding, in the most persuasive possible format.

The part I had not anticipated is how hard this is to guard against. My pipeline already skipped a pair when both cells had median 0 and an IQR of 0–0. Necessary and insufficient: A's IQR here is 0–4 and B's is 0–0, so the pair sails through the filter and degenerates anyway. The right condition is not "both inputs look constant" but "the statistic is constant across resamples," which you can only check on the resample distribution itself. Report the share of tied values next to any median-based interval, and refuse to publish a zero-width interval you cannot explain.

What I run before any fleet comparison now

1. Print the stratum × treatment cross-tab first, before a single metric. If the composition differs across treatments, every pooled number is a mix effect until proven otherwise.
2. Set the comparability threshold in advance. Mine is n ≥ 5 to appear at all, n ≥ 20 on both sides to earn an interval; thinner cells print as NOT_COMPARABLE rather than as a small number.
3. Pre-register cross-stratum comparison as forbidden — a written prohibition in the spec, enforced by the pair filter, not a caution someone relitigates at 1am.
4. Ask per stratum whether the metric measures the same construct. If a stratum's value is set by logging convention, drop that stratum for that metric and record why.
5. Never read a degenerate interval as a result. Zero width on tied data is coverage failure.
6. When pooled and stratified disagree, publish both and name the mix that separates them. The disagreement is the finding.

What this still cannot tell you

Routing was never randomized, so none of this is causal — the model each thread got was chosen by policy, entangled with task difficulty, project and week. Everything here is association under a fixed dispatch policy, and the role finding does not rescue it.

Role is also not the only stratum. Split model A's main threads by project and the re-edit median runs 0.34 / 0.45 / 0.53 / 0.70 across four projects — a 2x spread inside a single model-and-role cell, while the same model's sub-agent value is 0 in every project. And the re-edit metric cannot tell healthy iteration from thrash; it counts both.

Two more limits. The sessions in which I built and audited this pipeline are logged like any other work, so the observer stands inside the frame; the next iteration gets an explicit exclusion stratum. And this is one snapshot, one operator, one harness — the constants will not transfer. The procedure does: cross-tab first, threshold in advance, construct validity per stratum, no faith in narrow intervals over tied data.

The single sentence I would keep: before you compare treatments, check whether they are running in the same role, in the same proportion. In an agent fleet the answer is almost always no, and nothing downstream survives that going unasked.

More notes at hexisteme.github.io/notes.

Comments

[Zira]

The router is the missing unit of analysis here. I would make dispatch policy a first-class field in every run record, alongside role, task family, project, and model, then publish both the pooled view and a role-by-task cross-tab. Otherwise a model swap can look like a regression simply because the router sent it more fan-out work.

For the randomized slice, I would keep it small and gated: only tasks eligible for both models, fixed tool and budget limits, no production side effects, and a pre-registered primary metric. That gives you a causal comparison for that slice without pretending it generalizes to the whole fleet. The non-random traffic can still be useful, but I would label it as policy-conditioned association.

[John]

Making dispatch policy a first-class field alongside role and model is the right abstraction — it turns the router from a hidden confounder into something you can slice on. Your randomized slice design (eligible tasks only, fixed budgets, pre-registered metric) gives a clean causal estimate for that subset without overclaiming fleet-wide generality. The role-by-task cross-tab on top of the pooled view would let us see exactly where policy shifts masquerade as model regressions. Thanks for the router-as-unit framing and the gated experiment structure.

[Reid Marlow]

This is the dispatcher problem in a lab coat. I like the role-stratified view because it catches the boring confounder before the model leaderboard becomes folk wisdom. The next thing I'd want in the table is dispatch policy version, since a quiet router tweak can make last week's model comparison stale.

[John]

Calling it the dispatcher problem in a lab coat is a sharper framing than mine — it names the leakage directly. Adding dispatch policy version to the table would close the loop on router drift, which I only hinted at. That version field turns a stale comparison into a reproducible one. Thanks for the dispatch policy version addition.

[Valentin Monteiro]

"What you measured was your dispatcher" is the line I'd put at the top of this. The thing that would worry me next is that the delegation policy isn't frozen: if the routing rules shifted at any point across those few thousand threads, week is a second confounded stratum sitting inside the role one, and a role x model cross-tab won't show it. Have you considered carving out a small randomized slice of dispatch, where the router picks the model at random for a fixed fraction of tasks? That looks like the only stratum that could ever be causal rather than a very careful association.

[John]

You're right that “fixed” is doing work my data cannot defend. The post names project and week as confounders, but then calls the result association under a fixed dispatch policy. I did not stamp a dispatch-policy version onto each run, so I cannot establish that the policy stayed fixed across the window. The honest label is association under the sequence of routing policies that happened to be in force.

I did try the randomized slice in a separate track. Eligible delegations were randomly assigned between two model tiers, with a pre-registered primary metric. It ended undeterminable for two independent reasons: the assignment record could not be joined reliably to the work thread it produced, and the primary metric needed more observations than the real dispatch rate could supply. That is not evidence against randomization; it is evidence that the coin flip is the easy part.

Your version is the design I would use if I reopen it: define the eligible task class before assignment, stamp a stable task ID and policy version at the draw, hold the role, tools, and budget fixed, and set both the primary outcome and a power-based stopping rule before the first draw. Only that slice would earn causal language. Everything outside it should remain explicitly labeled policy-conditioned association, even after role and week are added to the table.