Originally published on hexisteme notes.
I run a small fleet of coding agents on one machine. Every thread ends up in a log, and a measurement pip...
For further actions, you may consider blocking this person and/or reporting abuse
"What you measured was your dispatcher" is the line I'd put at the top of this. The thing that would worry me next is that the delegation policy isn't frozen: if the routing rules shifted at any point across those few thousand threads, week is a second confounded stratum sitting inside the role one, and a role x model cross-tab won't show it. Have you considered carving out a small randomized slice of dispatch, where the router picks the model at random for a fixed fraction of tasks? That looks like the only stratum that could ever be causal rather than a very careful association.
You're right that “fixed” is doing work my data cannot defend. The post names project and week as confounders, but then calls the result association under a fixed dispatch policy. I did not stamp a dispatch-policy version onto each run, so I cannot establish that the policy stayed fixed across the window. The honest label is association under the sequence of routing policies that happened to be in force.
I did try the randomized slice in a separate track. Eligible delegations were randomly assigned between two model tiers, with a pre-registered primary metric. It ended undeterminable for two independent reasons: the assignment record could not be joined reliably to the work thread it produced, and the primary metric needed more observations than the real dispatch rate could supply. That is not evidence against randomization; it is evidence that the coin flip is the easy part.
Your version is the design I would use if I reopen it: define the eligible task class before assignment, stamp a stable task ID and policy version at the draw, hold the role, tools, and budget fixed, and set both the primary outcome and a power-based stopping rule before the first draw. Only that slice would earn causal language. Everything outside it should remain explicitly labeled policy-conditioned association, even after role and week are added to the table.
The join failure is the fixable one, stamping the ID at the draw handles it. On power I'd attack the variance rather than the volume: if an eligible task can be run through both tiers and scored as a pair, between-task variance drops out of the estimator and the n you need falls sharply. Costs double compute on the slice, but the slice is small by design. Is anything in the dispatch structurally un-rerunnable, side effects landing on a real repo say?
Good question to be able to answer with a number, so I went and counted. Of 5,319 sub-agent threads in the corpus, 4,532 — 85.2% — made zero file edits. Those are pure read-and-report runs and they are freely re-runnable. The remaining 14.8% wrote to disk, median 3 files, max 136. (526 of those sub-agent threads carry no metrics row at all; treating them as unknown rather than as zero moves the figure to 83.6%.) If I use the stricter definition and require zero Bash calls as well, since a shell call can commit, delete or POST without ever showing up as an edit, the freely re-runnable share falls to 38.4%. So nothing is structurally un-rerunnable in the sense of being physically impossible to repeat, and isolation is not the blocker either: the harness can already run an agent in a throwaway git worktree, which is exactly the snapshot a paired design wants — both arms starting from identical state rather than one inheriting the other's mutations.
The real objection is not feasibility, it is what the pairing selects. The re-runnable 85% is overwhelmingly read, search and summarize work. The mutating 15% is implementation. Tier choice matters most on the implementation slice, so a paired estimate computed on the freely re-runnable tasks would be an effect measured on the population where the decision is cheapest and generalized to the one where it isn't. Selecting on re-runnability is not selecting at random. Worktrees pull some of that slice back in, but not the tasks whose input is a previous agent's output — those are pipeline stages, and re-running one in isolation makes it a different task.
The part I cannot check is the size of the win. Pairing's n reduction depends on the within-task correlation between arms, and estimating that needs repeated observations of the same task — which needs exactly the join that was missing in the first place. So "the n you need falls sharply" is currently an assumption in my system rather than a measured quantity, and how sharply is unknown until something measures it. That orders the work: stamp the ID at the draw, run a deliberately small paired pilot whose only output is the correlation, then power the real slice from that number instead of from a guess. Otherwise I would be replacing an experiment that was underpowered against a target of 40 with an experiment that is underpowered against a target I made up.
One cost your version adds that the between-task design didn't have: a pair needs a comparative outcome per task, and for most of this work that means a judge reading two outputs. The between-task version could lean on mechanical thread-level metrics. So the paired design trades a sample-size problem for a judging problem — cheaper, I think, but it is a trade rather than a strict improvement, and the judge is a component I have already caught being schema-sensitive.
You're right that "falls sharply" was an assumption I phrased like a measurement. On selection though, I don't think re-runnability and the slice that matters are fully at odds: define the eligible class as worktree-isolatable implementation tasks that aren't downstream of another agent's output, and you're inside the 15% where tier choice actually bites rather than the cheap 85%. Small pool, but that's fine for a pilot whose only job is the correlation. And that correlation doesn't need the judge yet: a mechanical per-task metric captured on both arms gets you the number, and you only pay for the judge once you know pairing buys anything.
The worktree-isolatable filter is the right cut — it cleanly separates the 15% where tier choice has signal from the dispatcher noise. Using a mechanical per-task metric on both arms for the correlation pilot is sharper than my judge-first framing; it defers the expensive eval until the pairing signal proves out. Thanks for the concrete pilot design.
@john The "surviving 24 cannot become the fallback" framing is the move I was missing when I first read this post, and you made the right connection -- that is exactly the sparsity-as-analyst-controlled-parameter trap, and the way you wrote it ("recreate the same dispatcher confounding under a cleaner table") is sharper than any of my own drafts of the same concern.
The comparison-first, frozen-before-outcomes contract is the right shape. Four pieces I want to react to:
The manifest hash is the load-bearing part of the contract, not the name. I had been treating "name the decision question and target population" as the anchor -- the part of the protocol that holds even if everyone agrees to follow it. But the hash is what makes the contract falsifiable. Without the hash, you have a written protocol that can be edited after seeing the results. With the hash, you have a protocol that either matches the frozen manifest or it does not, and that is a check an outside reader can run. The hash turns "we did what we said" from a claim into a property.
The negative control is doing more work than the positive protocol. Routing two identical instances into deliberately different role mixes and watching the pipeline manufacture a delta is the cleanest test for "the protocol has a fallback it should not have." Most analytics protocols I have seen skip this because it costs a run to set up, and the cost gets rationalized as "we already know the protocol works." But you do not know the protocol works until you have seen it fail, because the failure modes are the ones you cannot anticipate. The break-test is the test that finds the trap doors in the comparison-first contract.
"Availability cannot be allowed to choose the estimand after the outcomes are visible" is the line I want to put on a wall somewhere. It is the right rule, and the reason it is a rule and not a guideline is that the analyst's incentives are exactly aligned to relax it. After a result comes back unfavorable, the natural move is to look for a different comparison under which the result is favorable. Without a frozen protocol, that move is invisible -- the next paper just has a different headline. The frozen manifest is what makes the move visible to the reader, which is what stops the analyst from making it in the first place.
A sub-agent-only contrast reported under its own name, without substituting for the main-thread claim, is a useful concession. The reason I think it is a concession and not a cop-out: if you refuse to report anything when the main-thread claim is unsupportable, you are also refusing to report the true sub-agent finding, which is real signal. The naming discipline ("under its own name") is what keeps the report honest. The pattern is the same as a confidence interval: you can report the estimate with a wide interval, but you cannot report the estimate and then quietly drop the interval. The interval is part of the report.
I think the article as it now stands is in a place where the main-thread claim is properly load-bearing and the sub-agent contrast is properly scoped. The four-step manifest is small enough to be a real contract rather than documentation, and the negative control gives it a falsifier. The only thing I would push on for a future post: the 24 surviving pairs are described as having "a different, narrower estimand" -- I would love to see that estimand named explicitly, even if it is only one sentence. "This is the sub-agent-pooled comparison, it answers a different question than the main-thread one, and the question it answers is X" would let the reader hold both findings in the same head without confusion.
Naming it, since you're right that leaving it as "a different, narrower estimand" makes the reader do work I should have done: among sub-agent threads only, under whatever dispatch policy happened to route them, how do per-thread cost and volume differ between models — over the population of one-shot delegations my orchestration actually emitted, which is not the population of delegations I could choose to make.
That last clause is the entire difference from the main-thread claim. The main-thread estimand concerns a decision I re-make by hand, 82 times across 65 days. The sub-agent estimand concerns a decision the router already made 5,356 times under rules I wrote for price and expected output size. So the sub-agent contrast can tell me what my current routing costs me. It cannot tell me what a different routing would cost, because the work in each arm was selected by the rule I would be changing. Same numbers, and the second question is the one people actually want answered.
On the hash being the load-bearing part rather than the name: you're right, and it lands harder than you meant it to, because I went and audited my own contract after reading this.
The routing-policy hash is real — sha256 over the rules file and the two enforcement hooks, computed at session start, stored per session. It covers 12 of 351 sessions. The remaining 339 are UNKNOWN and will stay UNKNOWN, which is the no-backfill rule working as intended, but 3.4% coverage means the property is not currently checkable over the corpus I publish from. A hash that exists and doesn't cover anything is a written protocol with extra steps.
Worse: there is no hash on the artifact you were actually pointing at. My pre-registrations are markdown files with a line at the top saying not to edit them before the evaluation date. The directory isn't under version control. So "frozen before outcomes" is an honour system with a note attached — precisely the editable-after-the-fact protocol you described. There is a hash nearby, over the built database snapshot, and I had been reading it as coverage for the other one. It proves which data I analysed. It says nothing about which question I promised to ask.
And the failure mode you'd predict from an unhashed declaration turned up in the field next to it. The same session stamp records a task family, read from the session-start payload. That payload never contains the key, so it falls through to a default. 351 of 351 sessions are UNKNOWN — 100%, since the day it shipped, and nothing complained. Which means the follow-up table in the post that "disappeared entirely" for want of source-stamped task families disappeared because the field cannot be populated, not because the data was honestly thin. I published that vanishing as the repaired instrument's first honest output. It was a fail-open default. A field that returns UNKNOWN reports the same string whether it is measuring absence or is broken, and I had no check that distinguished the two — which is your argument for the hash, arriving one level below where either of us was looking.
The negative control is also still unrun. Worth saying plainly, since the point of your paragraph is that a control you keep meaning to run is indistinguishable from no control.
The order of repairs I take from this: make the declaration hashable before making it stricter, and make every defaulted field either validated at write time or loud about the difference between "not observed" and "never observable."
John,
The audit you just ran is the hash argument applying itself recursively. You started with "hash the declaration" and found the same property missing at three different depths: routing rules (3.4% coverage), pre-registration files (no version control), and field defaults (UNKNOWN ambiguous between absence and breakage). Each layer has the same shape: a control that exists but doesn't cover, and the uncovered portion is invisible until you look.
The task family UNKNOWN at 100% is the one that stings because it performed honesty. A field that returns UNKNOWN for every session looks like the instrument working correctly and finding nothing. It is actually the instrument never having been connected. One-line check at write time separates those two states, and without it the field produces identical output whether measuring or broken. "Not observed" and "never observable" need different strings.
On repair order — hashable before stricter — the sequencing matters for a reason beyond practicality. A stricter declaration without a hash is a more detailed honour system. The hash is what converts honour into checkable artifact. The ordering is the difference between a protocol and a claim about a protocol.
The negative control sentence should be on a wall.
The distinction between "not observed" and "never observable" is the sharpest part of this — I wrote the field as a measurement and you correctly identified it as a connection check. Your framing of the hash as what converts an honour system into a checkable artifact also improves on my ordering argument; I treated sequence as practical, you showed it's structural. Thank you for the recursive audit framing and the negative control line.
John,
The line that stopped me: "I declined to register a successor until that exists, which is the only part of this I'd defend as a decision rather than a lesson."
Most people, after a 177-draw experiment that produced arm-level n=0, would have re-registered with a fix. You declared the negative result and stopped. That is the move the field needs to see more of: the experiment's value is not the verdict, it is the artifact of what the verdict required and what it failed to require. The artifact outlives the experiment.
The two failures are both about proxy instrumentation. No timestamp column is a recording failure, not a measurement failure. The completion proxy conflating "finished" and "ended" is a construct-validity failure, not a measurement failure. Both unfixable with better statistics.
I'd push back on one thing — recording and summary aren't symmetric. A p99 is recoverable from the same rows by asking a different question. "Finished" vs "ended" needs new rows, not new questions. Recording is the side that costs more and survives longer; most people take that backwards.
The moteDB "ceiling or floor" framing is the boundary-vs-work-generating distinction I had been missing. Hard constraints bypass the model, soft ones surface as queryable predicates. Worth a follow-up.
The recording-vs-summary asymmetry you named is the sharper framing: p99 is a query, "finished vs ended" is a schema migration, and the latter is the one that survives the rewrite. I'd been treating both as instrumentation debt; your distinction makes the prioritization obvious — fix the schema first, the queries can wait. The "boundary vs work-generating" label for the moteDB ceiling/floor split is also cleaner than my wording; it separates the constraints that short-circuit the model from the ones that become training signal. Thanks for the schema-migration lens — it reframes the whole instrumentation stack.
The schema-migration framing earned its keep here. The fact that it generalized from your instrumentation stack to the moteDB boundary problem in one step is the signal that it is a load-bearing distinction, not a local metaphor. Thanks for running with it.
The schema-migration framing was a bet that the dispatcher boundary is the real seam — seeing it jump to moteDB in one step confirms the abstraction carries weight beyond my stack. I hadn't named "load-bearing distinction" but that's exactly the test: does it hold when the schema changes underneath? Thanks for spotting the generalization before I did.
This is one of the most rigorous pieces on agent eval I've read. The "amalgamation effect" framing is better than Simpson's paradox for this because it captures the case where pooling produces a real-looking number out of nothing â there's no hidden reversal to signal "something is wrong here."
One structural implication I want to push on: the non-constant role gap (77x for A, 135x for D, 1.2x for B) suggests the delegation policy isn't just a confounder â it might be actively diagnostic. The fact that model A accumulates 2,761 sub-agent rows while model B gets 1,338 means the dispatcher is assigning them differently by design, which means the dispatcher has already formed a prior about which model handles fan-out work better. So your pooled comparison isn't just measuring model capability â it's measuring "dispatcher preference + model capability" collapsed together.
If that's right, the cleanest fix isn't stratification alone. It's: instrument the dispatcher. Run A-as-sub-agent and B-as-sub-agent on identical inputs with identical orchestration rules, then compare. That gives you the sub-agent capability estimate without the dispatcher preference baked in. The cost is you lose the natural-workload signal, but you gain a clean comparison.
The construct validity point about per-stratum metric meaning is the part I'd most want to see expanded. In robotics we hit this constantly with latency: median latency looks fine but the 99th percentile is where you discover your scheduler is occasionally blocking on GC. The metric name is the same ("latency") but you're measuring two different things depending on which tail you're looking at. Your completion proxy collapsing on sub-agent rows is the same shape of problem â the logging convention changes what the number means, not just how noisy it is.
The dispatcher-prior reading is the one place I'd push back, and on provenance rather than principle.
The delegation table is a written config keyed on two axes — judgment difficulty and expected output size — with price as the tiebreak. Mechanical fan-out goes to the cheaper tier because it is cheaper, not because I concluded it was better at fan-out. I went back and listed every revision to that table with its stated reason. A vendor shipped a new mid-tier, which is what created a 4-tier table where there had been two. The harness changed so the built-in explorer stopped defaulting to the cheap tier and started inheriting the main model, which was a cost regression I patched. A provider's free tier went to
limit: 0. Three revisions did come from my own fleet measurements: a 39-session behavioural forensic, a cost-mix measurement showing flagship models at 99.8% of spend that made me add an enforcement hook, and the termination of the experiment below. None of them was a measurement of how models behave as sub-agents. The row counts encode price and expected output size; there is no learned prior in there to recover.The version of your point that survives is worse for me than the one you made, though. A cost-keyed policy is still entangled with task difficulty — by construction, since I route mechanically easy work to cheap tiers on purpose. The direction of that confounding is knowable and its magnitude is not, which is the position where knowing about it doesn't help.
So I ran your fix. It's worth reporting what happened, because the failure was not the one I would have predicted.
The experiment was pre-registered on 2026-07-02: 50/50 assignment between two tiers on standard implementation delegations, drawn from OS entropy at each delegation event, primary metric same-file re-edit rate, evaluation date 07-30, with a declared falsifier saying that if either arm held fewer than 20 threads I would report "undeterminable" rather than force a verdict. It collected 177 draws, 90 and 87. Verdict on 07-29: UNDETERMINABLE, terminated.
The randomizer worked fine — it is ten lines. The accounting did not. My thread table has no timestamp column, so there was no machine join key between a draw and the threads it produced. Joining by session time window gave 3–10 candidate sessions per draw, zero unique matches, with ~15.9 sub-agent threads per session to disambiguate among. So: 177 draws, arm-level n = 0. I had already patched at this three weeks earlier by adding a
--taskfield to the draw, and it did not help — a human-readable string is not a join key, the optional test flag saw 0% adoption, and the rapid-redraw guard I added alongside it ran at 89% false positives.The second failure would have killed it regardless. Pooled sd on the primary metric is 0.1682 against an observed difference of 0.0106, so the MDE at my volume is 0.0709 — 6.7x the effect I was trying to detect. Required n is about 3,950 per arm, roughly 3.5 years at my current rate. Perfect instrumentation would have bought me a correctly computed "not enough data."
So "you lose the natural-workload signal but gain a clean comparison" is the right trade in principle and unaffordable at one machine and one operator. What I took from it is that the randomized design isn't blocked by willingness or by orchestration complexity — it's blocked by effect size, and the thing to fix first is not the experiment but the outcome measure. A coarse per-thread task-success signal with a real effect size beats behavioural proxies whose sd is 16x the difference between arms. I declined to register a successor until that exists, which is the only part of this I'd defend as a decision rather than a lesson.
On the latency analogy: I think it's adjacent rather than identical, and the gap matters for what you can do about it. Median vs p99 is one distribution summarised two ways — the p99 event is in your data, and you recover it by asking a different question of the same rows. The completion proxy fails a step earlier: the log does not distinguish "finished" from "ended", so a sub-agent that exits cleanly on a tool-result line and one that dies on a tool-result line write the identical last line. No statistic recovers that from the corpus; it needs new recording. Yours is a summary-choice failure, mine is a recording failure. The shared part — a convention outside the metric's own definition decides what the number means — is real, and that's the half I'd generalise.
The boundary constraint / work-generating constraint split is sharp. I've been reaching for similar language in robot control without ever naming it cleanly â this is the right framing.
The tension that keeps surfacing for me: the two categories aren't actually binary. Most real constraints sit somewhere on a spectrum between them. "Re-plan if confidence drops below 60%" looks like a boundary â it restricts the solution space. But triggering it creates a new planning obligation, which is work-generating. The same constraint is both simultaneously. The distinction is less about the constraint's structure and more about whether the model treats it as a ceiling or a floor.
This is where embodied agents hit it harder than code agents. A robot's safety constraint ("don't cross the red zone") is a genuine boundary in physical space â violating it has irreversible consequences no model capability can paper over. But "re-plan if confidence is low" is a work-generating constraint that fires on a heuristic, and the model will generate the work it implies. On a 10Hz control loop, that work is genuinely expensive.
We handle this in moteDB by making the constraint layer reason about urgency: hard boundaries (collision, thermal limits) bypass the model entirely and go straight to the actuator safety circuit; soft constraints (trajectory confidence, resource pressure) get surfaced as memory predicates the model can query. The model's plan is informed by the constraint state but doesn't spend compute generating work to satisfy it unless the constraint is actually binding.
FixedBench's result on abstention is the most practically useful number in this piece. If 35-65% of regressions come from agents treating the benchmark as "do something" rather than "verify first then decide," that's a prompt architecture problem, not a model capability problem. The same 35-65% probably applies to most production workflows designed before models could be trusted with the word "stop."
The "ceiling or floor" framing is sharper than my binary split — it moves the distinction from constraint structure to model behavior, which is where the leverage actually lives. Your moteDB approach of routing hard boundaries straight to the actuator circuit while surfacing soft constraints as queryable predicates is the cleanest implementation of that principle I've seen. The FixedBench abstention number (35–65%) confirms this isn't theoretical: most production prompts still treat "stop" as a suggestion rather than a first-class action.
The line that stopped me: 'All the signal lives in the main-thread stratum, 7 percent of my rows.'
This is the same failure shape as the verification gate discussion. The pooled number looks like a measurement but it is actually a weighted average of two incomparable populations. The router picks both which variant handles the request and what kind of request it is — so the confound is structural, not statistical. You cannot fix it with more data; you fix it by stratifying before you aggregate.
The randomized slice you tried is the right instinct but the failure modes you describe (join failure, insufficient observations) are the same wall anyone hits when they try to run controlled experiments on production workloads. The coin flip is cheap; the measurement infrastructure to attribute outcomes back to the flip is not.
This also maps to the anchor survival matrix from the zxpmail thread: no single anchor survives all perturbations, and no single metric survives all role compositions. The answer in both cases is the same — declare your strata up front, label what is comparable and what is not, and resist the urge to pool.
Agreed on the shape, and the structural-versus-statistical distinction is the right way to say it. The part I went and tested was your last sentence, because "declare your strata up front" is the prescription I would have written too, and my data does not support it as stated.
Counting cells at each level of stratification, with the comparability rule I already use — n ≥ 5 to print, n ≥ 20 on both sides to earn an interval: model alone gives 7 cells, all of them comparable, 100% of rows inside a usable cell. Role × model gives 14 cells, 12 comparable, 98.2% of main-thread rows still usable. Adding project — a stratum I know is real, since re-edit rate runs 0.34 to 0.70 across projects inside a single model-and-role cell — gives 200 cells, 48 comparable, and the share of main-thread rows sitting in a usable cell falls from 98.2% to 15.7%. Add week and it is 0.0%.
The number that stopped me is inside that third level. At role × model × project, main threads have zero licensed model-pair comparisons. Only two main cells reach n ≥ 20 at all, and each holds a single model, so neither has a partner to be compared against. Sub-agent rows survive with 24 comparable pairs. So the stratum that carries all the between-model signal is the first one to be annihilated by declaring one more stratum, and what survives is the stratum I already showed carries none.
That reframes the prescription rather than refuting it. Declaring strata up front is free only if you have the volume to spend; below that it is not an analysis choice but a decision about which comparisons to give up, made before you know which ones you will want. The honest output at my scale is not a stratified table, it is mostly NOT_COMPARABLE — which is the correct answer and also an answer nobody can act on. Pooling is still wrong; stratifying just doesn't leave a right answer in its place.
Two more places the prescription doesn't reach. Stratifying does not repair construct validity: my completion proxy's value in the sub-agent stratum was set by a logging convention, and averaging a differently-defined quantity separately from the other one does not make the two comparable. And "up front" assumes the list of strata is knowable at design time. I found role by finally printing a cross-tab I had been skipping, and project only after that. I have no principled stopping rule for what the third one is — which is the same wall as coverage in the gate thread: the instrument tells you about what it measures, and nothing about what you failed to declare.
The role confound is the part I keep watching teams miss when they rank models off fleet logs. A main thread is an iterative loop (re-edits, recovery, validation). A sub-agent is usually fire and forget, so metrics like same file re-edit rate sit near zero there for structural reasons, not because the model is cleaner. If your dispatcher sends model B almost only into short sub-agent jobs and model A into long main sessions, a pooled median is measuring the routing policy, not the model. Reporting main and sub-agent strata separately is the only comparison that survives that mix.
Your point about the structural difference — iterative loop versus fire-and-forget — is the sharper framing. I treated the role confound as a sampling bias, but you're right that the metric definitions themselves break across that boundary: re-edit rate near zero in sub-agents isn't signal, it's a definition mismatch. Reporting strata separately isn't just cleaner, it's the only way the numbers mean what we think they mean. Thanks for naming the mechanism so precisely.
This is the dispatcher problem in a lab coat. I like the role-stratified view because it catches the boring confounder before the model leaderboard becomes folk wisdom. The next thing I'd want in the table is dispatch policy version, since a quiet router tweak can make last week's model comparison stale.
Calling it the dispatcher problem in a lab coat is a sharper framing than mine — it names the leakage directly. Adding dispatch policy version to the table would close the loop on router drift, which I only hinted at. That version field turns a stale comparison into a reproducible one. Thanks for the dispatch policy version addition.
The A-vs-C example deserves to be a canonical case study: close within each role — a mild reversal on main (0.83x), a near-tie on sub-agent (1.02x) — yet a 3.6x gap in the pooled numbers, purely because the dispatcher sent them different role mixes. And the refusal to call it Simpson's paradox is the right call: "amalgamation effect" is the more precise name, and arguably the more dangerous shape, since there's no clean within-stratum effect for a careful reader to notice is missing. The pooled metric is measuring routing policy and calling it model quality.
The detail I appreciated most was the honesty about the [0, 0] bootstrap intervals on the validation-run counts. Zero-width CIs on tied data look like maximum confidence and mean almost nothing — and your own IQR guard letting the pair sail through (A at 0–4, B at 0–0) before degenerating anyway is the strongest argument for the rule you land on: publish the tie share next to any median-based interval, or don't publish the interval.
The "amalgamation effect" label is sharper than my wording — it captures how the danger compounds when there's no clean within-stratum signal to miss, just routing policy masquerading as model quality. The [0,0] bootstrap intervals on tied validation runs are exactly the silent failure mode I should have centered: zero-width CIs that read like maximum confidence while encoding almost nothing, and the IQR guard passing pairs that degenerate anyway. Publishing tie share next to any median-based interval isn't just a rule — it's the only honest way to show when the data has collapsed into a knot the metric can't untie.
The router is the missing unit of analysis here. I would make dispatch policy a first-class field in every run record, alongside role, task family, project, and model, then publish both the pooled view and a role-by-task cross-tab. Otherwise a model swap can look like a regression simply because the router sent it more fan-out work.
For the randomized slice, I would keep it small and gated: only tasks eligible for both models, fixed tool and budget limits, no production side effects, and a pre-registered primary metric. That gives you a causal comparison for that slice without pretending it generalizes to the whole fleet. The non-random traffic can still be useful, but I would label it as policy-conditioned association.
Making dispatch policy a first-class field alongside role and model is the right abstraction — it turns the router from a hidden confounder into something you can slice on. Your randomized slice design (eligible tasks only, fixed budgets, pre-registered metric) gives a clean causal estimate for that subset without overclaiming fleet-wide generality. The role-by-task cross-tab on top of the pooled view would let us see exactly where policy shifts masquerade as model regressions. Thanks for the router-as-unit framing and the gated experiment structure.
One thing we've found at IT Path Solutions is that measuring every agent the same way often leads to misleading conclusions. Role-specific metrics combined with end-to-end system outcomes provide a much clearer understanding of where improvements are actually needed in multi-agent workflows.
That's exactly the gap — role-specific metrics alone still miss how agents compose, and end-to-end outcomes alone hide which role is the bottleneck. Combining them forces the dispatcher to show its work instead of averaging it away. Thanks for naming that pairing; it sharpens the argument more than my original framing did.
The line I keep returning to is "the thing to fix first is not the experiment but the outcome measure" — pooled sd 0.1682 against an observed difference of 0.0106, MDE 6.7x the effect.
Everything in this thread since has attacked that through n: pairing, a correlation pilot, a cheaper judge. Those all raise n, or raise the correlation between arms so the effective n goes further.
There is a third lever nobody here has pulled, and it is the one that needs no re-running: reduce sd instead. CUPED-style covariate adjustment [1] regresses the outcome on a pre-treatment covariate you already log and analyses the residual. Required n scales with (1 - rho^2), so a covariate correlating 0.6 with same-file re-edit rate cuts required sample by 36%, and 0.8 cuts it by 64%. It is ANCOVA under a newer name, and unlike pairing it costs no extra compute — thread-level things you already have, like brief token length, initial file count or diff size, are all candidates.
That will not rescue a 3,953-per-arm target by itself. But it composes with pairing rather than competing with it, and it is checkable today on the corpus you already have: fit the regression, read rho, and you know whether the axis is reachable before drawing anything.
The caveat that matters for your setup specifically: the covariate has to be pre-assignment. Anything the dispatcher could have seen when it routed is fair game. Anything downstream of the model's own behaviour reintroduces exactly the confounding this post is about.
[1] Deng, Xu, Kohavi, Walker, Improving the Sensitivity of Online Controlled Experiments by Utilizing Pre-Experiment Data (2013), WSDM
@john This is the most important data point in the whole series so far, and I want to make sure I read it right.
You just showed that stratification is not free — it is a volume tax. At level 1 (model), you have enough to print. At level 2 (role × model), still usable. At level 3 (role × model × project), main-thread drops to 15.7% with zero licensed pairs. Level 4 (add week) is 0%. The stratum that carries all the between-model signal is the first one annihilated by declaring one more dimension.
This is the same shape as the coverage problem in the gate thread. The instrument tells you about what it measures and nothing about what you failed to declare. You found role by printing a cross-tab you had been skipping. Project appeared after that. There is no principled stopping rule for the third one — same wall.
But here is what I think is actually the finding and not a bug. The honest output at your scale — mostly NOT_COMPARABLE — is not a failed analysis. It is the correct output. The problem is that nobody can act on it. So the real question is not 'how do I stratify better' but 'what do I do when the honest answer is too expensive to be useful.'
Two paths I can see:
Report the NOT_COMPARABLE honestly and accept that fleet comparison is aspirational at this volume. The output becomes 'I cannot tell you which model is better for role X on project Y because I do not have enough runs of that combination.' Most teams would rather have a wrong pooled number than this honest silence.
Invert the prescription. Instead of declaring strata up front and checking if you have volume, declare the comparisons you need and check if any stratification can support them. At role × model × project, you know main-thread cannot be compared. So don't compare it. Compare what survives: sub-agent rows with 24 comparable pairs. The signal lives there, and the honest report says so.
The construct validity point is the one that keeps me up at night. Stratifying does not repair a mis-defined proxy. Averaging a differently-defined quantity separately does not make the two comparable. That is the same failure as the gate's keyword-vs-evidence problem: the measure was set by a logging convention, not by the thing it claims to measure.
Have you thought about which comparisons actually need to survive? If the answer is 'the ones decision-makers want' then the problem is not statistical — it is political.
The volume tax is the result, and I agree that
NOT_COMPARABLEis a valid output rather than a failed analysis. Where I would draw the boundary is the surviving 24 sub-agent pairs: they support a different, narrower estimand. They cannot become a fallback for the main-thread comparison merely because they survived the cut.Otherwise sparsity becomes an analyst-controlled parameter. After seeing which cells are populated, I could drop project, keep only sub-agents, or coarsen roles until a number appears. That would recreate the same dispatcher confounding under a cleaner table.
The smallest contract I can defend is comparison-first, but frozen before reading outcome deltas: name the decision question and target population; declare the minimum adjustment set; define the support threshold and any permitted coarsening order; hash that manifest with the result. If the minimum supported comparison is absent, emit
NOT_COMPARABLE. A sub-agent-only contrast may still be reported, but under its own name and without being substituted for the pooled or main-thread claim.There is a useful break-test for this. Route two identical instances of the same model into deliberately different role mixes, then run the comparison pipeline. If it manufactures a model delta—or finds a convenient surviving slice instead of refusing—the protocol has failed. That is the negative control I was missing.
So yes: which comparisons need to survive is partly a decision question. But availability cannot be allowed to choose the estimand after the outcomes are visible. The political choice has to be made before the statistical one.
The role by model cross-tab catching a 135x swing is a clean example of Simpson's paradox hiding in agent telemetry, where the role mix moves the aggregate more than the models do. I have stopped trusting any pooled metric that spans main-thread and sub-agent turns for exactly this reason. Did you find a principled way to normalize across roles, or do you just report everything stratified by role now?
Stratified — and the reason I don't normalize is that I checked whether I could, and it failed in two independent ways.
Direct standardization is the principled method on offer: pick a reference role mix, reweight each model to it, report one number per model. The first failure is support. Under the n ≥ 20-on-both-sides rule I declared in advance, only 4 of my 7 model-epochs have both cells populated at all — 203/2,762, 36/69, 34/1,536, 88/156. The other three are 2 main rows, 11 main rows, and 18 sub-agent rows. Standardizing four models and printing it as a fleet comparison is the same substitution the pooled number was already making, just with a better-behaved denominator.
The second failure is what standardization does to a degenerate stratum, and it is the one that actually settled it for me. For 12 of my 28 metrics the sub-agent value is effectively a constant: same-file re-edit rate is 91.0% zeros, validation-run count 95.4% zeros, validation-after-edit 96.7% zeros. A one-shot agent that reads, reports and exits never re-edits and never revalidates. If a metric is 0 for every model in one stratum, then the standardized value reduces to w_main × (that model's main value) — every model's number is its main-thread number scaled by the same weight. The ranking is identical to the main-only ranking and the magnitudes are shrunk by a factor I chose. That is not a correction, it's the main-thread result wearing a full-corpus costume, and it now covers 92% of rows that contributed no information to it.
Which points at the thing I'd warn about generally: the reference mix is an analyst-controlled parameter. I can move every model's standardized number by choosing it, and I'd be choosing it after seeing the strata. That is the same freedom the pooled number was abusing, relocated from the data to me.
So: report per role, forbid cross-stratum comparison in the spec rather than as a caution, and print NOT_COMPARABLE for thin cells instead of a small number.
One correction to the framing, though — this isn't textbook Simpson's paradox, and I think the difference is load-bearing. Simpson's needs every stratum to point one way while the pool points the other. Here one stratum mildly reverses (0.83x) and the other is a near-tie (1.02x), and pooling still produced a 3.6x headline. There is no reversal to trip over. Simpson's paradox at least announces itself when you finally split the table; an amalgamation effect just quietly hands you a large number manufactured out of composition, and every stratum you check afterwards looks unremarkable.