Sub-Agent Metrics Are a Different Population: Prove It With a Paired Benchmark Harness

A counterexample before the architecture

A team I advise ran a coding agent with a planner on the main thread and three sub-agents doing file edits. Their dashboard said: p50 task latency 41s, success rate 87%. When they split the same data by origin, the picture inverted: main-thread tasks were p50 22s at 96% success; sub-agent tasks were p50 78s at 71% success, and most sub-agent latency was queue wait for a semaphore plus retry amplification, not model time. Averaging the two populations hid a retry storm behind a healthy-looking mean.

The invariant the common implementation fails to preserve:

Metric-population invariant: two latency or success-rate samples are only comparable if they were drawn under the same arrival process, the same resource limits, and the same retry policy. Sub-agent metrics almost never satisfy this, because the orchestrator changes all three.

This article builds a minimal, executable harness to test that invariant for your system instead of trusting mine. A recent DEV discussion made the same qualitative point; here we make it a falsifiable property with an explicit denominator and an acceptance rule.

Declared assumptions

1. Your agent has one main thread (planner) that spawns 1..N sub-agents (workers) per task.
2. Sub-agents share a bounded resource: a concurrency semaphore, a rate limiter, or a model quota.
3. The orchestrator retries failed sub-agent calls with some policy (the usual suspect).
4. You can run repeated trials cheaply enough that N ≥ 30 paired runs per configuration is feasible. If you cannot afford repetitions, no statistical conclusion is available, full stop.

If assumption 4 is the blocker, this is where free tiers genuinely change the math. Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode currently offers free model access and a free server option, which is enough to host a harness like the one below and burn through the repetition budget a paired test needs, without committing budget before you know whether the experiment design is even right. Treat both as availability claims, not performance claims: I am not asserting anything about model quality, quotas, or duration, and the harness below is deliberately model-agnostic so you can point it at whatever endpoint you have.

The state model

Model one task as this state machine:

DISPATCHED --(planner)--> FANOUT --(k sub-agents admitted)--> RUNNING
RUNNING --(all succeed)--> JOIN --(planner synthesizes)--> DONE
RUNNING --(any fail, retries left)--> RETRYING --(re-admitted)--> RUNNING
RUNNING --(retries exhausted)--> COMPENSATED

The population-breaking mechanisms live in two transitions:

• Admission: sub-agents queue on a shared semaphore. Their measured "latency" includes wait time that main-thread work never experiences.
• RETRYING: retries multiply the number of admitted attempts. With retry budget r, a failure rate p under load turns into an attempt multiplier that main-thread tasks (retried differently or not at all) never see.

So "sub-agent p50" and "main-thread p50" are samples from different distributions by construction. The question is not whether they differ, but whether the difference is large enough to change a decision — that needs a test, not a vibe.

Minimal simulator

This is a proposal-grade artifact: runnable, dependency-free, deterministic under a seed. It simulates the state machine above, injects failure classes, and runs a paired comparison (same task, both configurations) with a permutation test, because sub-agent and main-thread latencies are not normally distributed and are correlated per task.

import random, statistics
from dataclasses import dataclass

@dataclass
class Task:
    base_work: float      # intrinsic model/compute seconds for this task
    splittable: bool      # can sub-agents parallelize it?

@dataclass
class Result:
    latency: float
    attempts: int
    ok: bool

def run_main(task, rng):
    # Main thread: no semaphore wait, one attempt, generous timeout.
    noise = rng.lognormvariate(0, 0.2)
    return Result(latency=task.base_work * noise, attempts=1, ok=True)

def run_fanned_out(task, rng, k=3, semaphore_slots=2,
                   fail_p=0.15, max_retries=2):
    # Sub-agents: parallelize if splittable, but queue on a semaphore
    # and retry on failure. Retries re-enter the queue.
    if not task.splittable:
        k = 1
    per_agent = task.base_work / k
    total_wait, attempts, t = 0.0, 0, 0.0
    for _ in range(k):
        tries = 0
        while True:
            tries += 1; attempts += 1
            wait = rng.expovariate(1 / (semaphore_slots * 2.0))
            work = per_agent * rng.lognormvariate(0, 0.3)
            total_wait += wait; t = max(t, wait + work)
            if rng.random() >= fail_p or tries > max_retries:
                break
    ok = tries <= max_retries
    return Result(latency=t, attempts=attempts, ok=ok)

def paired_experiment(n_tasks=60, seed=7):
    rng = random.Random(seed)
    tasks = [Task(base_work=rng.uniform(10, 60),
                  splittable=rng.random() < 0.6) for _ in range(n_tasks)]
    pairs = [(run_main(t, rng), run_fanned_out(t, rng)) for t in tasks]
    diffs = [m.latency - f.latency for m, f in pairs]
    obs = statistics.mean(diffs)
    # Paired permutation test: shuffle sign of each difference.
    trials, ge = 5000, 0
    for _ in range(trials):
        s = sum(d if rng.random() < 0.5 else -d for d in diffs) / len(diffs)
        if abs(s) >= abs(obs):
            ge += 1
    p = ge / trials
    success_main = sum(m.ok for m, _ in pairs) / n_tasks
    success_fan  = sum(f.ok for _, f in pairs) / n_tasks
    return obs, p, success_main, success_fan, pairs

if __name__ == "__main__":
    obs, p, sm, sf, pairs = paired_experiment()
    am = sum(m.attempts for m, _ in pairs) / len(pairs)
    af = sum(f.attempts for _, f in pairs) / len(pairs)
    print(f"mean paired latency delta (main - fanout): {obs:.1f}s  p={p:.4f}")
    print(f"success main={sm:.2f} fanout={sf:.2f}  avg attempts main={am:.1f} fanout={af:.1f}")

Testable properties

1. P1 (population separation): the permutation test on paired latency deltas rejects the null at p < 0.05 with the sign you predicted. If it does not, your pooling decision is still unjustified — absence of evidence is not evidence of absence at small N.
2. P2 (attempt amplification): avg_attempts_fanout / avg_attempts_main > 1 whenever fail_p > 0. If this fails, your retry instrumentation is broken, not your system.
3. P3 (injected failure monotonicity): raising fail_p from 0.05 → 0.15 → 0.30 must monotonically worsen fanout success and attempt count. If it doesn't, the harness is not exercising the RETRYING transition.

Failure injection matrix

Injected fault	Where	Expected signal
fail_p spike	sub-agent call	attempt amplification (P2)
semaphore_slots = 1	admission	wait dominates: latency ↑, work unchanged
max_retries = 0	RETRYING removed	success ↓, latency ↓ (fails fast)
splittable = false for all tasks	FANOUT	fanout strictly worse; pooling hides it

Acceptance rule (explicit denominator)

• Denominator: 60 paired tasks × 2 configurations = 120 runs per experiment; permutation test with 5,000 shuffles.
• Gate: you may merge sub-agent and main-thread metrics into one dashboard number only if P1 fails to reject at p ≥ 0.05 and the absolute mean delta is under a pre-registered decision threshold (e.g., 5s — pick it from your SLO, not from the data). Otherwise report them as separate populations with separate SLOs. Success rate and latency are gated independently.

Tradeoffs

Option	Latency signal	Cost of evidence	Main risk
Pool all metrics	Simple dashboards	Zero	Hides retry storms and queue wait (the opening story)
Separate populations, no pairing	Clear split	Low	Confounded by task mix differences
Paired harness (this article)	Decision-grade	N×2 runs per config	Simulator must match your retry/admission semantics
Full production A/B	Most realistic	Highest	Slow; ethics of injecting failures on users

Limitations and who should not use this

• The simulator models mechanisms, not your model's behavior. Point the same paired design at real endpoints before acting on it; the free compute tiers mentioned above are suitable for exactly this repetition-heavy phase, but validate that measured attempt counts match P2 before trusting latencies.
• The lognormal/exponential noise shapes are assumptions, not findings. Re-fit them from your traces.
• If your sub-agents run on isolated quotas with no shared semaphore and identical retry policy to the main thread, the invariant may genuinely hold — the harness will tell you (P1 won't reject), and that's a valid, publishable-negative result for your own design doc.
• Skip this entirely if you run fewer than ~30 comparable tasks per week; your denominator is too small for any conclusion and you should fix observability before statistics.

The counterexample question

Every invariant has an event order that breaks it. For your orchestrator: what happens when a sub-agent's retry is admitted after the planner has already timed out and compensated — does the late result get applied, dropped, or double-applied? Should the system reject the late completion, replay the join, or compensate again? If you can't answer with a state transition, your metrics pipeline is the least of your problems. The harness above is where I'd start answering it; if you try it against a free model endpoint, I'm curious which property fails first.