DEV Community

Mohammad Fauzel Sadeghizad
Mohammad Fauzel Sadeghizad

Posted on

Measuring the Multi-Agent Fork Tax

#ai #agents #llm #engineering

In our previous post, Giving a Coding Agent an Org Chart, we described Subsessions: a protocol layer over agent runtimes that enforces role-typed child sessions, versioned briefs, and append-only checkpoint journals.

The entire architecture rested on one foundational invariant:

The mother's context grows by $O(\text{report})$, never by $O(\text{child transcript})$. The human sees the whole org tree; the mother sees only structured reports.

We closed that post with an unresolved question:
The invariant holds mechanically—every gate, every report contract, every succession letter exists to enforce it. What we don't know yet is what it is worth on real workloads: we haven't published the measurement.

This post publishes the measurement.

We pre-registered a formal evaluation rubric (RUBRIC.md), generated deterministic fixtures across 20 programming tasks, collected 45 full execution runs across two experimental arms, and submitted the entire dataset to an independent adversarial verification audit (results/VERIFICATION.md).

The findings are unambiguous, humbling, and a stark reminder that in software architecture, no abstraction is free.

1. The Experiment: Inline vs. Orchestrated

To test whether subsession orchestration outperforms a single agent session, we evaluated two arms under identical conditions:

  1. The Inline Arm (Role: worker): A single autonomous session. The subsession toolset is structurally stripped (canSpawn: false). The agent receives the task prompt and executes all code navigation, editing, script execution, and verification directly within its own session.
  2. The Orchestrated Arm (Role: coordinator): A supervisor session authorized to spawn children (canSpawn: true, depthCap: 2). Under pre-registered protocol amendments, the coordinator is required to delegate work to at least two worker subsessions and synthesize their returned reports before delivering the final answer.

Strict Fairness Controls

To ensure we measured the structural cost of orchestration rather than model variance:

  • Identical Model: Every session across both arms resolved to the deployment default (bai-/glm-5.3-flash).
  • Identical Seed Fixtures: All repository files (80 TypeScript files), financial CSVs (360 rows in integer cents), server logs (3,000 lines), and meeting notes (347 KB across 200 sections) were deterministically synthesized from a seeded PRNG (Mulberry32).
  • Mechanical Scoring: Outputs were evaluated mechanically by checks/check.mjs using strict deep-equality against frozen JSON ground truths.
  • Interleaved Execution: Runs alternated in an A/B/B/A sequence to eliminate time-of-day API throttling biases.

2. The Headline Result: The Fork Tax

We measured 20 tasks divided across four distinct software engineering classes ($5$ tasks per class). Across the 19 completed pairs, orchestration was strictly more expensive in every single case.

Caveat: One model, one day, $n=5$ per cell, mandatory fan-out ($\ge 2$ workers).

Task Class Inline Tokens (med) Orch Tokens (med) Token Ratio Inline Wall (med) Orch Wall (med) Wall Ratio Mother / Inline
breadth-read (T01–T05) 76,393 304,824 3.99× 204 s 491 s 2.41× 2.65×
single-write (T06–T10) 49,269 215,631 4.38× 96 s 745 s 7.76× 3.18×
longdoc-transform (T11–T15) 170,631 592,452 3.47× 439 s 1,224 s 2.79× 1.30×
multi-step-tool (T16–T20) 104,321 390,129 3.74× 185 s 759 s 4.10× 1.70×

On accuracy, both arms achieved identical performance: both passed 100% of tasks T01 through T19. (Both arms failed T20 identically due to a disclosed ground-truth collation defect where the prompt asked for lexicographic ordering and the generator sorted directory-first).

On cost and latency, however, orchestration imposed a severe fork tax:

  • Token Inflation: Orchestrated execution required $3.47\times$ to $4.38\times$ more total billed tokens.
  • Wall-Clock Latency: Orchestrated execution was $2.41\times$ to $7.76\times$ slower. On single-write tasks, dispatching workers, awaiting tool handshakes, and merging reports took $745$ seconds versus $96$ seconds for the single inline agent.

3. Where Did the Tokens Go?

The most critical finding emerged when we disassembled the token accounting inside the subsession trees.

A naive assumption is that multi-agent systems burn tokens because multiple workers are exploring parallel paths. The telemetry revealed the exact opposite:

Orchestrated Run Token Spend (Median Across 19 Pairs):
├── Mother Session (Supervisor Loop): 55%
└── Worker Subsessions (Task Work):   45%
Enter fullscreen mode Exit fullscreen mode

Over half (55%) of all tokens in an orchestrated run were consumed by the mother session's own loop, not by the children doing the work.

Look at the operational numbers:

  • Across all 19 completed pairs, the median volume of child reports delivered to the mother was 3,529 bytes.
  • The median token spend of the mother session managing those reports was 213,219 tokens.
  • The mother spent $\sim 60\times$ more tokens driving her own coordination loop than the total content size of the reports she received.

This is the central paradox of context engineering: the $O(\text{report})$ invariant held mechanically on disk, but was economically irrelevant.

The mother's context was indeed protected from child transcripts: she received clean, 2~KB to 7~KB JSON summaries. But the cognitive tax of running a supervisor—formulating structured briefs, specifying filesystem globs, waiting for turns, polling status, reconciling discrepancies, and formatting output—dwarfed the entire cost of the task.

In 17 of the 19 completed pairs, the mother session's spend alone exceeded the total token usage of the inline agent doing the entire task from start to finish.

4. The Elusive Crossover Point

In our pre-registered protocol, we hypothesized that as task scale and document size increased, a measurable "crossover point" would emerge where the token savings of context isolation would overtake protocol overhead (Hypothesis 3).

The data refuted this hypothesis entirely:

  • Pairs where orchestration used fewer total tokens: 0 / 19 (0%).
  • Pairs where the mother alone used fewer tokens than inline: 2 / 19 (Task T13 at $0.99\times$, Task T17 at $0.86\times$).
  • H4 Invariant Bar ($\text{mother tokens} < 0.5\times \text{inline total}$): 0 / 19 (0%).

Even on the largest fixture in the benchmark—a 347~KB, 200-section executive meeting document—the inline agent completed the analysis in 170,631 tokens. The orchestrated arm burned 592,452 tokens ($3.47\times$). Why? Because modern LLM context windows (128k+) can digest a 347~KB document in a single gulp. Splitting the document across two workers meant both workers had to read their respective slices, serialize findings, and send them to the coordinator, who then had to read all findings and re-verify them.

5. Checkpoint Resumption and Deadlock Risk

One promised advantage of subsession architectures is state resilience: if a subagent dies or is terminated mid-flight, a successor can resume from an append-only checkpoint journal without re-running the entire task (Hypothesis 5).

To measure this, run R38-T19-orch was deliberately killed mid-task at 18,549 tokens (step 2, 78 seconds). It was then resumed with the frozen recovery prompt: "you were stopped mid-task; recover from your checkpoint and finish; report normally."

The result was an operational disaster:

  • The resumed leg burned 129,945 additional tokens (total: 148,494 tokens).
  • The coordinator attempted 5 child spawns. Two were rejected immediately because the model hallucinated unregistered role names (log-analyst-a, log-analyst-b).
  • The coordinator then spawned two valid worker subsessions. Both workers subsequently failed and terminated without delivering a final report.
  • Crucially, the coordinator runtime possessed no timeout or dead-child detection for asynchronous reports.
  • The coordinator emitted its final turn message: "Both workers are running. I'll merge and write the deliverable once their final reports arrive," ended its turn, and parked indefinitely.
  • The run remained frozen in a deadlock state for 53,046 seconds (~14.7 hours) until the experimenter manually force-retired the session.

The takeaway for agent harness designers is stark: without active quiescence monitoring and explicit timeouts on child report arrival, checkpoint resumption does not save tokens—it introduces unbounded deadlocks.

6. Real-World Mechanism Friction

Because both arms executed against the identical frozen tasks, the benchmark illuminated four failure modes unique to multi-agent architectures:

  1. Role Name Hallucination (UNKNOWN_ROLE): Coordinators repeatedly invented functional role names (doc-summarizer-a, summarizer, log-analyst) instead of adhering to the role table (worker). Each invalid spawn consumed a full LLM turn and tool round-trip before falling back to valid roles.
  2. Shared Tree Budget Starvation: In our initial configuration, runs were given a shared tree budget of 150,000 tokens. In run R15, the coordinator spent so many tokens setting up its planning loop that all 6 spawned children were admitted and killed at 0 tokens, unable to take their first step. We had to issue a protocol amendment (Amendment #4) raising the budget to 1,000,000 tokens so the cap would not bind asymmetrically on the orchestrated arm.
  3. Auto-Verifier False Negatives: In at least 8 runs, the runtime's automated verification hooks raced the filesystem writes of child sessions, falsely reporting deliverables as "missing" or "disputed." Coordinators had to burn extra turns reading the disk directly to dismiss the harness's own false alarms.
  4. Worker Unreliability: In task T04, a worker claimed it found 53 imports when it had delivered 52. In task T13, workers claimed 103 items where the true count was 100. In both cases, the coordinator caught the discrepancies by re-running the extraction itself from raw source files. The coordinator did add value by catching hallucinations—but it did so by duplicating the worker's effort, strictly increasing total token spend.

7. The Skeptic's View: Steelmanning the Attack

An adversarial verifier was commissioned to independently audit this benchmark (results/VERIFICATION.md). What would a skeptic say about these results?

Here is the honest counter-argument:

This benchmark is an engineered test where multi-agent orchestration was structurally guaranteed to look inefficient. By mandating delegation to $\ge 2$ children on tasks that operate on small repositories and a 347~KB document, the benchmark forced a complex corporate hierarchy onto problems that comfortably fit into a single model turn. Orchestration exists to conquer tasks that exceed context limits; testing it on sub-100k token workloads merely measures the administrative overhead of JSON briefs and prompt handshakes. Furthermore, measuring cumulative billed tokens penalizes the inter-agent communication channels that keep worker contexts clean. This benchmark does not prove orchestration is flawed; it proves that requiring an agent to form a committee to write a 5-paragraph essay is wasteful.

This critique is valid, and we accept it.

Our benchmark does not prove that multi-agent systems are inherently inferior. It proves something more specific and immediately actionable: mandatory fan-out on workloads that fit within an LLM's active context window imposes a $3.5\times$ to $4.4\times$ cost multiplier with zero accuracy benefit.

When given the choice in pilot run R02b, a capable coordinator looked at the task, judged it small, and chose to do it entirely inline. Our protocol forced it to delegate. The coordinator's initial instinct was right.

8. Reproducing the Benchmark

All data, scripts, ground truths, and per-run telemetry are open-source in the public repository under bench/:

cd bench

# 1. Verify deterministic fixtures and run self-test
node generate.mjs --verify
node selftest.mjs
# Expected: 20/20 PASS, 3/3 deliberate corruptions caught

# 2. Recompile CSV and recompute all metrics
node compile.mjs
node analyze.mjs
node stats.mjs
Enter fullscreen mode Exit fullscreen mode

Every number in this article is derived directly from results/results.csv. Per-run prompt definitions, metrics, and final reports are committed under bench/runs/.

Conclusion

The subsession protocol successfully solves the problem it was designed to solve: child transcripts do not leak into supervisory context.

However, solving the context isolation problem does not solve the cost problem. Coordination has mass. Writing briefs, inspecting outputs, managing tool permissions, and recovering from child stalls consumes more tokens than the work itself.

For AI engineers building agent workflows, the architectural rule of thumb is clear:

  1. Default to inline execution. If the codebase, document, or working memory fits within the model's context window, keep it in a single session.
  2. Do not force delegation. Allow autonomous supervisors to execute tasks inline when they deem them manageable.
  3. Reserve orchestration for structural walls. Deploy subagents only when context saturation degrades reasoning, or when strict security and tool sandboxing mandates an isolated execution boundary.

Until an agent hits a context or permission wall, the cheapest org chart is no org chart at all.

Resources & Artifacts

Top comments (0)