My Multi-Agent Crew Found 2.11x as Many Facts as One Agent. It Also Cost 4.5x and Was Less Precise

Every multi-agent walkthrough ends the same way. Here is the researcher, here is the critic, here is the editor, and here is the polished brief they produced.

What is missing is the comparison that decides whether any of it was worth doing: the same question handed to one agent, with both bills on the table.

So I built the crew and made that comparison the product.

$ python -m research_crew --compare --seeds 60

       found  bogus   prec   cost  fails  found/cost
-----------------------------------------------------
crew    9.50   4.55   0.69   18.0   0.00       0.528
solo    4.50   1.65   0.74    4.0   0.00       1.125

the crew found 2.11x as much and cost 4.50x as much.
per unit of spend, solo wins.

The crew found more than twice as many real facts — and was less precise than the single agent it replaced, at 4.5x the price. Both are true at once, and only one of them makes it into demos.

Repo: https://github.com/dev48v/research-crew — PUBLIC, MIT, standard library only, 23 tests, no API key and no network.
Run the whole sweep in your browser: https://dev48.infy.uk/agentlab/vol1-04-research-crew.html

Why fan-out costs you precision

Four researchers on one question do not find four disjoint sets of facts. They rediscover the same things, so real findings saturate. Their mistakes do not — each invents its own, so noise grows linearly in crew size.

Two curves with different shapes:

researchers	facts found	precision	cost	facts per unit cost
1	3.95	0.842	9.0	0.439
2	6.35	0.794	12.0	0.529
4	8.98	0.724	18.0	0.499
8	10.22	0.589	30.0	0.341
16	10.30	0.397	54.0	0.191

From 8 researchers to 16: +0.08 facts for +24 cost, and precision falls off a cliff. The efficiency peak is at two. Past the point where recall saturates, a wider crew converts money directly into noise.

The critic is not polish. It is load-bearing.

Because of that, the crew starts out behind the single agent on precision and only overtakes it once the critic is strict enough — at about 0.56 in this model:

critic strictness	crew precision	solo precision
0.0	0.628	0.736
0.5	0.724	0.736
0.7	0.778	0.736
1.0	0.920	0.736

Below that line you are paying 4.5x for a less trustworthy answer.

And the critic's own cost is never quoted

A filter has two error rates. Everyone discusses the first:

catch_rate      = 0.9  * strictness   # invented material it removes
collateral_rate = 0.25 * strictness   # CORRECT material it removes on the way

Turning the critic to maximum buys 29 points of precision and destroys 2.7 real facts to do it. There is no setting that gets both.

Those two lines started life as one clever expression, and a test killed it:

def test_the_critic_removes_correct_claims_too():
    """A filter with no collateral damage is a filter that is not filtering."""
    ...
    assert real_lost, "the critic never discarded a correct claim"

The test failed, and the reason was worth more than the fix: the original line had folded two different error rates into one formula, so the collateral damage was invisible — including to me, and I wrote it.

The runner, where the honesty lives

Retries are billed.

while attempts < max_attempts:
    attempts += 1
    spent_here += task.cost      # charged whether or not it succeeds
    report.spent += task.cost

A task priced 3 that fails once costs 6, and the report says 6. Counting only the successful attempt under-reports your bill by exactly your failure rate — which is the number you were trying to measure.

A task whose dependency failed is skipped, never run.

broken = [d for d in task.depends_on if d in failed]
if broken:
    report.results.append(TaskResult(task.id, task.role, "skipped",
        reason=f"depends on {', '.join(broken)}"))
    failed.add(task.id)
    continue

If every researcher dies and the editor still runs, it gets an empty context and writes a confident, well-structured summary of nothing — indistinguishable from a real one until somebody checks. The test asserts by["edit"].output == "".

And there are simply more things to fail. At a 20% per-task failure rate: crew 0.53 non-ok tasks per run, solo 0.05. Ten times the exposure, which is what seven tasks instead of one predicts.

Waves, not a flat topological order

The graph is the crew; the personas are the least load-bearing part.

def waves(self) -> list[list[Task]]:
    remaining = {t.id: set(t.depends_on) for t in self.tasks}
    done, out = set(), []
    while remaining:
        ready = sorted(tid for tid, deps in remaining.items() if deps <= done)
        if not ready:
            raise CycleError(f"dependency cycle among: {', '.join(sorted(remaining))}")
        out.append([known[tid] for tid in ready])
        done.update(ready)
        for tid in ready:
            del remaining[tid]
    return out

Kahn's algorithm collecting a whole level at a time. A flat topological sort is a correct order that has thrown away the only thing the graph existed to express. Cycles are refused here, at build time — not at run time with two agents waiting on each other and nothing printing.

critical_path() falls out of it: plan(2) + research(3) + critique(2) + edit(2) = 9, whatever the fan-out width. Sixteen researchers do not finish sooner than two. More workers buys coverage; it never buys latency.

Quality is counted, not judged

Ground truth is a fixed list of twelve facts, so scoring is set arithmetic — no LLM-as-judge in the measurement loop, because a judge that shares failure modes with the thing it judges cannot be the instrument. That is what lets 600 crews run in 0.16 seconds and give you the same numbers they gave me.

Swapping in a real model is one callable:

def nim_agent(task: Task, context: dict[str, str]) -> str:
    ...   # raise AgentFailure on a recoverable error, return text otherwise

run(research_crew("your question", n_researchers=4), nim_agent, budget=50_000)

So when is a crew worth it?

• One agent wins on facts-per-unit-cost at every setting tested. If spend is the constraint, the answer is not a crew.
• A crew wins on coverage — 2.11x as much found — and only with a critic above the break-even strictness.
• Keep it small. Two to four. The efficiency peak is at two and the recall ceiling arrives by eight.
• Count the failures before you fan out.

The numbers characterise this model of agent behaviour, not GPT-5. What transfers is the shape: saturating recall against linear noise, a critic that must earn its place, and a failure surface that grows with the crew. Plug in your own measured recall and hallucination rate and the same sweep answers it for your setup.

Project 4 of 5 in Agent Lab Vol 1 — finished tools you clone and run: https://dev48.infy.uk/agentlab.php