In August I ran a controlled experiment on my open-source multi-agent pipeline: the same task, done by 1, 4 or 6 agents, 30 runs each. The result was tidy. Four agents looked like the sweet spot: half the cost of six, and clearly better reports than one.
In October I reran it with the same preregistered protocol on a different model. This time:
- Held: four agents were cheaper than six again, by more than before (60% fewer tokens, 54% lower cost, 49% less time).
- Flipped: "four agents write better reports than one" did not replicate. This time the single agent had the fewest problems.
- New: the reviewer agent in the six-agent arm failed its own output check on the first attempt in 27 of 30 runs. The cause was a limit that existed in the code's schema but not in the prompt. Adding one sentence to the prompt took the failures caused by that limit from 21 to 3.
If I had published only the August run, this would be a neat "four agents is optimal" post. It doesn't hold on another model, so here are both runs.
What the system does
You give it a paper or a research topic, and it writes a commercialization screening report: scores for technology readiness (TRL), manufacturing readiness, patents and market, with a source cited for every claim. It is aimed at university tech-transfer offices, researchers and early-stage investors, and it produces a first-pass draft, not due diligence.
It works in two stages. First, ordinary code retrieves papers, patents and market sources, validates and deduplicates them, and freezes them into a source registry. Then six LLM agents work only inside that registry: academic, patent and market analysts run in parallel, followed by a writer, a reviewer and a scorer. The weighted total score is computed by code, not by a model.
The experiment
Predictions before spending. Before any paid run, I committed the protocol, the decision rules and four predictions to the repository, so I couldn't move the goalposts after seeing results:
| Prediction | |
|---|---|
| P1 | The four-agent writer passes the report-format check on the first attempt more often than the single agent |
| P2 | Six-agent reports have no more numbers that can't be matched to a source than four-agent reports |
| P3 | Going from six agents to four cuts cost and tokens by at least 20% |
| P4 | One agent is cheapest, but loses to four agents on at least one quality measure |
Three arms. A single "do-everything" agent that writes the report directly; four agents (three domain specialists plus a writer); and six agents, which is the full production pipeline with a reviewer and a scorer added. All three read the same frozen evidence and are held to the same report format and checks. Ten topics (from CAR-T cell therapy to room-temperature superconductors) × 3 repetitions × 3 arms = 90 runs, in Latin-square order.
One caveat up front. The evidence was frozen in advance, so the experiment does no retrieval and calls no search API. The costs and timings below cover the model calls only and are much lower than a full production run. Only the six-agent arm has a scorer, so scores are not compared.
Run 1: August, DeepSeek
The 90 runs cost $1.80 and took about 2.7 hours. The main results:
- Six → four agents: compared topic by topic, 57% fewer tokens, 47% lower cost and 38% less time, with no quality loss.
- One → four agents: uncited claims fell from 115 to 24, at 2.4 times the cost.
- The reviewer barely edited. The final report kept a median 99.998% of the draft's text. A later blind comparison found that two evaluators each preferred the reviewed version in 7 of 9 pairs, so the reviewer wasn't useless; it just changed very little.
- P1 was wrong. The four-agent writer failed its first format check 5 times, the single agent once.
Before the full study I spent under 6 cents on a pilot, which showed that the counter recording format-check failures was stuck at zero. The framework was calling a different entry point, so the counter was never attached. I fixed that before running the real thing.
Why rerun it
Two months later, production was on a different model and a lot of code had changed. I wanted to know whether August's conclusions survive a model change.
So I wrote a second preregistration, with the four predictions and every decision rule copied unchanged, the same frozen evidence, and qwen3.8-flash as the model. To be clear, that is a small model chosen for its free quota, not the model production uses.
Before the paid run I hit another problem: an offline check showed the experiment script would crash on start in the four- and six-agent arms. After August, the writer and reviewer prompt templates had gained new variables, and the script never passed them in. The fix was three lines, and it was found offline rather than by a paid run.
The 90 runs recorded $0.64 at list-price upper bounds (about ¥3.4 actually charged, after the free quota) and took about 3.3 hours with no interruptions.
Result 1: the savings held
Four versus six agents, paired by topic: in August, 57% fewer tokens, 47% lower cost and 38% less time; in October, 60%, 54% and 49%. P3 held both times, and the savings grew.
The six-agent arm also had a new problem: it completed only 23 of its 30 runs. More on that in Result 3.
Result 2: the quality conclusion flipped
In August, uncited claims were 115 for one agent, 24 for four and 38 for six: splitting the work clearly helped. In October they were 39, 166 and 115. Comparing report-format findings topic by topic, the single agent did better than four agents in 27 of 30 pairs. P4 was falsified, and P1 failed again (this time no arm failed a first attempt at all).
There is a catch, which I found only after seeing the results.
After August, the production writer template gained a "decision-output contract" section. It requires the report to state the assessment mode, the decision owner, the cost and time (or "not established"), and so on. Lines like these describe a mode or an absence, so there is nothing to cite, but the screen counts them as uncited claims. The four- and six-agent arms use today's template; the single agent still runs its frozen August prompt, which has no such section.
Separating those lines out: 91 of the four-agent arm's 166 findings and 66 of the six-agent arm's 115 are contract lines. On all other lines the counts are 39, 75 and 49. The single agent still has the fewest, but the gap shrinks from more than 4× to under 2×.
The rule for checking numbers also changed in September: figures in a unit the system can't recognize are now counted as unmatched. That makes this run's numeric check close to uninformative.
So the reversal is partly the model, partly the template and partly the measuring stick itself, and I can't separate the three.
Result 3: the sentence that wasn't in the prompt
The six-agent arm completed only 23 runs because of the reviewer agent: in 27 of 30 runs its first output failed its own schema validation, and in 7 it still failed after a retry. By cause:
| Cause | First attempt failed | Still failed after retry |
|---|---|---|
| A correction's reason longer than 300 characters | 21 | 5 |
| More than 20 corrections | 5 | 0 |
| Text to replace appears more than once in the draft | 1 | 2 |
The reviewer prompt said "at most 20 corrections" and "find and replace text up to 2,000 characters", but never said "reasons up to 300 characters". That limit existed only in the code's data schema.
So I preregistered a small follow-up: add only that sentence to the prompt, change nothing else, and rerun the same 30 six-agent runs ($0.36 recorded):
- First attempts failing on reason length: 21 → 3
- First attempts failing for any reason: 27 → 6
- Still failing after a retry: 7 → 2
- Completed runs: 23 → 28
One side effect: the median number of corrections per review fell from 9 to 5.5. Whether the reviewer became more cautious, whether long reasons simply come with long edits, or whether it is day-to-day noise, this experiment can't tell. It also can't say whether fewer edits are better or worse.
The same reviewer also behaved very differently on the two models. In August it barely changed anything; in October it made 241 corrections, and the final reports kept a median 99.17% of the draft text. (In production, a failed review falls back to the writer's draft, so the report still ships. But that is a paid review that changed nothing.)
What these results don't show
- The differences can't be attributed to the number of agents. Between the two runs, the model, the provider, the writer template and both measurement scripts changed, across 669 commits.
- qwen3.8-flash is not the production model. Production was on qwen3.7-max-preview on October 23. These results say nothing about it.
- The prompts are asymmetric. The four- and six-agent arms run today's prompts; the single agent runs August's.
- Quality was screened by code only. The screens check whether claims are cited and whether numbers can be found in the sources. Nobody checked whether the reports are right, and the reviewer's edits were not judged one by one.
- The sample is small. Ten topics × 3 repetitions gives effect sizes, not statistical significance.
Four lessons for anyone building agents
- Every constraint in your schema belongs in your prompt. One sentence took the failures from that limit from 21 to 3.
- When you rerun an experiment, lock more than the model. Lock the prompt templates and the measurement scripts too, or you won't know what changed the result.
- Spend a few cents checking your measuring instruments first. In August the pilot found a broken counter; in October an offline check found a script that would crash on start.
- Preregister, then publish both runs. Publishing only the run that flatters you is more misleading than publishing nothing.
About the project
- GitHub: https://github.com/shuxiachai/academic-commercialization-agent
- Sample reports, no key or sign-up (five full assessments, one in Chinese): https://academic-commercialization-agent.up.railway.app/samples
- August protocol and results: https://github.com/shuxiachai/academic-commercialization-agent/blob/main/docs/prereg-2026-08-21-agent-topology-ablation.md
- October rerun results: https://github.com/shuxiachai/academic-commercialization-agent/blob/main/docs/results-2026-10-24-agent-topology-ablation-qwen.md
- One-sentence reviewer fix: https://github.com/shuxiachai/academic-commercialization-agent/blob/main/docs/results-2026-10-24-reviewer-reason-bound-qwen.md
The protocols, results and limitations, failures included, are in the repository, along with the scripts that compute every number. The per-run output directories are not committed. Questions are welcome in GitHub Discussions, and the good first issues don't need an API key.
I'd especially like feedback from people in tech transfer, research or early-stage investing: after reading a sample report, what is useful and what is wrong?
And a question for other agent builders: how many agents does your pipeline have, and have you had a model ignore your output schema? How did you fix it?
I'm Jingbo, a student at the University of Sydney. This post was drafted with AI assistance; every number was checked against the result documents linked above.
Top comments (2)
The most useful result here is not a universal agent-count rule, but the disciplined record of what changed with the model, prompt, and metric. Freezing evidence was a strong control; now the next experiment needs frozen prompts and evaluation semantics too. Otherwise an apparent architecture win can just be measurement drift wearing a lab coat.
Thanks, Alex. "Measurement drift wearing a lab coat" is exactly what happened here. Next time I'd pin the prompt templates and the screening scripts to a commit alongside the frozen evidence, so a rerun can only change the one thing I mean to change. How do you keep eval semantics fixed in your own work?