Does going multi-agent pay off, and can you auto-pick a pattern per task?

I ran six multi-agent patterns across three benchmarks and three models. By "a team of agents" I mean setups like critic-actor or an orchestrator with subordinates. Almost everywhere, the team lost to a single agent. It lost on accuracy and on cost, and on cost sometimes by 4x.

This could have been a dull "MAS isn't worth it, let's go home" note. But there was an exception. The harder the task is for the model, the better the team's chances of winning. So I have two questions. Does the system's topology affect the result at all? And if it does, can we automatically pick a suitable pattern for a specific task? The answer to the second one is no, for now, and the most interesting part is exactly why.

I show the examples on the FEDOT.MAS multi-agent framework, and all the experiment code (patterns, runs, and plots) lives in the repo. I'm measuring a tool I write myself. But the conclusions mostly come out against multi-agent, so this is more self-criticism than advertising.

MAS patterns

Each pattern is a way to wire several model calls into one system in a particular order. For comparison I used the following:

single. One agent, one pass. The baseline everything else is compared against.
chain. A two-step chain. One agent breaks the task down, the other solves it from the breakdown.
voting. Two agents solve the task independently, and a third judge merges their answers into one.
eval_optimizer. A generator proposes an answer; a critic accepts it or sends it back for revision.
orchestrator. At each step a coordinator decides which subordinate works next, and keeps going until it says "done".
blackboard. Agents write to a shared board. These are a researcher, a skeptic-verifier, and an assembler. The framework supports all these patterns and their combinations, and a system is assembled from primitives via an overloaded + operator. Here's the eval_optimizer pattern:

from fedotmas.adapters.pydantic_ai import PydanticAI
from fedotmas.sdk import agent, Condition

llm = PydanticAI("openrouter:openai/gpt-oss-20b")

gen = agent(
    "gen", takes=dict,
    prompt="Solve the task; if the critic left a comment, take it into account.",
    input="Task: {task}\nComment (if any): {verdict}"
)
critic = agent(
    "critic", takes=dict, labels=["approve", "revise"],
    prompt="Review the solution: approve if correct, otherwise revise.",
    input="Task: {task}\nSolution: {draft}"
)

system = gen.into("draft") + critic.into("verdict")

# draft -> verdict, and round and round until the critic says approve
loop = system.loop(until=Condition(key="verdict", op="eq", value="approve"))

# start point and run
start_point = {"task": task, "draft": "", "verdict": ""}
answer = await loop.run(start_point, llm=llm)

The other five are assembled similarly. One interesting note: a node in the system can be a single LLM agent, an entire MAS, or just a plain mechanical Python function.

Experiment setup

For the tests I took several standard benchmarks of varying difficulty from different domains. GSM8K is grade-school math. MMLU is multiple-choice questions across domains from chemistry to history. LogiQA is logic problems. As executor models I used gpt-oss-20b, ministral-8b, and llama-3.1-8b. I didn't take the whole benchmark, just a 100-question subsample. For the multi-domain benchmarks I used a bootstrap sample with a fixed seed. Caveat: at this stage I only considered benchmarks that don't require tools for the agents.

I gave each role a simple, direct prompt with no special tuning per pattern, and kept the prompts at the same level across all benchmarks. This was to measure the system architecture, not prompt engineering.

We'll also need a reference point: a per-task oracle. This is as if an omniscient someone picked the ideal pattern for every task. It's the ceiling that shows how much you could squeeze out of selection in principle.

One important caveat. The oracle chooses in hindsight, from already-computed runs. So there's an assumption that on a rerun the chosen pattern would give roughly the same accuracy. That's why the choice is taken over the average of three runs, not one, so it reflects the pattern's stable behavior on the task, not a one-off lucky hit.

First result: easy tasks

Let's start easy. On GSM8K and MMLU the picture is the same. The single agent sits on the cost/accuracy frontier, and there's nowhere to push it from there.

On GSM8K with gpt-oss-20b, single gives 0.94 for 398 tokens per task. The best multi-agent pattern gives 0.95, but at 1692 tokens. A one-point difference sits within the noise, and you pay 4x more. On MMLU it's exactly the same. Any "smart" layer on top of a single pass either doesn't help or slightly hurts, while always costing 2-4x more.

The harder the task, the more selection makes sense

Now the interesting part. If it were only "MAS isn't worth it", the article would end here. But let's look at that same gpt-oss-20b across benchmarks in increasing difficulty.

The model is the same, but the gap between the best fixed pattern and the per-task oracle grows. On GSM8K it's +3 points, on MMLU already +6, and on LogiQA +9. So the harder the task is for the model, the more you could gain by picking a pattern per individual task.

This headroom reflects how hard the task is for the model, not the model's size. The small ministral-8b on easy GSM8K gives only +1 point of headroom, because it's strong there. But llama-3.1-8b of the same class, yet weak on these tasks (0.63-0.80), gives +11…16 points. It's not about parameter count, but about how hard the task is for that specific executor. In some cases a critic helps, in others a single pass is enough.

So there's something to choose between.

If selection matters, let's select automatically

An obvious solution suggests itself. Put a selector in front of the system, an agent that picks a pattern from the task text. In FEDOT.MAS the selector is a single agent with a limited choice from a menu of patterns. Did it improve quality?

No.

Here are the numbers on LogiQA with gpt-oss-20b. The single agent gives 0.77, random pattern choice 0.73, the selector only 0.71, and the oracle 0.89. The selector didn't just fall short of the ceiling. It came out worse than both the single agent and a random guess. And it's like that everywhere. In none of the columns did the selector beat the single agent. It only beat random choice where the options contained an obvious outlier, like the looping orchestrator.

It's not that the selector is "dumb". It's about information. Which pattern solves this particular task on this particular model is an empirical fact that lives in the results matrix, not in the task text. You can't derive it from the wording alone. Tasks look the same but get solved by different patterns for reasons that simply aren't in the text. So a smarter judge won't save you here either. The problem isn't intelligence, it's the absence of signal at the input.

What to do about it? One option is to train the selector on examples like "this kind of task on this model is best solved by this pattern". That's fine-tuning already, and a topic for a separate discussion.

Conclusions and what's next

What did we learn, and what might be useful in practice?

An orchestrator on a weak model is dangerous. Without a hard step limit it loops and burns tokens.
Pattern selection only pays off on tasks that are hard for the model. It's worth honestly noting a weak spot in the measurement. In patterns with verifier checking, the verifier is the same model as the generator, so it essentially judges itself, with no independent signal. That's why the honest next step leads to where multi-agent has a principled way to win. That's code with real test execution, like HumanEval, where the critic stops being a "second opinion" and becomes a test runner with a real signal. Or hard benchmarks like GAIA2 that require tools and interaction with the outside world.

For now, all the results are on GitHub and can be reproduced on other models.