DEV Community: Dmitry Glhf

Does going multi-agent pay off, and can you auto-pick a pattern per task?

Dmitry Glhf — Mon, 15 Jun 2026 12:14:32 +0000

I ran six multi-agent patterns across three benchmarks and three models. By "a team of agents" I mean setups like critic-actor or an orchestrator with subordinates. Almost everywhere, the team lost to a single agent. It lost on accuracy and on cost, and on cost sometimes by 4x.

This could have been a dull "MAS isn't worth it, let's go home" note. But there was an exception. The harder the task is for the model, the better the team's chances of winning. So I have two questions. Does the system's topology affect the result at all? And if it does, can we automatically pick a suitable pattern for a specific task? The answer to the second one is no, for now, and the most interesting part is exactly why.

I show the examples on the FEDOT.MAS multi-agent framework, and all the experiment code (patterns, runs, and plots) lives in the repo. I'm measuring a tool I write myself. But the conclusions mostly come out against multi-agent, so this is more self-criticism than advertising.

MAS patterns

Each pattern is a way to wire several model calls into one system in a particular order. For comparison I used the following:

single. One agent, one pass. The baseline everything else is compared against.
chain. A two-step chain. One agent breaks the task down, the other solves it from the breakdown.
voting. Two agents solve the task independently, and a third judge merges their answers into one.
eval_optimizer. A generator proposes an answer; a critic accepts it or sends it back for revision.
orchestrator. At each step a coordinator decides which subordinate works next, and keeps going until it says "done".
blackboard. Agents write to a shared board. These are a researcher, a skeptic-verifier, and an assembler. The framework supports all these patterns and their combinations, and a system is assembled from primitives via an overloaded + operator. Here's the eval_optimizer pattern:

from fedotmas.adapters.pydantic_ai import PydanticAI
from fedotmas.sdk import agent, Condition

llm = PydanticAI("openrouter:openai/gpt-oss-20b")

gen = agent(
    "gen", takes=dict,
    prompt="Solve the task; if the critic left a comment, take it into account.",
    input="Task: {task}\nComment (if any): {verdict}"
)
critic = agent(
    "critic", takes=dict, labels=["approve", "revise"],
    prompt="Review the solution: approve if correct, otherwise revise.",
    input="Task: {task}\nSolution: {draft}"
)

system = gen.into("draft") + critic.into("verdict")

# draft -> verdict, and round and round until the critic says approve
loop = system.loop(until=Condition(key="verdict", op="eq", value="approve"))

# start point and run
start_point = {"task": task, "draft": "", "verdict": ""}
answer = await loop.run(start_point, llm=llm)

The other five are assembled similarly. One interesting note: a node in the system can be a single LLM agent, an entire MAS, or just a plain mechanical Python function.

Experiment setup

For the tests I took several standard benchmarks of varying difficulty from different domains. GSM8K is grade-school math. MMLU is multiple-choice questions across domains from chemistry to history. LogiQA is logic problems. As executor models I used gpt-oss-20b, ministral-8b, and llama-3.1-8b. I didn't take the whole benchmark, just a 100-question subsample. For the multi-domain benchmarks I used a bootstrap sample with a fixed seed. Caveat: at this stage I only considered benchmarks that don't require tools for the agents.

I gave each role a simple, direct prompt with no special tuning per pattern, and kept the prompts at the same level across all benchmarks. This was to measure the system architecture, not prompt engineering.

We'll also need a reference point: a per-task oracle. This is as if an omniscient someone picked the ideal pattern for every task. It's the ceiling that shows how much you could squeeze out of selection in principle.

One important caveat. The oracle chooses in hindsight, from already-computed runs. So there's an assumption that on a rerun the chosen pattern would give roughly the same accuracy. That's why the choice is taken over the average of three runs, not one, so it reflects the pattern's stable behavior on the task, not a one-off lucky hit.

First result: easy tasks

Let's start easy. On GSM8K and MMLU the picture is the same. The single agent sits on the cost/accuracy frontier, and there's nowhere to push it from there.

On GSM8K with gpt-oss-20b, single gives 0.94 for 398 tokens per task. The best multi-agent pattern gives 0.95, but at 1692 tokens. A one-point difference sits within the noise, and you pay 4x more. On MMLU it's exactly the same. Any "smart" layer on top of a single pass either doesn't help or slightly hurts, while always costing 2-4x more.

The harder the task, the more selection makes sense

Now the interesting part. If it were only "MAS isn't worth it", the article would end here. But let's look at that same gpt-oss-20b across benchmarks in increasing difficulty.

The model is the same, but the gap between the best fixed pattern and the per-task oracle grows. On GSM8K it's +3 points, on MMLU already +6, and on LogiQA +9. So the harder the task is for the model, the more you could gain by picking a pattern per individual task.

This headroom reflects how hard the task is for the model, not the model's size. The small ministral-8b on easy GSM8K gives only +1 point of headroom, because it's strong there. But llama-3.1-8b of the same class, yet weak on these tasks (0.63-0.80), gives +11…16 points. It's not about parameter count, but about how hard the task is for that specific executor. In some cases a critic helps, in others a single pass is enough.

So there's something to choose between.

If selection matters, let's select automatically

An obvious solution suggests itself. Put a selector in front of the system, an agent that picks a pattern from the task text. In FEDOT.MAS the selector is a single agent with a limited choice from a menu of patterns. Did it improve quality?

No.

Here are the numbers on LogiQA with gpt-oss-20b. The single agent gives 0.77, random pattern choice 0.73, the selector only 0.71, and the oracle 0.89. The selector didn't just fall short of the ceiling. It came out worse than both the single agent and a random guess. And it's like that everywhere. In none of the columns did the selector beat the single agent. It only beat random choice where the options contained an obvious outlier, like the looping orchestrator.

It's not that the selector is "dumb". It's about information. Which pattern solves this particular task on this particular model is an empirical fact that lives in the results matrix, not in the task text. You can't derive it from the wording alone. Tasks look the same but get solved by different patterns for reasons that simply aren't in the text. So a smarter judge won't save you here either. The problem isn't intelligence, it's the absence of signal at the input.

What to do about it? One option is to train the selector on examples like "this kind of task on this model is best solved by this pattern". That's fine-tuning already, and a topic for a separate discussion.

Conclusions and what's next

What did we learn, and what might be useful in practice?

An orchestrator on a weak model is dangerous. Without a hard step limit it loops and burns tokens.
Pattern selection only pays off on tasks that are hard for the model. It's worth honestly noting a weak spot in the measurement. In patterns with verifier checking, the verifier is the same model as the generator, so it essentially judges itself, with no independent signal. That's why the honest next step leads to where multi-agent has a principled way to win. That's code with real test execution, like HumanEval, where the critic stops being a "second opinion" and becomes a test runner with a real signal. Or hard benchmarks like GAIA2 that require tools and interaction with the outside world.

For now, all the results are on GitHub and can be reproduced on other models.

How Finite State Machines Make an Agent More Reliable, and Where pydantic-graph Fits In

Dmitry Glhf — Thu, 04 Jun 2026 21:23:30 +0000

When you write a library, sooner or later you run into the engine. Not the pretty external interface, not the wrappers, but the part inside that drives the process through its states: it generated something, checked it, decided what to do next, repeated. A couple of flags, a while loop, a big if in the middle, and a month later you can't even remember which transitions are possible in there and why one of the branches is unreachable.

Recently I was building exactly this kind of engine and stumbled onto a library that does the job noticeably more cleanly. It's called pydantic-graph. Hardly anyone writes about it, even though the whole of pydantic-ai (the agent framework from the authors of Pydantic) is built on top of it. Below I'll walk through it with a concrete example: a reliability harness for weak language models.

Let me clarify the term up front, since it's everywhere right now. A harness isn't just MCP, skills, and memory. It's also robustness, including for very small models. It's that second part I'm using as the example. But the article isn't so much about models as about the approach itself. The main idea is simple: this is a convenient way to assemble an engine for anything that has states and transitions, without drowning in your own loop.

The problem: the model is capable, but trips over small things

The example I worked everything out on is this:

Take a small model (in my case it's Llama 3.1 8B Instruct) and ask it to solve problems from HumanEval+ (a code benchmark for LLMs). Low-resource models don't exactly fail on such tasks: they succeed about half the time. Off the cuff, with no scaffolding at all, this model scores a pass@1 of around 0.61. Roughly speaking, about two attempts out of five end up in the trash, even though the solution is usually almost correct and stumbles on something trivial.

I ran 20 samples per problem across all 164 problems and looked through the error log. Here are the characteristic examples:

NameError: name 'encode_cyclic' is not defined. The model called a helper function that it forgot to write out itself.
max() arg is an empty sequence. The empty input isn't handled, and the solution crashes on it.
The most common case: the code runs without a single complaint, but returns the wrong thing on a hidden test. The logic diverged from the requirement by one step.

All these failures share a common trait, and it matters for everything that follows. Each of them is visible automatically, and once an error is visible without a human in the loop, you can react to it without knowing anything about the model's internals and without touching its weights. That observation is where the next idea came from.

The same 164 problems, 20 samples per problem, temp=0.2. Figures are for hidden tests (plus), with public tests (base) in parentheses. The pass@k curve (number of attempts):

k	plus	base
pass@1	0.607	0.665
pass@2	0.654	0.712
pass@5	0.698	0.758
pass@10	0.722	0.782

The idea: the engine drives the agent

The idea is actually trivial. We treat the agent as a black box and run it through a series of special nodes that work with that box. Generated an answer, checked it; if it's bad, we append to the prompt what the problem was and generate again. Once the attempt budget runs out, we return the best of the variants and honestly mark it as imperfect.

The crucial condition this was all built around: in this loop, only the model should think, and only in one place. Generation is so far the single step where there's any nondeterminism. Checking, the "fix or give up" decision, and assembling the next prompt are ordinary, predictable code.

A construction like this is natural to describe as a finite state machine. Nodes are states, edges are transitions, with strict types at the seams. This is where the library enters the frame.

pydantic-graph: a state graph built on types

It installs separately; you don't need to drag all of pydantic-ai along with it:

uv add pydantic-graph==1.105.0  # version at the time of writing

What's worth knowing about this library: it's a state-graph engine built entirely on types. A node is a class, and a transition is expressed by the return type of a method. It sounds unusual, but in practice it's convenient, because the type checker and the IDE can see the topology of the graph: you can't silently draw an impossible edge.

The library itself doesn't have to know anything about language models. Yes, pydantic-ai is built on top of it, but at its core it's a general-purpose state machine. It suits any process with states: an order-processing pipeline, a state machine in a game, retries when calling an external service. What won me over is that it reads like ordinary Python without hidden magic, while the graph stays verifiable.

The agent contract: a prompt in, a string out

For the engine to work with any agent, it needs a minimal shared language. As a first approximation I decided to limit it to a single function: a prompt goes in, a string comes out.

This is the narrow waist through which anything can be threaded into the engine. In this design, AgentFn doesn't depend on scale, so you can wrap it around any chunk of the system. You can put the harness on a single agent inside a multi-agent system, in which case AgentFn is exactly that agent, while the planner and the other executors stay outside.

Or you can wrap it around the whole team. The planner, the executors, all the internal orchestration get hidden inside, and from the outside the harness still sees a single agent: a prompt in, a string out. The engine doesn't care what's going on outside.

How the graph is built

The working nodes return into a central decision node (Decide).

Here's what one node looks like in code. Pay attention to the signature of the run method.

That -> Validate is the detail that kept me on the library. It's not a comment or a line in a config; it's a real type annotation. The type checker knows that from the generation node you can only get to the validation node. If I mistakenly return a node that shouldn't be reachable at that point, it gets flagged in the IDE before I even run anything.

Everything is assembled with a builder.

A quick word on the roles of the nodes. The generation node is the only place with the model. The validation node is deterministic: it reduces the messy semantics of an answer to a simple verdict: passed or not, what kind of error, a short reason, the fraction of checks passed. The decision node (Decide) is pure logic over that verdict, the history of attempts, and the remaining budget, with no model inside. The repair node assembles the next prompt from the last error and calls generation again. The fallback node fires when the attempts run out: it returns the best-scoring variant and marks the result as degraded.

Validators: swappable checks in a single slot

The validation node (Validate) doesn't know in advance what counts as a correct answer. It accepts any callback of the form "string in, verdict out," and that turns it into a slot you can plug anything into.

There's room to get creative here. The cheapest checks are mechanical: parsing, a linter, type checking, validating JSON against a schema, the plain "does it even run?" They're universal and work on any string, but the signal they give is weak: they only catch format issues and crashes. My sense is that the strong signal is always domain-specific (for code, that's running the tests). On top of that you can hang a model in the judge role, either one judge or a whole ensemble.

What's more, a combination of validators is itself a validator, from the engine's point of view. So no matter how many there are, to the engine it's always a single slot. The complexity hides in the combinator, and the graph stays the same.

The repair node and where it can grow

Right now my repair node is fairly simple: here's the error, regenerate taking the feedback into account. A kind of prompt engineering. But you can build in different repair strategies. For example, route by error type: if it crashed at runtime, try a bigger model. Or run several models supplied by the user and take the best candidate.

What came out in the end?

Mode	pass@1 on HumanEval+
Baseline	0.607 ± 0.034
With harness	0.623 ± 0.035

The harness raises pass@1 from 0.607 to 0.623. The deviation came out to ±0.034, and at first glance the gain dissolves into it. But both modes were run on the same 164 problems, so what you should look at is the paired per-problem difference, and that comes to +0.016 ± 0.007, i.e. consistently positive, if small.

What I liked along the way, beyond the numbers themselves: all of it turned out to be extensible and strictly typed. The graph reads directly in the IDE, and the transitions can't be mixed up unnoticed.

Conclusion

The idea of applying finite state machines in development isn't new in itself. But the library that implements it struck me, personally, as interesting and underexposed. I won't claim everyone needs it. Nor will I risk saying that finite-state logic is the one right solution for the idea in this article.

If the model is already fine-tuned to answer reliably in the required format, the corresponding harness validator simply stops doing anything, and the absence of a gain here is a perfectly logical outcome. If you have a completely linear pipeline with no branching, pydantic-graph will be overkill, since a plain function is enough.

But if the process branches, has states, and it matters to you that the IDE holds its topology for you, the library fits well. And, to repeat, it doesn't have to be about language models at all.

All the code is packaged into a library and available on GitHub.

Automating machine learning with AI agents

Dmitry Glhf — Tue, 06 Jan 2026 01:08:59 +0000

When solving competitions on Kaggle, you start to notice a pattern. It's easy to create a baseline: upload the data, run CatBoost or LightGBM, and get the baseline metric. It takes half an hour. But to get into the top solutions, you need to try dozens of preprocessing options, hundreds of feature combinations, and thousands of hyperparameter sets.

Existing AutoML systems don't help much. They work according to a fixed scenario: they try a predetermined set of algorithms, select the best one according to the metric, and return the result. AutoGluon trains several models and creates a multi-level ensemble, but each run starts from scratch. TPOT generates a pipeline through a genetic algorithm, but does not learn from the mistakes of previous runs.

The main problem is that these systems do not reason. They do not analyze why a particular approach worked or failed. They do not adapt to the specifics of the task. They do not accumulate experience between runs. Each new task is like the first for them.

Humans work differently. If a data scientist sees unbalanced classes, they immediately know that stratification and threshold selection are needed. If they have seen a similar task before, they apply what worked then. If the first attempt fails, they analyze why and try a different approach.

With the advent of language models, it became possible to create a system that works closer to humans. LLMs can analyze data, reason about method selection, and learn from examples. But one model is not enough. It can miss an obvious mistake or get stuck on the wrong approach. We need an architecture that allows the system to check itself and accumulate experience.

The idea with two agents

The first version was simple: one agent receives data, trains the model, and returns predictions. The problem quickly became apparent. LLM sometimes skips data checks or forgets to process gaps.

The solution came from reinforcement learning. In Actor-Critic methods, one agent acts, while the second evaluates these actions. Why not apply this approach to AutoML?

The Actor receives data and a set of tools for analyzing, processing, and training models. It explores the dataset, decides what steps are needed, and generates a solution. The Critic looks at the result from the outside and checks if everything is done correctly. If the Critic finds problems, the Actor receives feedback and tries again.

This architecture solves a key problem: one agent may make mistakes, but two agents with different roles will catch most of them.

The diagram shows how the system works. Actor has access to five specialized MCP servers with tools for working with data and models. Critic works without tools, only analyzing Actor's reports. There is an iterative exchange between them: the decision from Actor goes to Critic for evaluation, and feedback is returned. After each iteration, the experience is stored in memory, from where it is retrieved when working with similar tasks.

Tools for the agent

LLM can reason on its own, but it needs tools to work with data. I have divided them into several categories: data preview, statistical analysis, processing, and model training.

For example, the preview tool returns structured information:

{
    "shape": (150, 5),
    "columns": ["sepal_length", "sepal_width", "petal_length", "petal_width", "species"],
    "dtypes": {"sepal_length": "Float64", "species": "String"},
    "sample": [{"sepal_length": 5.1, "species": "setosa"}, ...]
}

The agent sees the dimension, column types, and first rows. This is sufficient to make decisions about further steps.

An important point for data processing: the agent must apply the same transformations to train and test. If the categorical feature was encoded as {“red”: 0, “blue”: 1} in the training sample, the same encoding must be used in the test sample. To do this, mappings are saved in JSON files:

mapping_path = Path(output_dir) / f"{column}_mapping.json"
with open(mapping_path, "w") as f:
    json.dump(mapping, f)

This is critical for categorical classification. If the target variable was encoded as numbers, the model will return numbers, but we need the original class labels.

Each training tool returns three things: the path to the model, the path to predictions, and metrics on train. Paths are generated with a timestamp and UUID, so the agent can experiment with multiple algorithms simultaneously without conflict.

Model Context Protocol

But what happens if you need to add more tools, for example, for a specialized domain? When the number of tools exceeds ten, problems arise with their support, management, and scaling.

Model Context Protocol, and in particular the FastMCP framework, solves this problem. MCP allows you to package tools into separate servers that the agent calls as needed.

I created five MCP servers: file_operations for working with files, data_preview for CSV previews, data_analysis for statistics, data_processing for transformations, and machine_learning for training models and ensemble predictions.

Evaluation of decisions

Actor generates a solution in the form of a structured report with four sections: data analysis, preprocessing, model training, and results. Critic receives this report and analyzes each section separately.

Instead of one LLM judge, I used four specialized ones. The first checks the quality of data analysis: whether the agent studied the distribution, checked for gaps, and detected anomalies. The second looks at preprocessing: whether gaps are processed correctly, categories are encoded correctly, and there is no data leakage. The third evaluates the choice of model and hyperparameters. The fourth analyzes the results and overall methodology.

judges = [
    LLMJudge(rubric="Evaluate data_analysis: Is exploration thorough?"),
    LLMJudge(rubric="Evaluate preprocessing: Are steps appropriate?"),
    LLMJudge(rubric="Evaluate model_training: Is selection justified?"),
    LLMJudge(rubric="Evaluate results: Are metrics calculated correctly?"),
]

Each judge returns a score between 0 and 1 with justification. The average score is compared to the acceptance threshold (usually 0.75). If higher, the decision is accepted. Otherwise, Critic forms feedback from all judges' comments and passes it to Actor for the next iteration.

This approach is more stable than a single judge. A single LLM may be too strict or miss an obvious error. Four specialized judges smooth out subjectivity.

Isolated workspace

When an agent works with files, it should not be given access to the entire file system. Isolation is necessary. I created a dedicated directory for each session in ~/.scald/actor/ with three subdirectories: data for copies of the source data, output for intermediate files, and workspace for models and predictions.

The source CSV files are copied to data dir. All tools work only within these directories. The agent cannot accidentally overwrite important files or read other people's data.

After completion, all artifacts are copied to the session directory with a timestamp, and the workspace is cleared. After some time, you can open this directory and understand what exactly the agent did: which models it trained (load them from .pkl), what metrics it received, and what steps it performed.

Memory and learning

After each iteration, the system saves the experience. The Actor report and Critic evaluation are recorded in the ChromaDB vector database. When the agent receives a new task, the system searches for similar past solutions based on semantic similarity using the Jina embedding model.

self.mm.save(
    actor_solution=actor_solution,
    critic_evaluation=critic_evaluation,
    task_type=task_type,
    iteration=iteration,
)

# Search for similar
actor_memory, critic_memory = self.mm.retrieve(
    actor_report=actor_solution.report,
    task_type=task_type,
    top_k=5,
)

The solutions found are passed to the agent as context. If the system has previously solved a similar classification task with unbalanced data, it will remember what helped then.

Interestingly, it is not only successful solutions that are useful. When Critic says, “you forgot to process the gaps,” this is valuable information for future tasks. Semantic search finds such cases as well.

Main cycle

When all components are ready, all that remains is to put them together. The cycle runs until the maximum number of iterations is reached or the Critic makes a decision. At each iteration, the Actor solves the problem taking into account feedback, the Critic evaluates the solution, the experience is stored in memory, and relevant context is extracted for the next attempt.

It is interesting to observe how the Actor learns from feedback. The first iteration is usually simple: basic preprocessing and one model. The Critic finds problems: “you did not check the class balance” or “you did not do feature engineering.” The second iteration is more accurate: the agent adds the missing steps, tries several models, and creates an ensemble.

Problems during development

Encoding the target variable

The first version failed on categorical classification. The agent encoded the target into numbers, trained the model, but forgot to decode the predictions back. The output was numbers instead of class labels.

The solution required explicit instructions in the system prompt:

If you encode target column, you MUST DECODE predictions before returning.
Use decode_categorical_label with the mapping path from encoding step.

Unique file names

When the agent experiments with multiple models, the files overwrite each other. I tried to delegate this to the agent via prompt, but LLM does not always generate unique names. The correct solution turned out to be to do this at the tool level with timestamp and UUID.

What was the result?

To fully solve the initial tasks of deep analysis, preprocessing, and training on data, it is necessary to continue developing the system by adding specialized agents and tools. Nevertheless, the system is already working: running it on a dataset for several iterations gives predictions, and all intermediate results are saved.

I tested the system on tasks from OpenML. On the christine dataset, it showed an F1-score of 0.743, outperforming Random Forest (0.713) by 4% and surpassing AutoGluon with FLAML, which failed to cope with this task at all. On cnae-9, the result was 0.980 against 0.945 for the best competitor, FLAML, which is 3.5% better.

There were also failures. On the Australian dataset, the system showed 0.836, losing to AutoGluon (0.860) and other baseline methods. Interestingly, on blood-transfusion, the result of 0.756 was better than Random Forest (0.712) and AutoGluon (0.734), but worse than FLAML (0.767).

The cost of running varies from $0.14 to $3.43 depending on the complexity of the task and the number of iterations. The running time is unpredictable: from a minute to half an hour.

In fact, the value of the result is not that the metric is better or worse than a particular AutoML framework. The value lies in the fact that modern LLMs allow for more intelligent automation of routine tasks, while the modularity of MCP paves the way for the creation of an ecosystem of specialized agents. This solves the original “fixed scenario” problem of classic AutoML: now the system can be adapted to any task by simply connecting the necessary agents, while maintaining a single cycle of iterative improvement and experience accumulation.

The limitations are clear. The system is good for tabular data with gradient boosting algorithms. For deep learning or time series, other tools are needed. Quality depends heavily on the size of the underlying LLM.

Usage

For those who want to try it, installation is simple:

pip install scald

You can use it either from the terminal via CLI:

scald --train data/train.csv --test data/test.csv --target price --task-type regression

Or via Python API:

from scald import Scald

scald = Scald(max_iterations=5)
predictions = await scald.run(
    train="data/train.csv",  # .csv or dataframe
    test="data/test.csv",
    target="target_column",
    task_type="classification",
)

To work, you need an API key from a provider compatible with OpenAI (for example, OpenRouter). You will also need a key from Jina for embeddings in the memory system (the service provides a large number of free tokens upon registration).

All code is packaged in a library and available on GitHub.