I counted handoffs in my own agent graph last week. Six agents, eleven hops per task, and every single hop does the same dumb thing: a model finishes thinking, renders its internal state into a paragraph of English, and the next model reads that paragraph back in and starts over.
Two of those agents exist almost entirely to translate. Not to reason. To translate.
That is the decoding tax, and until I read SwarmEvolve I had filed it under "cost of doing business." The paper's premise is that the prose bus between agents is a design choice, not a law of nature. Hundreds of cheap agents, reasoning in latent space, handing off KV caches instead of sentences, running an evolutionary search loop. The reported result is best-known scores in 3 to 8 minutes on tasks where comparable frameworks need around an hour, and a Qwen3 8B coming out ahead of solutions AlphaEvolve found with frontier models.
I have not run it. I read the abstract and the method section twice and argued with it once. What follows is me comparing it against the substrate I actually ship, and I'm aware that's an unfair comparison in both directions.
What my hops actually cost
Concretely, one hop in my setup: the orchestrator writes a prompt of 600 to 1400 tokens depending on how much history it's carrying, the worker emits 300 to 800 tokens of reasoning plus a structured payload, and everything gets re-tokenized and re-encoded by the next model. In my measurements that's a few hundred milliseconds of pure serialization before anyone thinks, plus the prompt tokens on both ends.
The money is annoying. The information loss is the real problem. When a model finishes a chain of thought, it does an argmax over a distribution and I only keep the winner. The 0.31 / 0.28 / 0.24 spread that told me "this was close, watch it" is gone. I get a confident sentence and no uncertainty attached. Then I spend engineering effort trying to reconstruct the uncertainty downstream, usually by asking the next agent to rate its confidence, which is a model grading its own summary of a thought it no longer has access to.
What the latent substrate changes
If I'm reading the paper right, the swarm never renders. Agents operate on hidden states, and the handoff is the cache itself, so the next agent starts pre-loaded with the previous agent's actual computation rather than a description of it. Cheap agents become genuinely cheap because a partial forward pass is a fraction of a full generation. Hundreds of them stop being a budget question.
And because the loop is evolutionary, the fitness signal is execution, not opinion. Which is exactly why it works: in a search loop, nothing benefits from eloquence. The scorer runs the code and returns a number. Prose in the middle of that is pure overhead.
Here's the comparison as I'd actually make it:
| Prose handoff (mine) | Latent handoff (SwarmEvolve) | |
|---|---|---|
| What survives a hop | A summary, biased toward the winner | The distribution, plus the reasoning behind it |
| Cost per hop | Full generation + re-encode | Partial forward pass + cache copy |
| Wall clock | Serialized, seconds per hop | Batched, hundreds in parallel |
| Model freedom | Any model, any vendor | Same family, same tokenizer, same positions |
| Debuggability | Full transcript to read | Nothing to read |
| Failure mode | Agent contradicts itself loudly | Swarm agrees with itself quietly |
| Best fit | Human-readable artifacts, heterogeneous models | Machine-checkable fitness, high iteration count |
The failure mode I'm actually worried about
Not the latency. Not the infra.
Consensus collapse. If every agent in the swarm forks from a shared KV cache, they inherit a shared bias. Averaging over a correlated population isn't a swarm, it's one agent with extra steps and a bigger bill. Evolutionary search mitigates this with mutation and selection pressure, but the mutation operator is now the only source of genuine diversity, and I'd want to see how much of the reported gain survives when you ablate it. Prose handoffs have the same problem in a milder form: my six agents all read the same system prompt and they drift toward agreement too. At least with prose I can see the drift happening in the transcript. With latents I'd have to build probes into the cache to notice.
Second worry: serving. Passing caches between agents means the infrastructure has to keep them warm and addressable. Prefix caching in vLLM gets you partway, but this is a different contract, and it locks you to a single model family. No routing to a bigger model for the hard subproblem, no swapping in Claude for the writing step. I've built pipelines where that heterogeneity was the entire value.
Third: there's no transcript. When something goes wrong at 3am, I have a number that got worse and no conversation to read. I'd pair this with a sampled, periodic render of agent states just so I have something to stare at.
So what would I actually change on Monday
Not much, and that's the honest answer. The split is clean once you look for it: if the fitness function is machine-checkable and the agents are homogeneous, the inner loop belongs in latent space. Code optimization, kernel search, scheduling heuristics, anything where "did the test pass" is the reward. Keep the outer loop in text, because that's where a human approves things and where the artifact is prose anyway.
The bigger shift is in the first question you ask when designing a swarm. I've been asking "how many agents and what roles" for two years. The better question is whether they need to speak at all. If two agents are exchanging information that a scorer will later judge mechanically, the prose between them is a tax I chose to pay for my own convenience as a debugger, and I should stop pretending it's a reasoning requirement.
The paper's timing is good for another reason: it makes frontier-model evolution look expensive for no reason. An 8B in a better-performing substrate beating a frontier model in a worse one is the same lesson small-model people have been shouting for a year. Substrate over scale. Remove freedom, remove rendering, remove the tax.
I'll try to reproduce a small version of this against a toy optimization task and report back with real numbers, or with the specific reason it fell apart on my hardware. My guess is it's the serving layer, not the idea.
Top comments (0)