Amid a month of models designing proteins and topping coding leaderboards, a quieter paper landed that's a useful counterweight: a benchmark called Reconstruction, which measures whether a model can recover a research paper's core idea from its pre-publication bibliography alone. Frontier models score between 3% and 15%. That number is worth sitting with precisely because it's boring news in a loud month.
Why the benchmark is built the way it is
The setup matters, because most "can AI do science" claims fall apart on leakage — the model has read the paper it's being tested on. Reconstruction is engineered against exactly that. It uses a hard temporal cutoff so the target work is after the model's training, anonymized reference IDs, information isolation from the seed, and frozen bibliographies. The task: given only what a paper cited, before it was written, infer what the paper did. Seven frontier models, 643 papers, machine learning plus five Nature-family science domains.
That's a genuinely hard and genuinely fair test of something specific — not "is the model smart," but "can it generate the novel connection that a real research idea is, from the same starting materials a researcher had." And the answer, for a single model working alone, is: mostly not. 3 to 15 percent.
The multi-agent number is the actually-interesting one
Here's where it gets useful rather than just humbling. A reference-only multi-agent pipeline — cross-model review plus a Swiss-tournament structure over candidate hypotheses, no web search — lifts the match rate to roughly 23–42% across the six domains. That's about a 2.4x improvement over the best single model.
Two things fall out of that, and they point in opposite directions, which is why I trust it.
The optimistic read: structure helps, a lot. The same underlying models, arranged into a system that generates candidates and tournaments them against each other, more than double their solo performance. This is the multi-agent thesis with an actual number attached instead of a vibe — and it lines up with what I keep seeing, that the harness around the model extracts capability the model can't express alone.
The sober read: 42% is the ceiling here, and it took a whole tournament pipeline to get there. Even orchestrated, these systems miss the majority of research ideas that real scientists reached from identical references. Multi-agent isn't a magic solvent for the thing models are worst at — genuine novel synthesis. It's a meaningful multiplier on a low base.
What I take from a low score
I find benchmarks like this more useful than the leaderboards models win, because they map the edges instead of the center. And the edge here is clarifying: models are extraordinary at generating and verifying against a check, and still weak at the specific act of original synthesis — inventing the connection nobody wrote down yet.
That's a good map for deciding what to actually delegate. Hand a model the work where the answer exists and needs finding, generating, or checking, and it's superhuman. Hand it the work where the answer doesn't exist yet and has to be invented from sparse signal, and you're in 3-to-15-percent territory, where it's a brainstorming partner at best and needs you in the loop. Knowing which side of that line your task is on is worth more than knowing this week's leaderboard order.
Wiring models into a multi-agent tournament to claw from 15% to 42% is also just a good technique to have in your pocket for hard generative problems — even if, this week, the headline is how far 42% still is from done.
If you've used a multi-agent setup to beat what a single model could do on a genuinely generative task, I'd like to hear how big the lift was — the honest numbers are harder to find than the hype.
Top comments (0)