I've spent a decade reading papers and trying to build things from them. The hard part was never the idea. It's the gap between what the paper says and what you actually need to know to make it work.
There's a new benchmark that finally names this problem. IdeaAMBIG measures how underspecified research ideas are when it comes to implementation. The finding, as I read it: even the best models can't reliably fill in the gaps that papers leave open. The bottleneck isn't generation. It's codification.
Let me bust the myth properly.
The myth: "We need better idea generation."
Every lab, every startup, every grant application is chasing the same thing — the next big idea. We've built entire industries around brainstorming, around "ideation," around prompt engineering for novelty. And sure, LLMs can generate ideas all day. They can generate a thousand before breakfast. That was never the constraint.
The reality: ideas are cheap. Specifications are expensive.
I've spent more hours than I care to count staring at a method section, trying to figure out what the authors actually did. The learning rate is in a footnote. The batch size is "standard." The data preprocessing is "straightforward." The evaluation metric is "as in [12]." And somewhere in there, the one detail that turns a 42% result into a 68% result is just... missing.
This isn't a dig at the authors. Papers are compressed. They have page limits, reviewer expectations, and a culture that rewards novelty over reproducibility. But the consequence is real: the gap between "I understand the idea" and "I can implement the idea" is where most research dies.
IdeaAMBIG quantifies this. It takes research ideas and measures how much implementation-critical information is missing. The results, as I understand them, show that even frontier models — the ones we're all told can "reason" and "plan" — struggle to fill those gaps. They can paraphrase the idea. They can restate the motivation. But ask them to specify the exact architecture choice, the precise training schedule, the edge cases the authors handled implicitly — and they fall apart.
That matches my experience. I've tried to get agents to reproduce papers. The agent reads the abstract, gets excited, writes a plan. Then it hits the method section and stalls. It doesn't know the hidden details. It can't infer them. And it certainly can't ask the authors — they're busy, or the paper is from 2019 and the code is gone.
What this means for the people building agents
If you're building research agents — and I know a lot of you are — this benchmark is a wake-up call. The agent's job isn't to generate ideas. It's to bridge the gap between an idea and an implementation. And that gap is bigger than we thought.
The practical takeaway: don't optimize for idea novelty. Optimize for specification completeness. Can your agent produce a spec that a competent engineer could implement without asking a single clarifying question? That's the real test.
I've started adding this to my own evaluation pipeline. When I test an agent on a research task, I don't just check whether it produces a plausible plan. I check whether the plan is implementable — whether it specifies the details that matter. It's a different kind of eval, and it's harder. But it's the one that actually predicts whether the thing will work in production.
The uncomfortable conclusion
The field has spent the last few years obsessed with generating ideas. We've built benchmarks for creativity, for novelty, for "out-of-the-box thinking." But the real bottleneck — the thing that actually determines whether research becomes product — is the unglamorous work of specification. Writing down the details. Making the implicit explicit. Filling in the 40% of the paper that the authors left out because they assumed you'd figure it out.
I'm not saying idea generation doesn't matter. It does. But it's not the constraint. The constraint is codification — turning a vague research direction into something a machine, or a human, can actually build.
Maybe the next big benchmark shouldn't test how creative a model is. It should test how well it can read a paper and produce a complete, unambiguous implementation spec. That's the skill we actually need.
I haven't run IdeaAMBIG myself yet — it's new, and I want to see the full dataset before I trust the numbers. But the direction is right. We've been measuring the wrong thing for years. This is a correction.
Top comments (0)