Two of the biggest AI labs made the same declaration this week from opposite directions: biology is the next great test of AI, and today's models are not yet ready for it. New benchmark work shows frontier agents scoring as low as 17% on a task as basic as retrieving viral genetic sequences -- and returning different answers each time you ask the same question -- while OpenAI launched GeneBench-Pro, a benchmark built to measure the judgment-heavy analysis that real computational biology demands. The most useful finding is also the most humbling: the fix for the retrieval failures wasn't a smarter model, but a dumber, exact tool.
Key facts
- The failure: on the VirBench viral-retrieval benchmark, frontier agents scored as low as 16.9% and returned counts of 106, 15, or 5 for the same query.
- The fix: adding one deterministic lookup tool pushed accuracy above 90% for every model, peaking near 99.7%.
- The new yardstick: OpenAI's GeneBench-Pro targets multi-stage genomics and biomedicine analysis.
- Primary sources: OpenAI (GeneBench-Pro) and Anthropic ('Paving the way for agents in biology').
Here is why this matters beyond biology. A large language model works by predicting likely text, which is exactly the wrong tool for a task that demands an exact answer. Asked to retrieve every genome for a given virus, a model reasoning its way through the request is guessing at a factual lookup, and guesses vary -- hence 106 results one run and 5 the next. Biologists ran a benchmark called VirBench to quantify it and found frontier agents flailing, scoring as low as 16.9%. Then they handed the agent a plain, deterministic command-line tool (gget virus) that just queries the database and returns the truth. Accuracy jumped above 90% across the board, near-perfect at the top. The lesson, spelled out in Anthropic's 'Paving the way for agents in biology' work, is that in domains where exactness is non-negotiable you don't want the model doing the retrieval -- you want it orchestrating a hard tool that does. It's the difference between asking a brilliant colleague to recite a phone book from memory versus asking them to look it up.
That reframes what 'better at biology' even means, which is where GeneBench-Pro comes in. OpenAI designed it not to test rote lookup but 'judgment-heavy analysis' across multi-stage problems in genomics, quantitative biology, and translational biomedicine -- the kind of open-ended reasoning where there's no single command to call. The two efforts are complementary: VirBench exposes where models need deterministic crutches, and GeneBench-Pro probes the harder territory where judgment can't be outsourced to a tool. Together they mark biology as the labs' main frontier now that coding benchmarks are saturating.
The caveat worth keeping: a benchmark result is not a cure. Scoring 99.7% on retrieval once you're handed the perfect tool says more about the tool than the intelligence, and GeneBench-Pro's judgment tasks are precisely the ones where models still hallucinate confidently. But the deterministic-tool result is a genuinely portable insight -- it echoes the same lesson coding agents are teaching, that reliability comes from wrapping models in verified tools rather than trusting them to reason from scratch. It also pairs naturally with Anthropic's Claude Science workbench, launched the same week, which is built to give models exactly those verified tools. See OpenAI's GeneBench-Pro announcement and Anthropic's Claude Science for the two halves of the story.
Originally published on Ground Truth, where every claim is checked against the primary source.
Top comments (0)