Intology's Locus beat human-tuned Qwen3-1.7B (51.6% vs 49.4%) on PostTrainBench by scaling compute 64x, showing AI research agents need longer timescales.
Intology's Locus scored 51.6% on PostTrainBench, beating the official human-tuned Qwen3-1.7B's 49.4%. The win came from removing the benchmark's compute cap, scaling from 70 to 4,500 H100 hours per run.
Key facts
- Locus scored 51.6% vs 49.4% for human-tuned Qwen3-1.7B
- PostTrainBench+ budget: 4,500 H100-hours vs 70 baseline
- Agent runs 100 hours across a cluster, not 10 on one H100
- First public AI agent beats human post-training tuners
- Single run per setting; stability not disclosed
Intology's automated research system, Locus, has post-trained a model that beats the official human-tuned Qwen3-1.7B release, scoring 51.6% against 49.4% According to @rohanpaul_ai. The result is the first public demonstration of an AI agent outperforming human researchers at the specific task of improving another model's post-training.
The benchmark and its constraints
PostTrainBench is the benchmark used to score AI agents that post-train other models, and it gives each agent one H100 and 10 hours. That is a tight budget — roughly 10 H100-hours per run — designed to measure how efficiently an agent can navigate the post-training pipeline.
Intology's different path was to remove that limit and let agents run for 100 hours across a cluster. So Intology raised the ceiling and called it PostTrainBench+, taking the total budget from 70 H100 hours to 4,500. The 64x increase in compute changes what the benchmark measures: not just efficiency, but the ability to sustain a long experimental loop.
The capability being measured here is sustained experimental search: allocating compute, running parallel jobs, reading evaluations, abandoning weak branches, and scaling the promising ones. At 10 hours, an agent can barely complete one or two training runs. At 100 hours across multiple GPUs, it can iterate dozens of times, learning from each evaluation.
What this means for automated research
The result suggests that the bottleneck for AI-driven research is not algorithmic cleverness but timescale. When given the compute budget to actually explore, Locus found configurations that human tuners missed. The 2.2 percentage point gap on PostTrainBench is modest in absolute terms, but it is the direction of the delta that matters — an AI system outperformed the human baseline on a task that was previously considered human expertise.
One run per setting is a serious limitation. The source does not disclose how many seeds were run, whether the result is stable across random initializations, or how Locus compares to other automated research systems under the same relaxed budget. The company did not disclose the full methodology.
Still, this makes a good case that automated research systems need to be evaluated at the timescale where research decisions compound. If the trend holds, the next generation of post-training pipelines may be designed by AI agents that run for weeks, not by humans working in days.
What to watch
Watch for Intology to publish replication runs across multiple seeds, and for PostTrainBench maintainers to respond — either by raising the official compute cap or adding a 'research timescale' track. If Locus's lead holds across seeds, expect competitors to follow with similar long-horizon research agents within two quarters.
Originally published on gentic.news
Top comments (0)