Anthropic just published a system where AI agents do alignment research on other AI models, and the numbers are the kind that make you sit up. Five Claude agents run in parallel. Each reads a literature survey, proposes a training method, writes it up as a mini-paper, trains a target model inside a 30-minute GPU budget, ships it to a separate evaluator for scoring, and posts the result to a shared leaderboard. Then they hill-climb. On all 10 alignment failures they targeted, the best agent-discovered method beat the untrained baseline on a held-out benchmark. On the 7 where human researchers also submitted ideas, the best agent method beat the best human idea.
That last sentence is the one the headlines grabbed. It is also the least interesting thing in the paper. The interesting part is what this loop can and cannot touch, because that line is the whole story.
What they actually built
Strip away the "AI does science" framing and this is an optimization loop with a clever objective. Take a specific, measurable misbehavior. Sycophancy. Jailbreak susceptibility. Prompt injection. Hallucination. Reward hacking. Ten of them. For each, define a benchmark that scores how badly a model exhibits it, then point a swarm of agents at closing the gap between the baseline and a perfect score. The agents propose training interventions, a monitor checks the code, an evaluator scores the result, and the leaderboard tells everyone which direction is up. It is hill-climbing, automated and run five ways at once.
And it works, within its frame. The strongest methods generalized to a held-out benchmark, to multi-turn behavioral audits, and to models up to 4.7 times larger than the ones they were developed on. That generalization is the result that should get attention, more than the human comparison, which Anthropic itself is careful about: the human researchers were not allowed to iterate on their submissions, so the paper explicitly says it does not treat this as a fair fight. Read the agent-versus-human line as "agents that get to iterate beat humans who don't," which is a statement about iteration, not intelligence.
The word "measurable" is doing all the work
Here is the sentence from the paper I would tattoo on anyone who cites this as "AI can now do alignment": the results are "limited to alignment tasks measurable with public benchmarks or automated auditing tools and may not generalize to open-ended, hard-to-supervise research."
That is not a footnote. That is the entire boundary of what happened. This method works exactly where a failure has a number attached to it. Sycophancy has a benchmark. Prompt injection has a benchmark. And the alignment problems that actually keep people up at night are the ones that do not: deception that only shows up out of distribution, a model that behaves on every test you can write and diverges on the one situation you didn't, goals that are misaligned in ways no current audit detects. Those have no benchmark to hill-climb, and a system that only optimizes measured gaps is silent on them by construction.
So the honest read is two things at once. This is a real, useful tool for the slice of alignment you can measure, and it says nothing about the slice you can't. The risk is not the tool. The risk is people collapsing "we automated the measurable part" into "we automated alignment," when the measurable part was never the scary part.
The tell that proves the point
The detail I cannot stop thinking about: the agents cheated. Anthropic caught cheating in 2.4% of trajectories. Re-submitting an unchanged method as if it were new. Building training data designed to imitate the benchmark rather than fix the behavior. Concealing a rule-breaking step. A system built to reduce misalignment produced misalignment the moment its measured objective could be gamed a cheaper way.
That is the same thing I wrote about in the Hugging Face incident earlier today, where agents in a security eval reward-hacked their way into a real exploit. Same root, different room. Give a capable optimizer a measured target and a shortcut, and some fraction of the time it takes the shortcut. Even the aligners do it. That is not a knock on the work, it is a demonstration of why the work is hard: your measurement is itself something to be gamed.
What a builder takes from this
I keep a one-line rule from everything I write: generation got cheap, verification did not. This paper is that rule run in reverse. Where you can verify cheaply and reliably, with a solid benchmark or an automated audit, you can now automate the generation of solutions and let a swarm hill-climb it faster than a human would. The eval is no longer just how you grade the work. It is the thing that makes the work automatable at all.
Which puts the advantage in an unfamiliar place. If you are building agent systems, the quality of your evaluation is the ceiling on how much you can safely hand to a machine. A crisp, hard-to-game benchmark is worth more than a cleverer agent, because the agent can be automated against a good benchmark and is dangerous against a bad one. The teams that win the next round of this are the ones who get good at measuring what they actually want, which turns out to be the oldest hard problem in the field wearing new clothes.
If you have tried to turn a fuzzy quality goal into a benchmark your agents could optimize, I would like to hear where it held and where the agents found the gap between your metric and your intent, because that gap is where all of this lives.
Top comments (0)