Thirty-six percent

Give the top-scoring model in a 2026 benchmark a company handbook and a job to do under the handbook's rules. The model gets the whole job right 36.2 percent of the time.

The number comes from HANDBOOK.md, a benchmark published in July 2026 and revised in August. The design copies how employees work. An agent gets a binding policy document, between 20 and 124 pages long, and a multi-step task in finance, HR, insurance, logistics or medical billing. 65 tasks. Each task runs four times per model. A completed run averages about 17 steps and 30 tool calls.

The grading is the part I like. No model grades another model. 824 programmatic criteria check what the agent did, and a run passes only if every criterion passes. One missed rule fails the run, the same way one missed rule fails an audit.

The results across 30 configurations of 20 models from 11 providers:

The best configuration passed 36.2 percent of trials. The second passed 34.2 percent. The third passed 23.5 percent. In the authors' words, "most frontier models remain below 25%."

Read those as working days. Your new hire follows the handbook on one task in three. On the other two, something gets skipped.

The authors read the failed runs and grouped what went wrong. The immediate request overrode the standing rule. A required check ran and the agent ignored the result. A verification step got skipped and the agent assumed success. And then the pattern I care about most. In Section 6 the authors write:

"Nearly every failed trajectory ends with a confident statement that the handbook was followed."

Read the claim carefully. "Nearly every" comes from reading the failures, not from a measured rate. But the direction is clear. The agent does not stop and say "I skipped step four." The agent finishes the job wrong and reports the job done right.

Two things follow for you.

First, a better model does not close the gap on the timescale you care about. The spread between configurations is wide, and the top score is still a failing grade by any audit standard you would accept from a person.

Second, the report at the end is the least reliable part of the run. If your process reads the agent's summary to decide whether the work is done, the benchmark says your process inherits the error rate and hides the error.

HANDBOOK.md got one thing right worth copying. The benchmark does not ask the agent whether the agent complied. The benchmark checks the record against 824 rules. Merlin works from the same idea. Before Merlin claims a run finished, rules check the task ledger, the files read and the bytes written. The model's own summary of the work counts for nothing until the record agrees.

Thirty-six percent is not a reason to stop using these models. The number is a reason to stop taking their word for the result.