A control layer wrapped around an unmodified language model reached 95.3 percent raw accuracy on Terminal-Bench 2.1, succeeding on all 89 tasks at least once across 445 trials. The system, called StateM, changed no model weights at all. Its final-score API usage came to about 15 dollars, against 574.68 dollars for the reference run it beat.
Key facts
- StateM reaches 95.3 percent raw accuracy on Terminal-Bench 2.1 across 445 trials, covering all 89 tasks.
- Final-score API spend was about 15 dollars versus 574.68 dollars for the GPT reference run.
- Ranked the number one paper of the day on Hugging Face with 129 upvotes; the code is public.
- Primary source: arXiv:2608.15089, with a project page.
The observation StateM starts from is one that anyone who has run a long agent task recognizes. "Long-horizon agents can fail even when their underlying models can solve the constituent steps," the paper says. "They may lose track of mutable state, fail to reactivate lessons from earlier executions, skip known procedures, or stop prematurely." The failure is not ignorance. It is bookkeeping.
So StateM does bookkeeping. It is an agent-native runtime that organizes execution around durable states, phase-local context, checked transitions, recoverable runbooks, and versioned procedural practices that both agents and humans can inspect. Think of the difference between a competent cook improvising from memory and the same cook working from a kitchen ticket system: same skills, but now there is a physical record of what stage every dish is at, what has already been checked, and what must happen before the next step is legal. Nothing about the cook changed. The error rate does.
The results run across four models. StateM raises GPT-5.5 xhigh from an 83.1 percent reference to 92.1 percent, above GPT-5.6 Sol Ultra's 91.9 percent. The runbook then transfers unchanged to GPT-5.6, where with Sol xhigh it hits the 95.3 percent headline. A frozen profile lifts GPT-5.6 Luna from 76.7 to 85.4 percent, above the 84.9 percent Sol xhigh reference. And under 38 dollars of adaptation raises DeepSeek-V4 Flash from its published 82.7 percent to 88.1 percent under standard timeouts, and to 89.1 percent on an 88-task common core.
Why it matters is the cost asymmetry. A frontier reference run cost 574.68 dollars in API usage. The runbook that beat it cost about 15 dollars to score and 52.22 dollars in total campaign spend on the DeepSeek side. If the reusable artifact in agentic AI is the runbook rather than the checkpoint, then the economics of catching up change fundamentally, because a runbook is a text file and a checkpoint is a datacenter.
This is not an isolated paper. Two others landed the same week doing the same thing from different angles. HarnessEval-W brings the harness paradigm to evaluation itself, decomposing each judgement into subproblems handled by specialized sub-agents and producing an inspectable evidence tree instead of a scalar score, applied to 18 world models over 330 cases. ClawGym II goes further and runs reinforcement learning through the harness: sandboxed rollouts, a serving proxy at the model boundary capturing calls, prefix-tree reconstruction of multi-turn trajectories, then PPO and GRPO over the recovered tree. It improves Qwen3-30A3B's Pass@1 on its benchmark by 9.98 points through OpenClaw and 14.81 through Claude Code, stable over 200 to 400 optimization steps. If you want the reinforcement-learning machinery underneath that, see our lessons on RL post-training and GRPO.
Two honest caveats. First, transfer is not free. The runbook moves from GPT-5.5 to GPT-5.6 untouched, but DeepSeek needed provider-specific rework. The harness is a reusable artifact, not a plug-and-play one. Second, the headline dollar figure compresses reality: 15 dollars is the final-score API bill, and the full DeepSeek adaptation and evaluation campaign came to 52.22 dollars, with about 37.02 dollars of that spent on adaptation.
We also checked something that would have changed the headline. A record on a retired benchmark is worth much less than a record on a live one, and our own prior reporting suggested Terminal-Bench 3.0 launched on August 13. It did not. The maintainers list Terminal-Bench 2.1 as released and active with a live leaderboard, while Terminal-Bench 3 remains under construction. StateM's number is on a benchmark people are still submitting to. The paper also notes its own submission pull request was open and not yet merged into the leaderboard at the time of writing.
This is now a repeated finding rather than a novel one, and the repetition is the point. We have covered a harness that moved DeepSeek by twenty tasks, models that rewrite their own harness for 16 points, and a stronger model building a wrapper that nearly doubled a weaker one's score. Our lesson on the harness explains why the same model can look two different sizes depending on the code around it.
Originally published on Ground Truth, where every claim is checked against the primary source.
Top comments (0)