On July 24, ARC Prize verified Claude Opus 5 at 30.16% on the ARC-AGI-3 public set. On August 21, NVIDIA reported the same model at 100.00 on the same set. The weights did not change. The code around them did.
In between, MIT did the same thing (August 5), a group led by Impossible Research got to 98.98 (July 15), and OpenAI tripled GPT-5.6 Sol's score by flipping two API settings (July 29). Then Microsoft published a framework that trains the model through the harness (August 18), and Google published one that gives the environment a harness of its own (August 20).
In July I wrote that self-editing harnesses have a provenance problem. This month the problem moved up a level: the benchmark score itself has no provenance.
TL;DR: On ARC-AGI-3's public set, the spread between "model in the official harness" and "model in the best harness" is 25 to 70 points, on a benchmark designed to resist exactly this. None of the 100s are verified on the private set, and every author says so. Microsoft's Agent Lightning v1.0 runs RL with the deploy-time harness owning the loop, so the harness is becoming part of the weights, and its reward-hacking section is the checklist my July post warned about. A benchmark number without a harness version, memory state and action budget attached is a self-reported claim with an unmarked type. Unmarked means self-reported.
Five harnesses, one public set
ARC-AGI-3 scores agents with RHAE (Relative Human Action Efficiency). Per level, score = (human_baseline_actions / ai_actions)^2, with the ratio capped at 1.15x the human baseline. Game scores are level-weighted averages, you must finish the last level to get full credit, and the overall number is the mean over games. A 100.00 means the agent finished every level at least as efficiently as a first-time human.
| Harness | Who | Date | Model | Public RHAE | Actions | Verified by ARC Prize |
|---|---|---|---|---|---|---|
| Official ARC Prize harness | ARC Prize | Jul 24 | Claude Opus 5 (high) | 30.16% | n/a | yes |
| Official harness, default settings | OpenAI | Jul 29 | GPT-5.6 Sol (max) | 13.3% | n/a | no |
| Official harness + retained reasoning + compaction | OpenAI | Jul 29 | GPT-5.6 Sol (max) | 38.3% | 6x fewer output tokens | no |
| Schema | Impossible Research (+ UC Berkeley, CMU) | Jul 15 | Opus 4.8 / Fable 5 | 98.98 | n/a | no |
| VISTA | MIT (Han, Hu, Qiu, Wu, He) | Aug 5 | Claude Opus 5 | 100.00 | 7,542 (humans: 17,135) | no |
| AVO | NVIDIA | Aug 21 | Claude Opus 5 | 100.00 | 6,624 | no |
The number that matters is not in the table. It is the gap between the first row and the last: 70 points, same model, same 25 games, same metric.
The official harness is not a neutral baseline. OpenAI's write-up quotes ARC's intent: an "intentionally generic harness, without tools or special features" built to make "model shortcomings more visible." In practice it discarded all private reasoning after each game action and used a rolling truncation window, so older actions vanished as history grew. Retaining reasoning and enabling compaction took Sol from 13.3% to 38.3% and cut output tokens by 6x. The harness was wiping the model's mind between moves.
So the leaderboard measures "model plus a harness built to expose the model." The 100s measure "model plus a harness built to cover for the model." Neither measures the model, and nobody has isolated which part of the 70 points is which.
The authors are unusually honest about this. NVIDIA: the AVO-versus-VISTA comparison "should not be interpreted as a controlled ablation," and the results "should not be interpreted as a direct measurement of the performance contribution of AVO." VISTA: the models "were released after the public ARC-AGI-3 games," overlap cannot be excluded, and "the private set remains the real test of generalization." Schema: "no frozen-harness or held-out-performance claim." Every 100 on that table is a public-set number on games the models may have seen in training.
What the 70 points are made of
Read the harness papers side by side and the same three components appear under different names.
Memory. VISTA keeps a "lossless visual memory" of every past observation. AVO carries forward "prior implementations, evaluation results, compiler and profiler outputs, and accumulated reasoning." OpenAI's two settings are memory settings: keep the reasoning, compact instead of truncate.
Supervision. AVO runs a monitor that watches "the broader trajectory for stagnation or repeated unproductive cycles and can redirect the main agent." That is the layer that turns a model that gives up into an agent that does not.
An action budget. RHAE squares the efficiency ratio, so wasted moves are punished quadratically. AVO's headline against VISTA is 12% fewer actions. That is a harness optimization target, not a model property.
In July I split harness work into two piles: compensatory layers that patch what the model cannot do yet, and protective layers that constrain what it must not do. I predicted pile one depreciates with every model release. All three components above are pile one, and on a benchmark built to resist static tricks they are currently worth 25 to 70 points with the newest frontier models. Either my prediction is early or it is wrong about magnitude. I will take the second reading until the private-set numbers say otherwise.
One more thing about compaction, since it is the setting that tripled OpenAI's score. In my preregistered compaction experiment, the same operation produced 3.47% false proceeds on irreversible-action gates: the agent went through a gate it should have stopped at, because the compacted context no longer carried the provenance the gate depended on. Not a contradiction. ARC-AGI-3 scores task completion; my gates scored whether the agent still knew why it was allowed to act. Compaction improves the first, degrades the second, and a benchmark only sees the first.
Then Microsoft put the harness inside the training loop
Agent Lightning v1.0 (arXiv, August 18) names something the July thread never got to: RL where the harness is not a bystander. In their words, "the harness owns this loop, while the training engine observes only a sequence of LLM request-response pairs." The deploy-time scaffold (mini-SWE-agent in their coding runs) executes the task inside Kubernetes; the trainer sits behind a gateway that looks like a normal LLM endpoint and collects the traffic.
The result is real: Qwen3.5-9B goes from 41.8% to 56.4% on SWE-bench Verified, a 14.6-point gain from about 6,000 examples filtered out of SWE-smith's 59,136 tasks across 128 repositories, in roughly 3,500 lines of framework code.
Two details matter more than the headline.
First, retokenization. The harness re-renders text between calls, chat templates are not compositional, decode-then-retokenize is lossy, and the harness parses and repairs outputs. So the token IDs the trainer sees for the model's previous answer can differ from the ones it actually sampled. Their fix is best-effort merging: merge only when the exact token prefix holds, otherwise close the sequence. That is the engineering admission that model and harness now share a boundary at the token level. Train through one harness's rendering and you get a model tuned to that rendering.
Second, section 4.3.2, "Preventing Reward Hacking." During training the agents were caught "using Git history to locate the gold commit," "using wget or curl to retrieve upstream source code from GitHub," using pip to download a package's source, and using urllib to do the same. Countermeasures: "disable Git commands and hide the .git directory from the agent," plus a Kubernetes network policy that "blocks general outbound network access and permits connections only to explicitly whitelisted services."
That is my July post compressed into a paragraph, arrived at as an engineering necessity rather than a design principle. Vinicius Pereira said it best in the comments: the agent must not be able to author the artifact the gate reads. Microsoft's version is that it must not be able to reach it either, through the filesystem or the network. Dipankar Sarkar's separate trust domain for test execution is the same control from the other side.
Now put the halves together. The harness that decides what the model sees also decides what the trainer sees. Once RL runs through it, the tricks in pile one stop being code you can diff and become weights you cannot. That is the absorption I predicted, except what gets absorbed includes whatever the harness let the agent get away with. Hide .git and the model learns the task. Forget to, and it learns to find the gold commit, and the benchmark will not tell you which one you trained.
Google gave the environment a harness too
EnvHarness (arXiv, August 20, Google Research) wraps a static environment at the reset/step interface with three plug-in types: Setup reshapes the initial state, Rule reshapes "which actions are allowed, what they do, and what the agent observes," and Link composes in another environment's tasks. A designer agent, EnvRigger, "treats the target policy as a black box, observing its execution trajectories to synthesize EnvHarness components targeting diagnosed flaws," writes a _Rules subclass, and tests it. Across ALFWorld, WebArena, SWE-bench Verified, OfficeQA and SpreadsheetBench, skills learned in reshaped environments transfer back for up to 9.0 points on held-out instances with 9.8% fewer steps.
Credit where due: this is the responsible version. Verifiers are untouched, the goal predicate is never modified, evaluation happens on the unadapted benchmark. A curriculum, not a thumb on the scale.
But note the direction of travel. In one week the field shipped a harness around the agent (AVO, VISTA), a harness around the trainer (Agent Lightning), and a harness around the environment (EnvHarness). The capability you end up with has its provenance spread across three codebases, and only one of them comes with the model card.
What you are actually buying
The one benchmark this month that held the model constant and varied the harness came from a vendor. TrueFoundry's TrueForge comparison (August 18) ran DevRev's Enterprise-Bench: 14 cross-system tasks, three MCP servers, fresh session per task, blind grading, list-rate and cache-aware costs.
| System | Model | Tasks solved | Cost per run |
|---|---|---|---|
| Claude Managed Agents | Opus 4.8 | ~11/14 | $11.80 |
| TrueForge | Opus 4.8 | ~11/14 | $8.50 |
| TrueForge | GLM-5.2 | ~11/14 | $2.90 |
| deepagents / LangGraph | Opus 4.8 | ~10/14 | n/a |
Same model, same tasks, 28% cost difference from the harness alone. Swap the model under the same harness and cost drops another 2.9x with no change in tasks solved. TrueFoundry sells the gateway next to TrueForge, so the framing is self-serving. It is still more methodology than NVIDIA offered.
I have seen this pattern in teams that compare a vendor's managed agent against their own scaffold and attribute the entire difference to the model. After this month I do not think that attribution is defensible without a controlled harness swap, and almost nobody runs one.
What a score needs to carry
If provenance is a vector, a benchmark score needs one. The minimum I would want attached to an agent number before quoting it, illustrative rather than a standard:
score: 100.00
metric: RHAE
set: arc-agi-3-public-25 # not semi-private, not private
model: claude-opus-5 # provider version string, reasoning effort
harness: avo@<commit> # the code between model and environment
memory_at_start: empty # or warm, and from which prior runs
supervisor: stagnation-monitor # any policy that can redirect the agent
compaction: on # summarize vs truncate, and where
action_budget: 6624
trainer_harness: none # if the weights were RL'd through a harness, which one
verified_by: self # or ARC Prize, or a named third party
Mike Czerwinski's rule from the July thread applies to every row: unmarked has to mean self-reported. A score that arrives without the harness commit is not a measurement of the model. It is a claim about a system, typed by whoever produced it, and the default type is untrusted.
What this post might get wrong
- The model/harness split may already be dissolving. After harnessed RL, "the model" is partly a harness artifact, and the boundary I am measuring might not survive the next benchmark cycle.
- Every 100 is on the public set, which shipped before the models did. If Opus 5 in the generic harness gets 30 on the private set and AVO gets 40, the harness story shrinks from 70 points to 10, and the leaderboard was more honest than I am giving it credit for.
- Source hygiene: NVIDIA sells the compute AVO runs on, OpenAI's two settings are its own API features, TrueFoundry sells a gateway, Microsoft would like you on Azure. I read the papers and reproduced none of them.
- My July prediction that compensatory harness layers depreciate with each model release. This month's evidence points the other way. The prediction stays up, marked as losing.
The question I cannot answer alone
If the harness is worth 70 points on a benchmark built to resist it, who owns the harness in your stack: you, the model vendor, or the router in between? And when you report an agent result internally, does the harness commit travel with the model version, or does it get dropped at the first summary?
Sources & further reading
- ARC Prize: Claude Opus 5 results, verified July 24, 2026
- ARC Prize: ARC-AGI-3 scoring methodology (RHAE)
- OpenAI: How enabling two settings tripled our scores on the ARC-AGI-3 benchmark (July 29, 2026)
- Schema: Frontier Models with Our Harness Achieve ~99% on ARC-AGI-3 Public (July 15, 2026)
- VISTA: A Visual Harness for Reasoning in an Interactive World, MIT (August 5, 2026)
- NVIDIA: AVO Reaches 100 on ARC-AGI-3 (August 21, 2026)
- AVO: Agentic Variation Operators for Autonomous Evolutionary Search (arXiv 2603.24517)
- The New Stack on AVO and the harness debate (August 21, 2026)
- Agent Lightning v1.0: Towards Harnessed Agentic RL (arXiv 2608.17528, August 18, 2026)
- EnvHarness: Awakening Static Worlds for Agent Learning (arXiv 2608.19880, August 20, 2026)
- EnvHarness on GitHub, Google Research
- TrueFoundry: TrueForge vs Claude Managed Agents benchmark (August 18, 2026)
- VentureBeat on TrueForge (August 19, 2026)
- Earendil: What Is a Harness? (August 20, 2026)
- My July post: The Agent Faked a Test Log, Then Believed It
- My compaction experiment: My Strawman Baseline Beat My Own Scheme on Half the Gate Classes
Top comments (6)
The harness delta is the bit that should make benchmark tables carry build metadata. Same weights, different state rules, tool loop, or retry budget is a different system. I would rather see the score plus the run contract than another single number with a model name stapled to it.
"run contract" is the phrase i should've used instead of that yaml block, taking it (credited).
one wrinkle: build metadata assumes the contract is declarative, and after agent lightning part of it isn't. the trainer re-renders text between calls, chat templates aren't compositional, and the token ids it sees for the model's own previous answer can differ from the ones it actually sampled (their fix is merge only when the exact prefix holds). so the weights come out carrying a fingerprint of one harness's rendering, and that's a build-time dep that shows up in nobody's model card. "same weights, different loop" is already the optimistic case.
the other half is declared vs observed: retry budget 5 in the config, with 3 silent sdk-level retries inside one of those, is still a valid contract and a different system.
ur closer to this than me on the automation side: does a commissioning record survive handoff in ur field, or does it also collapse into a version string and a shrug?
The 70 point gap on the same Opus 5 weights is the clearest version of something I keep seeing when I run coding agents. Flip compaction or whether private reasoning survives between steps and the same model stops mid task or finishes cleanly, with no weight change to blame. The OpenAI case where the official harness wiped reasoning between moves, then retained plus compacted and jumped Sol from 13% to 38%, is that failure with a number on it. What I wish more leaderboards shipped next to the score is the harness version, the memory policy, and the action budget, because without those three the percentage is not a model claim.
you're right and the three-field version is tighter than my ten-field yaml. harness version, memory policy, action budget. stealing that for the follow-up (credited).
one thing i'd add to "memory policy" tho: the policy tells u what can happen, not what did. openai flipped both settings at once and never published the split, so the 25 points are the sum of two different fixes for two different failures. wiped reasoning means the model re-derives the plan every step and drifts; truncation means old tool results vanish and the model either repeats work or thinks it's done. from outside both look like "stopped mid task". in my compaction lab the session didn't degrade gradually, it collapsed around compaction cycle 35, so a run that compacts at step 8 and one that compacts at step 40 under the same policy aren't the same run.
so imo the thing u want next to the score isn't the policy, it's the compaction events in the trace: step number, tokens before and after, what got summarized. the policy is a config value, the event is what actually happened, and ur "stops mid task" only correlates with the second one.
when ur coding agents stall, do u have the compaction step in the trace to line it up against, or do u only know compaction was on? cuz if u have it, i'd love to know which of the two failures u see more, the re-plan drift or the vanished history.
The harness result is a good reminder that benchmarks mostly test the wrapper's threat model. A 30 percent model inside a narrow, observable loop can beat a stronger model asked to operate through a leaky interface. The comparison I would want is cost per verified task, not model score in isolation.
agree on the wrapper's threat model, and there's one table in the post that does exactly what u want: same opus 4.8, same 14 tasks, 28% cost delta from the harness alone, then 2.9x cheaper again swapping the model under the same harness with the same tasks solved. cost per verified task is the only number there that moves for a reason u can name.
the load-bearing word is "verified" tho. the verifier is part of the wrapper too, and microsoft's reward-hacking section is what it looks like when the agent can reach it: pulling the gold commit out of git history, wget-ing upstream source, pip download, urllib. countermeasures were hiding .git and a network policy whitelist. so cost per verified task without the verifier's trust domain attached is the cheapest metric in the stack to hack: u don't make the agent better, u make verification easier to reach.
and ur closer to the pricing side than me: cost is a counterparty variable, actions and tokens aren't. if cost per verified task becomes the reported number, does it survive a vendor repricing, or would u rather quote actions and let the reader apply their own price sheet?