The same model scored three times higher on a hard reasoning benchmark because of two settings in an API. OpenAI reported on July 29 that switching on retained reasoning and compaction took GPT-5.6 Sol from 13.3% to 38.3% on the public task set of ARC-AGI-3, while spending six times fewer output tokens. Nothing about the model changed. What changed was whether it was allowed to remember its own thinking between moves.
Key facts
- The headline number: 13.3% to 38.3% on the ARC-AGI-3 public set, with output tokens cut about sixfold.
- When: July 29, 2026, in a post by OpenAI's Ilan Bigio and Ted Sanders.
- The setting that mattered most: the official harness discarded the model's private reasoning after every game action, and truncated history once the conversation passed 175,000 characters.
- Primary source: How enabling two settings tripled our scores on the ARC-AGI-3 benchmark.
ARC-AGI-3 asks a model to play unfamiliar 2D puzzle games it has never seen, scored on how efficiently it acts compared with a human baseline. It is designed to reward learning inside a single session -- forming a theory of the game, testing it, and revising it. ARC Prize deliberately runs it through a plain, generic harness with no tools and no vendor-specific features, on the reasoning that a simple harness makes model weaknesses visible and keeps comparisons between labs fair.
OpenAI went looking for why its model was flailing and found two properties of that generic harness. First, after every action the model's private reasoning was thrown away. Sol could still see a log of its past moves, but not the plans or insights behind them, so, as the post puts it, it "was asked to figure out the game anew" each turn. Second, the harness used rolling truncation -- once the conversation exceeded 175,000 characters, the oldest messages were deleted. So the model was losing both its thinking and, eventually, its observations.
Turn those two things off and the behavior inverts. With reasoning retained, Sol thought less before each action, because it no longer had to reconstruct the puzzle from nothing, and it held coherent strategies across a whole game. With compaction, long runs got summarized instead of amputated, so hard-won insights survived. The result was a higher score at a fraction of the output cost.
The analogy is uncomfortable but exact. Imagine being handed a Rubik's cube, allowed one twist, then having your memory of everything you had worked out erased -- but keeping a list of the twists you made. You would look like a much worse solver than you are. Our explainers on agent memory and context windows cover why this is the central engineering problem in long-running agents, and prompt caching explains the related trick of not paying twice for the same prefix.
What OpenAI concludes is the genuinely important part, and it works against the company's own marketing interest: "evals rarely measure models in isolation -- they also measure a bundle of less visible choices about API settings, harness design, and prompting." Anyone comparing two models is really comparing two model-plus-harness-plus-settings bundles. Our guide to how AI is benchmarked walks through why that makes leaderboards harder to read than they look, and the same ambiguity showed up in Claude Opus 5's verified ARC-AGI-3 result earlier this month.
Here is the part to hold onto, though. ARC Prize's own verified results page still shows Sol at maximum reasoning effort averaging 13.33% on the public set and 7.78% on the hidden semi-private set -- and credits it as the first model to win an ARC-AGI-3 public game, ft09, at 87%. The semi-private set is where claims about generalization get tested, and OpenAI's 38.3% says nothing about it. ARC's institutional position, that hand-configured harnesses can lift performance in one environment without transferring anywhere else, is not refuted by this post; ARC has previously documented a custom harness taking a model from zero to over 97% on one environment while doing nothing on another.
Both sides of this are defensible, which is what makes it interesting. OpenAI is right that a generic harness understates what a deployed system can do, and its recommendation -- if you are building on the API, use the settings the vendor ships in its own products -- is sound engineering advice. ARC is right that if every lab supplies its own scaffolding, cross-model comparison collapses. The honest caveat is that OpenAI published no per-setting ablation, no run-to-run variance, no replay artifact for the 38.3% run, and no semi-private replication, so we cannot say how much came from retained reasoning versus compaction. What we can say is that "which model is better" is now an under-specified question.
Originally published on Ground Truth, where every claim is checked against the primary source.
Top comments (0)