AI’s most fervent and optimistic promoters promise a future where AI is innovating its way out of society’s biggest problems. AI systems will conduct scientific research, discover new drugs and materials, run engineering projects, and autonomously execute complex tasks usually requiring teams of experts working over long periods of time.
But fulfilling that promise requires more than a good prompt and basic context engineering. AI systems need to pursue goals over long horizons, encounter situations they’ve never seen before, learn from failed attempts, adapt their strategy, and keep making progress without a human telling them what to do next.
They need to solve novel problems that aren’t represented in their training data. But are today’s models actually capable of that?
That’s one of the questions ARC Prize is trying to answer with ARC-AGI-3, a benchmark that drops AI systems into unfamiliar environments without giving them the rules or even telling them what the goal is. They have to experiment, figure out how the environment works, and learn what winning means in this ambiguous context.
AWS engineers built an agent to take it on. Using Claude Opus 5 and the open source Strands Agents SDK, the agent completed all 183 levels across ARC-AGI-3’s 25 public environments with a 99.95% relative human action efficiency (RHAE) score over one 8 hour run, spending about $830 in tokens to do it. NVIDIA has recently reported a similar result, with its AVO agent, also using Opus 5, scoring 100% RHAE on the public game set.
Then compare that to ARC Prize's standard evaluation of Opus 5, which scored 30.16% on ARC-AGI-3. The model across these runs is the same, but the system surrounding it is very different.
These results give us some data showing how important agent harnesses are for getting what you want out of AI. A model by itself provides the reasoning and judgment, but the harness gives it a surrounding system for managing context, maintaining state, taking actions, observing their consequences, and generally interacting with the world around it. How you build the harness matters, so having open source examples of systems that successfully handle these kinds of long-horizon tasks gives us a chance to look beyond the benchmark score and understand which design choices actually made the difference.
I work closely with the Strands team at AWS, and the entire Strands ARC-AGI-3 harness is open source. So let's dig into it to understand how they did it and how the agent works. You can find the code for yourself on GitHub.
The public vs private game set for ARC-AGI-3
Before we get further into the details I do want to clarify one thing, especially for those of you who aren't deeply familiar with how ARC-AGI-3 works.
The public game set has 25 games that are essentially the practice exam, and humans can play these games too. But the official ARC Prize rankings come from private, held-out evaluation, and the 2026 competition runs offline on Kaggle where hosted API models aren't allowed at all. The public set gives researchers a common set of environments for developing and experimenting with their agent systems, while the competition evaluates them against new unseen environments.
That competition is still open. But that doesn't mean that the public results don't give us anything useful. Acing the public benchmark is still a strong signal about which agent architectures can successfully handle long-horizon tasks.
Inside the Strands harness
First off, it's good to know that the agent starts with zero game knowledge in its system prompt. You can check the entire system prompt out by taking a look at the code directly.
The available controls are presented as generic labels like ACTION1, ACTION2, and ACTION3, without telling the agent what those actions actually do. Even the board itself is represented as raw numeric values. If it wants to understand what anything does, it has to try something and observe what happens.
This prompt is the same across all 25 games, and everything the agent knows about a game, it learned by playing it. So if it doesn't know anything, how does the agent navigate the game board?
Before the agent takes its first turn, the harness writes the initial board state to a game log. Then the model is told to read that log, analyze the board, and decide what actions it wants to try.
To do that, the harness gives the agent a small set of tools for working with files and code. It can read files, search them with grep and regex, write and edit files, and execute Python. So even though the board isn't pasted directly into the model's prompt, the agent can inspect it through the log and use these tools to start figuring out what it's looking at.
Once the agent chooses an action, the harness executes it and records the action and resulting board state back to the log. The agent can then inspect what changed, form a theory about what its action did, and decide what it wants to try next. Each new action and resulting board state adds another piece of evidence to that growing history.
Keeping that history in a file rather than continually adding it directly to the context window is important because things would get unruly pretty fast. A single board is 64x64, and over hundreds of actions the agent can build up a massive interaction log that can grow to tens of megabytes.
The agent can use its tools to decide what parts of that history are actually useful, performing its own context engineering. It can search for previous actions, write Python scripts to compare board states or look for patterns, save notes about what it has figured out, and use those findings to decide what to try next.
Why these tools?
The specific tools the agent was given didn't come from nowhere. The approach was based on PRO-LONG, which tested how well an agent using GPT-5.5 performed on ARC-AGI-3 as it was progressively given more ways to work with its interaction history.
With read-only access to its history, the agent scored 23.1%. Adding grep and regex brought that to 27.2%. But the biggest jump came when the agent was given access to Python, pushing the score to 38.3%. Finally, adding the ability to write and edit files brought it to 41.2%.
That jump from giving the agent the Python tool suggests that giving an agent access to its history isn't enough, but giving it a way to programmatically analyze and derive insights from that history can make a significant difference in performance.
For the Strands run, across the 25 games, the agent wrote 734 scripts for itself. Some of those scripts captured pieces of what the agent had learned about how a game worked. It built parsers for interpreting board states, renderers for representing them, and even simulators of game mechanics.
The team describes some of these scripts as small world models. In 10 of the 25 games, the agent imported code it had written earlier and continued building on top of it.
Those tools also run inside a bubblewrap sandbox with no network access and writes are restricted to the agent's workspace. That means that the agent can analyze everything it has learned through its own interactions, but it can't go online and find a walkthrough or inspect the underlying game implementation.
Relative Human Action Efficiency
The agent passed all the levels and scored 99.95% relative human action efficiency (RHAE). RHAE is a measure of how efficiently the agent solves the game by comparing the number of actions it takes on each level against a baseline from humans playing the game for the first time. This is arguably a more important metric than completion alone because ARC Prize defines AGI as “a system that can match the learning efficiency of humans.”
An agent could eventually solve a game by taking thousands of actions and exhaustively trying every possibility, but that wouldn't demonstrate the same learning efficiency as a human. RHAE helps us measure that difference. This is the breakdown for the Strands agent:
| Compared with human baseline | Levels | Share |
|---|---|---|
| Beat human baseline by enough to reach maximum score | 150 | 82.0% |
| Beat human baseline | 7 | 3.8% |
| Matched human baseline | 3 | 1.6% |
| Used more actions than human baseline | 23 | 12.6% |
Across 160 of the 183 levels, the Strands agent matched or beat the human baseline for action efficiency. And on 150 of those levels, it performed well enough to receive ARC Prize's maximum per-level efficiency score. So even though the overall score is 99.95%, it performed better than the human baseline most of the time. You can see how scoring works in ARC Prize's methodology docs.
What this means
This result on ARC-AGI-3 is impressive, and because it's open source we can all learn from how the pattern works. The takeaways for me are: externalize the history, give the agent tools to query it, let it write its own code to turn what it learns into insights and reusable programs, and let it decide what stays in active context for the next decision.
The code is surprisingly easy to understand, and the patterns could be used directly with whatever agent you're building. Go check it out.
- Strands Harness Code: https://github.com/strands-labs/benchmark-harnesses/tree/main/arc-agi-3-agent
- The PR, which is a great writeup on its own: https://github.com/strands-labs/benchmark-harnesses/pull/12
- Scorecard: https://arcprize.org/scorecards/8a10b024-3560-448f-ac31-becc48affe5b
- PRO-LONG: https://github.com/alexisfox7/PRO-LONG
- ARC-AGI-3: https://arcprize.org/arc-agi/3
Top comments (2)
The 30% → 99.95% gap on the same weights is the number I'd tattoo on every "which model is best" thread. It reframes the whole eval conversation: a bare-model benchmark is measuring the harness's absence, not the model's ceiling.
What I'd love to see teased apart from these runs is where the 8 hours and ~$830 actually went. In our own long-horizon work the cost curve is rarely uniform — it's a few environments where the agent thrashes on a bad hypothesis and burns tokens re-deriving state it already had. The harness wins usually come from two unglamorous places: durable state that survives across attempts, and cheap "did that action actually change the world?" feedback so a failed experiment gets abandoned fast instead of rationalized.
Did the Strands run keep per-environment token accounting? Curious whether the efficiency score is evenly earned or dominated by a handful of environments where the state-management design paid for itself. That distribution would tell you which harness features to invest in next.
I was able to ask one of the engineers who worked on this to provide me with the per game cost basis and this is what he sent over:
The text is a little small but if you zoom in you can see the per game cost!