Most of the work that makes an AI agent good never happens inside the model. It happens in the harness — the code that feeds the model its observations, defines its tools, trims its context, and decides what to do with each response. When an agent fails, the usual fix is a human editing that harness: rewording a tool description, adding a memory store, changing how a screenshot gets summarized. The Continual Harness work, from the teams behind Gemini Plays Pokémon and the PokeAgent benchmark, pushes on a sharper question — what if the model edited the harness itself, while the run was still going?
The harness is where agents actually live
Gemini Plays Pokémon was a public demonstration: a Gemini model worked through a Game Boy Pokémon title via a harness that turned the game into something a language model could reason about. The harness converted pixels into labeled screenshots, a map of the current area, and an inventory list, then exposed button presses and pathfinding helpers as tools. The model never touched raw emulator memory. It saw whatever the harness chose to show it, and it acted only through the tools the harness defined.
That structure is not specific to Pokémon. A coding agent doesn't see your repository — it sees the files a retrieval step pulled in. A browser agent doesn't see a webpage — it sees an accessibility tree some extraction code produced. The harness is the agent's entire sensory system, its motor system, and its memory. The model is one component inside it.
Which means most of the leverage in agent quality sits in the harness, not the weights. Teams running long agent tasks spend their time there: tightening tool descriptions, adding retry logic, changing how context gets summarized so the model stops losing the thread on long runs. That iteration is real engineering, and almost all of it happens offline — a human watches a failure, edits the scaffolding, and starts a fresh run.
When people say an agent "got smarter," they often mean the harness got better — the model checkpoint never moved. That is worth internalizing before you reach for a fine-tuning budget.
What "continual" changes
The Continual Harness pattern moves that improvement loop inside the run. The agent is given write access to parts of its own harness. When it hits a recurring failure — say it keeps walking into a ledge because the pathfinding helper doesn't model one-way tiles — it can propose a change to that helper, apply it, and continue with the improved tool in hand. The scaffolding at hour ten is not the scaffolding the run started with.
This is online adaptation, and it sits between two things developers already know. It is not fine-tuning: the model weights stay frozen. It is not ordinary in-context learning either, where the model only writes itself a note. The improvement lands as durable code — a function the agent rewrote — so it persists, it is inspectable, and it can be reverted. The model is playing the game and maintaining the controller at the same time.
The reason this matters beyond a Pokémon stream: the manual harness-tuning loop is a bottleneck. Every agent team has a backlog of "the tool description is slightly wrong" and "the memory step drops the wrong thing" fixes that a human has to notice, diagnose, and ship. An agent that can do a slice of that work itself, on the failures it is actually hitting, compresses that loop from days to minutes.
An agent with write access to its own harness can also break it. A bad edit to the navigation tool can strand the run; a bad edit to the context-trimming step can quietly degrade every later decision. This pattern is only safe with versioned edits, a fixed core loop the agent cannot touch, and automatic rollback when a change makes the feedback signal worse.
Borrowing the pattern for your own agents
You do not need a Game Boy emulator to use this. The pattern reduces to four decisions.
Separate the editable surface. Decide explicitly which parts of the harness the agent may rewrite — tool wrappers, prompt templates, retrieval filters — and which are permanently off-limits: the loop that calls the model, the kill switch, anything that touches credentials or external writes. The self-improving part should be a small, well-fenced area.
Treat every harness edit as a commit. A self-improvement is a diff. Give it a message, a test, and a revert path. If you cannot answer "what did the agent change, and how do I undo it," you do not have a continual harness — you have an agent slowly corrupting itself.
Give it a feedback signal it can act on. Pokémon has an obvious one: progress through the game. Your agent needs an equivalent — task success rate, an eval suite, a latency budget. Without a metric, the agent edits blind, and you cannot tell improvement from regression.
Start narrow. Let the agent tune tool descriptions and retry thresholds long before you let it rewrite tool implementations. Widen the editable surface only as the rollback machinery proves itself.
If you want to watch a constrained version of this loop before wiring it into an autonomous run, an AI-native code editor is the closest everyday analog: an agent proposes edits to real code, and you approve or reject each diff.
The Continual Harness result is not that an agent finished a Pokémon game. It is that the harness — long treated as fixed scaffolding a human owns — can be a live, model-editable surface. For anyone building agents that run for hours, that reframes where the next improvement comes from.
Originally published at pickuma.com. Subscribe to the RSS or follow @pickuma.bsky.social for new reviews.
Top comments (0)