Further Optimizing the Vision-Only Harness: the Notes Rule

A short follow-up to I replicated the vision-only Pokémon run Anthropic showcased on the Fable 5 launch page. Rereading the harness code, I found the notes rule was still doing design work on the model's behalf — so I changed it, and compared the effect across two runs, one on each side of the change. The code and every log quoted below are in the project repo.

Rereading the code: the notes rule was still constraining

The principle of this harness is thinness: each turn the model gets one screenshot plus its own notes, and its only tool is pressing buttons (1 turn = 1 screenshot + 1 decision; that's what turn means in the logs). Notes are the single non-visual mechanism allowed — a context window cannot hold thousands of turns, so something has to survive compression.

I had always thought of the notes as "the model's own thing," but going back through the code showed otherwise: in the system prompt and the tool description, I had prescribed a fixed checklist — record your location, your objective, your team, your lessons learned — plus "keep them compact" and "update whenever something important changes." That is a memory schema I designed, not one the model chose. For a project whose whole point is observing the model's capability, this too was scaffolding, hidden in the prompt's wording.

For the record: the official Fable 5 run in the launch-page article used the old rule — the frozen-protocol tag sits on exactly that version. The change described here came after, and alters none of that article's conclusions.

The change: three prompts, before and after

The change is commit f658c9d (the old version is the frozen-protocol commit 51343ca) and touches three passages in three files. Original text first, then the differences.

The Memory bullet in the system prompt (prompts.py)

Before:

Memory:
- Use update_notes to maintain a compact running memory (location, objective,
  team, key learnings). Your conversation history is periodically compressed
  away; your notes are the only thing guaranteed to persist. Update them
  whenever something important changes.

After:

Memory:
- Notes are your own long-term notebook, shown back to you every turn. What
  they contain and how they are organized is your call (see update_notes).

The update_notes tool description (tools.py)

Before:

Overwrite your entire notes scratchpad with new text. This is your ONLY
long-term memory: older conversation history is periodically discarded, but
notes survive. Keep them compact: current location, current objective, team
status, and key learnings (what worked, what failed).

After:

Overwrite your notes with a new full version. Notes are your long-term
notebook: conversation history is periodically summarized away, but notes
persist - and anything you leave out of a new version is forgotten. What to
record, how to format it, and when to rewrite, extend, or restructure it are
entirely your call: whatever best helps you beat the game. Keep the whole
notebook within about 1,000 tokens.

The compression-time handoff prompt (previously inline in agent.py, now SUMMARY_PROMPT in prompts.py)

Before:

Pause for a moment. Write a concise progress summary: where you are, what you
were doing, what worked and what didn't, and your immediate next objective.
Reply with the summary text only - do not call any tool this turn.

After:

Pause. Your conversation history is about to be deleted and replaced by
whatever you write now (notes are kept separately). Write the handoff your
future self needs to continue playing seamlessly - what to include is your
call. Keep it concise. Reply with text only - do not call any tool this turn.

The difference is plain from the text: the old versions are full of my checklists (what to record, how, and when), and the new ones delete all of that, keeping only one physical rule (whatever you leave out is gone forever) and one budget (about 1,000 tokens), handing format, content, and cadence back to the model; the handoff prompt likewise goes from "summarize along my outline" to "your history is about to be replaced by whatever you write — the handoff is yours to design." In one sentence: a bounded blank page. Two lines are pre-printed on it — whatever you leave off disappears forever, and this is all the paper you get — the rest of the blank space is the model's.

Old vs. new: the same model, two notebook lives

The comparison material is Kimi K3 (moonshot/kimi-k3) on the same Chinese fan-translated FireRed ROM, under the same 2,000-turn cap, one day apart: the 07-24 run used the old rule, the 07-25 run the new one. Caveats first: different starters (Bulbasaur vs. Charmander), random encounters, and it's N=1 against N=1 — I believe the differences below come mainly from the rule change, but strictly speaking they can't be fully attributed to it.

The numbers first. Under the old rule it wrote 160 notes updates, averaging ~700 characters per version, and the final version had shrunk to 493 characters — thinner and thinner. Under the new rule it wrote only 69 updates, averaging ~2,000 characters, with a final version of 2,054 — rewrite frequency halved, per-version size tripled, direction reversed. The cause isn't hard to guess: the old rule encouraged updating "whenever something important changes," so it made frequent small edits, and every edit is a transcription, and every transcription can drop something; the new rule says outright that anything left out is forgotten, which turned rewriting into a deliberate act.

A notes version under the old rule (07-24 21:31, verbatim)

Chinese FireRed/LeafGreen. Player=RED. Bulbasaur Lv8, FULLY HEALED (22/22). Pokedex + 5 Poke Balls.
LOCATION: Tall grass at NORTH edge of Pallet Town (Route 1 entrance area). Wild Pidgey Lv3 encounter IN PROGRESS.
GOAL: Catch this Pidgey (first catch). Then continue to Viridian City Mart (Oak's Parcel). Then catch more on Routes 1/22/2.
KEY LESSON: Screenshots lag 1-2 steps; move slowly, verify each screen.

Four lines, and they are exactly the four items on my checklist: status, location, objective, lesson. Nearly all 160 versions in that run follow this template. Note the "Viridian City Mart (Oak's Parcel)" in the GOAL line — by the time this version was written, the parcel had been delivered half an hour earlier. That stale objective is the seed of the 470-turn detour below.

A notes version under the new rule (07-26 00:01, excerpted; full version ~2,000 characters)

RIGHT NOW: On ROUTE 1, at the SOUTH edge of the big tall-grass block [...]
MISSION: Backtrack NORTH through Route 1 → Viridian City → RE-TEST north exit
  (Route 2 → Viridian Forest → Pewter 尼比市). [...]
ROUTE 1 GEOMETRY (corrected this session): [...] The tile directly NORTH of
  the grass block's west column is a SOLID BUSH strip (my many UP presses
  there did nothing while DOWN worked — inputs fine, obstacle real). [...]
INPUT BUG: After closing dialog/sign, next dir presses often eaten [...]
PALLET = DEAD END (confirmed): lab locked, water south, rival's house just
  a sign. Route 1 south leads ONLY to Pallet.
VIRIDIAN REFRESHER: [...] PC door faces SOUTH; signboard front-left of door
  (sidestep right to door column, then up). [...]
PARTY: Charmander (小火龙 ♀) Lv13, HP 37/37 FULL. Moves: 利爪(Scratch TL)/
  嚎叫(Growl TR)/火花(Ember BL 25/25)/合金爪(MetalClaw BR Steel — SAVE for
  Brock). NO Potions. HP<15 → 逃跑(Run).
BATTLE FLOW: A through intro → "小火龙！就决定是你了！" → A ONCE → menu →
  A (利爪 TL) [...] NEVER queue dir+A while text prints.
BAG: 5×精灵球, 城镇地图, 教学电视机, 1×解毒药.
KEY CHINESE: 常磐市=Viridian 尼比市=Pewter 真新镇=Pallet [...]
  利爪=Scratch [...] 合金爪=MetalClaw [...]
GOTCHAS: (1) Dir presses while text open ADVANCE text. (2) Signs/NPCs:
  adjacent+face+A once. (3) First press new dir may only TURN — re-press.
  (4) SHORT queues(2-5), verify EVERY turn; never repeat blocked dir >2×.
  (5) Ledges hop DOWN only. [...]

A dozen sections, not one of which came from my prompt: a rolling RIGHT NOW, a standing MISSION, map knowledge filed by place name, a reusable battle routine, a self-built Chinese vocabulary table, a list of gotchas. Finished business gets nailed up as a section header (PALLET = DEAD END (confirmed)). Note the INPUT BUG section too — it files its wrong conclusions just as neatly, which foreshadows "the disease it didn't cure" below.

Dropping things is not a theoretical risk — the old-rule run died of it. To replay where that stale objective came from: at 21:02 it delivered Oak's Parcel and wrote Parcel DELIVERED into its notes; a rewrite 16 minutes later lost that line and copied a stale objective back in — get Oak's Parcel quest resurrected as a to-do. Once compression pushed the real events out of history, the notes were the only memory left, and the notes said the quest was still open. It walked back to Pallet Town to redo a finished quest, looped for roughly 470 turns, and got its party wiped by a Lv5 Pidgey along the way.

The new-rule run reached the same story beat and behaved completely differently: the moment it got the parcel it marked parcel already obtained here on its map notes, and after delivery the goal chain advanced to "Pokédex next"; later, after one mistaken detour to Pallet, it nailed down PALLET = DEAD END (confirmed) and never went back. Complete an item, cross it off — nobody taught it that discipline. "Anything left out is forgotten" forced it into existence.

It also began grading its knowledge. Map claims get tagged (verified), walkthrough recall gets suffixed with trust screen over memory, dubious leads get marked UNVERIFIED LEAD. When it was stuck it wrote a line I'd like to frame:

FEELING STUCK — need to re-verify the map from scratch rather than trust old notes.

The blank page gave it room to design a memory system, and it actually designed one.

The cost, and the disease it didn't cure

Two honest entries. First, a thicker notebook means more input tokens every turn: the bill went from $16.60 to $22.85 between the two runs, about 38%. Second, the blank page cured "copying things out" but not "writing wrong things in": the new-rule run still wrote a hallucinated mechanic into its notes (treating roadside cone trees as "climbable steps") and stamped it hard-won, don't re-explore blindly, after which no on-screen evidence could talk it down. The channel for errors entering the notes got narrower — and the errors that did get in became harder to dislodge. That is the next thing to watch, not something this change could fix.

The blank page became an observable

The most interesting output of this optimization is that the notes format moved from "input I prescribed" to "part of the model's output." Hand the same blank page to different models and they will write different memory systems — how they partition it, what counts as verified, how often they rewrite, what they discard and what they keep. Those choices project capability just as directly as button sequences do. Next time I start a run with a new model, the first thing I'll read is its notebook.