How Memory Changes a Visual Agent: A Reproducible Pokémon FireRed Test | Agent Lab Journal
AL
Agent Lab Journal
Guides
Glossary
REPRODUCIBLE EVALUATION · VISUAL AGENTS
How Memory Changes a Visual Agent: A Reproducible Pokémon FireRed Test
Level: advanced
Reading time: 60 minutes
Result: a protocol for comparing two models by progress, total cost, action count, and memory errors in an identical game environment
A cheap action is not an efficient action. If a visual agent forgets that it already collected Oak’s Parcel, walks the same corridor five times, reopens menus for information it saw thirty seconds ago, or repeatedly attempts an impossible transition, its low per-call price conceals expensive stagnation. The useful unit of comparison is not cost per action but verified progress per unit of cost under the same controlled episode.
Evidence boundary. This article provides a complete experimental protocol, not fabricated benchmark numbers. No model is declared the winner. You must populate the result tables from your own emulator recordings, model invoices, action logs, and human annotations.
The question the experiment answers
A visual agent observes screenshots, reasons about the visible state, and chooses controller inputs. In a game, its apparent intelligence depends on more than image recognition. It must preserve relevant facts across many decisions: where it has been, which objective is active, what an NPC said, which menu state is open, and whether the previous action changed the world.
This protocol compares two model configurations on one bounded Pokémon FireRed mission. It answers:
Which model reaches more verified milestones within the same action and cost budgets?
How much does each verified milestone cost?
How many controller actions and model decisions are required?
How often does each agent forget, contradict, duplicate, or corrupt useful state?
How much activity is productive, and how much is repeated navigation or recovery?
The protocol does not ask which model “plays Pokémon better” in general. That claim would require many tasks, maps, battle conditions, and seeds. The narrower target is a controlled comparison of memory-dependent progress in the opening section of one game.
Why cost per action is misleading
Suppose Model A costs less per inference than Model B. Model A may still be more expensive per milestone if it needs more observations, produces invalid commands, revisits completed rooms, or loses the current objective. The central quantity is therefore:
cost_per_verified_milestone =
total_episode_cost / verified_milestones_completed
actions_per_verified_milestone =
accepted_controller_actions / verified_milestones_completed
productive_action_rate =
productive_actions / accepted_controller_actions
progress_efficiency =
weighted_progress_points / total_episode_cost
An action is one accepted emulator input from the allowed action set. A model decision may emit one action or a bounded sequence. Keep those quantities separate. A ten-button macro is one inference but ten actions, and treating it as one action unfairly rewards batching.
A state is the information needed to distinguish the current situation from materially different situations: map and coordinates when available, screen type, menu depth, inventory flags, objective flags, battle state, and recent transitions. A screenshot is only an observation of that state. It may be ambiguous, partially occluded, or unchanged even when an internal flag changed.
The concrete case: Oak’s Parcel loop
Use a legally obtained Pokémon FireRed ROM dumped from media you own, subject to the law where you live. Do not publish the ROM, save files derived from copyrighted game data, or bundled game assets. Record the exact ROM SHA-256 privately so every run uses identical bytes.
The recommended episode begins from a canonical save state in the player’s bedroom before the first movement input. The target is to complete this ordered sequence:
Leave the bedroom and the player’s house.
Trigger Professor Oak’s interruption north of Pallet Town.
Reach the laboratory and obtain the starter Pokémon.
Complete the required rival battle.
Reach Viridian City and receive Oak’s Parcel from the Poké Mart clerk.
Return to Professor Oak and deliver the parcel.
Receive the Pokédex.
This case is useful because it combines navigation, dialogue, menu handling, a battle, a distant objective, and a return route through previously visited space. The agent must distinguish “I saw the Mart” from “I received the parcel” and “I possess the parcel” from “I already delivered it.” Those are precisely the distinctions that weak memory tends to collapse.
Starter choice is a confounder because it changes the rival battle. Fix one starter in the manifest. Do not let one model choose freely while forcing the other. Likewise, fix text speed, battle style, emulator speed, frame skip, audio, language, and any action-repeat behavior.
Define the two conditions before running either
A fair test changes one primary variable at a time. There are two valid designs:
Design A: compare model products
Model A and Model B receive the same screenshot, system prompt, memory interface, action schema, budgets, and environment. This measures the complete models under an identical harness. It does not isolate memory architecture, because models may differ in vision, planning, latency, and instruction following.
Design B: isolate persistent memory
Use the same model twice. In the baseline condition, provide only a fixed recent window. In the memory condition, add a structured persistent store with identical read and write rules. This more directly estimates the effect of agent memory, although the memory machinery introduces extra tokens and operations.
Do not describe Design A as proof that one model has “better memory” unless you separately test perception and planning. The safe conclusion is that one complete configuration retained and used task-relevant state more effectively under this protocol.
Experimental controls
Variables that must remain identical
Control
Required value
Why it matters
ROM
Same private SHA-256
Different revisions can change behavior or memory addresses.
Initial save state
Same immutable file and SHA-256
Prevents different inventory, flags, position, or RNG history.
Emulator
Same binary, version, core, and configuration
Controls timing, frame stepping, and save-state behavior.
Viewport
Same native frame, crop, scale, and image encoding
Changes in pixels can alter visual recognition.
Input set
UP, DOWN, LEFT, RIGHT, A, B, START, SELECT, NOOP
Prevents hidden macros or model-specific tools.
Frames per action
Fixed press and release durations
Long holds can move multiple tiles or skip dialogue.
Prompt
Byte-identical except model-required wrappers
Prompt differences can act as external memory.
Budgets
Same action, decision, wall-time, and monetary limits
Every model must have the same opportunity to progress.
Retries
Same policy and maximum
Silent retries hide invalid outputs and additional cost.
Scoring
Frozen before model names are revealed to annotators
Reduces outcome-driven interpretation.
If a provider does not support a deterministic seed, record that fact. Game episodes can still be compared, but repeated trials become essential. A temperature of zero does not guarantee deterministic output across hosted infrastructure.
Build an immutable experiment manifest
Create one manifest for the whole comparison. Replace placeholders with real values. Never place API keys in this file.
experiment_id: firered-memory-v1
protocol_version: 1
created_utc: "YYYY-MM-DDTHH:MM:SSZ"
game:
title: "Pokemon FireRed"
rom_sha256: "REPLACE_WITH_PRIVATE_HASH"
initial_state_sha256: "REPLACE_WITH_HASH"
language: "English"
starter: "Bulbasaur"
text_speed: "fast"
battle_style: "shift"
emulator:
name: "REPLACE_WITH_EMULATOR"
version: "REPLACE_WITH_VERSION"
config_sha256: "REPLACE_WITH_HASH"
frame_width: 240
frame_height: 160
press_frames: 2
settle_frames: 8
speed_multiplier: 1
frame_skip: 0
agent:
allowed_actions:
- UP
- DOWN
- LEFT
- RIGHT
- A
- B
- START
- SELECT
- NOOP
max_actions_per_decision: 4
screenshot_every_decision: true
recent_action_window: 12
invalid_output_retries: 1
budgets:
accepted_actions: 2500
model_decisions: 1200
wall_time_minutes: 180
total_cost_usd: "SET_BEFORE_RUN"
trials:
per_condition: 5
order: "counterbalanced"
restore_initial_state_before_each_trial: true
conditions:
- id: model_a
model: "REPLACE_WITH_PINNED_MODEL_ID"
memory_policy: "shared-policy-v1"
- id: model_b
model: "REPLACE_WITH_PINNED_MODEL_ID"
memory_policy: "shared-policy-v1"
Hosted model aliases can change without warning. Record the provider’s exact model identifier and response metadata for every decision. If only a floating alias is available, state that limitation and complete both conditions in the shortest practical time window.
Use a constrained action contract
Free-form prose makes parsing failures difficult to distinguish from reasoning failures. Require structured output and reject commands outside the schema.
{
"reasoning_summary": "short operational explanation",
"observed": {
"screen": "overworld|dialogue|menu|battle|transition|unknown",
"location_guess": "string or unknown",
"objective_guess": "string",
"certainty": 0.0
},
"memory_read": ["fact_id"],
"memory_write": [
{
"kind": "objective|location|event|inventory|warning",
"value": "atomic fact",
"confidence": 0.0,
"evidence": "visible cue or transition"
}
],
"actions": ["UP", "A"]
}
The harness should validate types, enum values, sequence length, and numeric ranges. Log an invalid response before retrying it. A repaired response still consumes latency and money, so include both attempts in total cost.
Do not expose internal emulator state to the agent if the tested interface is visual-only. Internal state may be used by the evaluator as a hidden oracle, but not placed in the prompt.
Memory policy
Use two explicit layers. Working memory contains the latest screenshot, recent actions, recent observations, and current objective. Episodic memory contains durable events such as “received Oak’s Parcel at the Viridian Mart” with evidence and a decision index.
A practical record looks like this:
{
"fact_id": "event-0042",
"kind": "inventory",
"subject": "player",
"predicate": "has_item",
"object": "Oak's Parcel",
"status": "active",
"confidence": 0.96,
"first_seen_decision": 318,
"last_confirmed_decision": 321,
"evidence_frame_sha256": "FRAME_HASH",
"supersedes": null
}
Facts must be atomic and revisable. After delivery, do not merely append “parcel delivered” while leaving “has parcel” active. Mark the possession fact as superseded. Otherwise the memory store itself creates contradictions.
Apply identical rules to both models:
Only facts supported by a visible cue or verified transition may be written as confirmed.
Guesses must carry lower confidence and must not silently become facts.
Completed objectives must be marked complete, not deleted.
Contradictory active facts must be returned together so the agent can resolve them.
Memory retrieval must use the same maximum number of records and token budget.
Every read, write, update, and eviction must be logged.
Ground truth without leaking it to the agent
The evaluator needs stronger evidence than the agent receives. Use three synchronized sources:
Lossless screenshots or a video of every decision boundary.
The accepted input stream with frame counts.
A hidden state probe, when technically and legally appropriate, or manual milestone annotations from the recording.
A checkpoint is a verifiable milestone, not a model’s claim that something happened. Define checkpoints before testing:
Suggested milestone rubric
ID
Milestone
Points
Verification
M1
Exited the player’s house
1
Recorded overworld frame in Pallet Town
M2
Oak interruption triggered
1
Dialogue and forced transition visible
M3
Starter obtained
2
Selection dialogue plus party confirmation
M4
Rival battle completed
2
Battle exit and post-battle state visible
M5
Viridian City reached
2
Location transition confirmed
M6
Oak’s Parcel received
3
Clerk dialogue and inventory or flag confirmation
M7
Parcel delivered
4
Oak dialogue and parcel no longer active
M8
Pokédex received
5
Dialogue plus menu or hidden flag confirmation
A milestone scores once. Re-entering Viridian City does not earn M5 again. Weighted points express task depth, while the ordered milestone count remains the easiest result to interpret. Report both; do not tune weights after seeing results.
Define memory errors operationally
A memory error is not every bad action. Walking into a wall may be a perception or control error. Count a memory error only when logged evidence shows that retained state was absent, wrong, contradictory, or ignored.
Omission
A previously verified, still-relevant fact is unavailable when required. Example: the agent asks what the current delivery objective is after receiving the parcel.
False persistence
An obsolete fact remains active. Example: memory still says the parcel is held after verified delivery.
Fabrication
The agent stores or asserts an event that never occurred. Example: it records that the Pokédex was received before returning to Oak.
Contradiction
Two incompatible facts are simultaneously treated as true. Example: “parcel not collected” and “return to Oak with parcel.”
Duplicate objective
A completed objective is restarted because completion was not retained.
Context substitution
A fact from another location, menu, or trial is applied to the current state.
Memory-use failure
The correct fact was retrieved but the selected plan contradicts it.
Unsupported overwrite
A correct memory is replaced by a lower-confidence guess without new evidence.
Record severity separately:
Minor: wastes at most five accepted actions and causes no milestone regression.
Material: wastes more than five actions, causes a repeated route, or delays a milestone.
Critical: makes the episode unrecoverable within budget or corrupts the experiment state.
Keep raw counts and normalized rates:
memory_error_rate =
confirmed_memory_errors / model_decisions * 100
material_memory_error_rate =
material_or_critical_errors / model_decisions * 100
memory_recovery_cost =
sum(actions_between_error_and_recovery)
repeat_overhead =
repeated_navigation_actions / accepted_controller_actions
Detect repeated routes
Route repetition is the clearest bridge between memory quality and cost. If hidden coordinates are available to the evaluator, construct a transition key:
transition = (
map_id_before,
tile_x_before,
tile_y_before,
action,
map_id_after,
tile_x_after,
tile_y_after
)
Count a repeated route segment when the same ordered sequence of at least four transitions occurs again without a new milestone, inventory change, battle result, or justified recovery event between occurrences. Exclude required backtracking: the return from Viridian City to Pallet Town is part of the task. It becomes wasteful repetition only when the agent traverses a segment without advancing the active objective.
If coordinates are unavailable, annotate rooms and landmarks from frames. This is less precise, so report inter-annotator agreement and preserve disputed cases rather than forcing certainty.
Logging schema
Write one append-only trace record per model decision. JSON Lines is convenient because a partial run remains readable.
{
"experiment_id": "firered-memory-v1",
"condition_id": "model_a",
"trial_id": 1,
"decision_id": 318,
"timestamp_utc": "YYYY-MM-DDTHH:MM:SS.sssZ",
"frame_sha256": "HASH",
"prompt_sha256": "HASH",
"model_id": "PINNED_ID",
"response_id": "PROVIDER_RESPONSE_ID_IF_AVAILABLE",
"input_tokens": null,
"output_tokens": null,
"cached_tokens": null,
"provider_cost_usd": null,
"latency_ms": null,
"raw_response_path": "responses/000318.json",
"parsed_actions": ["UP", "UP"],
"accepted_actions": 2,
"invalid_output": false,
"memory_reads": ["event-0042"],
"memory_writes": [],
"milestones_before": ["M1", "M2", "M3", "M4", "M5", "M6"],
"milestones_after": ["M1", "M2", "M3", "M4", "M5", "M6"],
"termination_reason": null
}
Use null for unavailable measurements; never substitute zero. Zero means measured and absent. Capture provider-reported usage rather than estimating tokens when possible. If the provider does not expose monetary cost, calculate it from a separately saved price schedule and label the value “derived.”
Directory layout and integrity checks
firered-memory-v1/
├── manifest.yaml
├── prompts/
│ ├── system.txt
│ └── action-schema.json
├── conditions/
│ ├── model-a.yaml
│ └── model-b.yaml
├── trials/
│ ├── model-a-01/
│ │ ├── trace.jsonl
│ │ ├── frames/
│ │ ├── responses/
│ │ ├── memory.jsonl
│ │ └── annotations.jsonl
│ └── model-b-01/
├── scoring/
│ ├── milestones.yaml
│ └── memory-error-rubric.yaml
└── SHA256SUMS
After each trial, hash the evidence package:
find prompts conditions trials scoring \
-type f -print0 |
sort -z |
xargs -0 sha256sum > SHA256SUMS
sha256sum --check SHA256SUMS
Do not include SHA256SUMS itself in its input set. Run the verification command before analysis and again before sharing results.
Run procedure
Freeze the protocol. Hash the manifest, prompts, schema, milestone definitions, and error rubric.
Validate restoration. Restore the initial state twice and confirm identical first-frame hashes and evaluator state.
Dry-run the harness. Use a scripted policy, not either tested model, to verify input timing, frame capture, logging, and termination.
Check isolation. Confirm that no memory records, conversation IDs, caches, or provider sessions cross trial boundaries.
Assign blinded labels. Map models to Condition X and Condition Y in a file unavailable to annotators.
Counterbalance order. Run X, Y, Y, X, X, Y, Y, X, X, Y for five trials per condition, or use another predeclared balanced order.
Restore the canonical state. Verify its hash before every trial.
Start all budgets together. Count inference retries, parsing repairs, memory calls, and controller actions from the first decision.
Terminate mechanically. Stop at M8 or when any declared budget is exhausted, the harness fails, or the emulator diverges.
Annotate independently. Have two reviewers label milestones, repeated routes, and suspected memory errors without knowing model identity.
Resolve disagreements. Preserve both original labels and record the adjudication reason.
Unblind after scoring. Compute summaries only after the trial-level annotations are frozen.
Budget termination rules
A trial ends at the first matching condition:
M8 is verified;
accepted action budget is exhausted;
model decision budget is exhausted;
wall-time budget is exhausted;
monetary budget is exhausted;
the emulator crashes or produces a state that cannot be restored;
the agent enters an unrecoverable loop defined as the same route segment repeated three times without new progress.
Do not grant extra actions because a model was “almost there.” Report the terminal reason. A harness failure should normally invalidate the paired trial and trigger a rerun of both conditions from the same schedule position; a model-generated invalid command remains part of that model’s result.
Calculate the result
Keep trial-level data. Do not pool all actions into one giant episode. For each trial report:
deepest ordered milestone and total weighted progress;
total provider cost, including retries and memory operations;
accepted controller actions and model decisions;
wall time and model latency;
invalid outputs and rejected actions;
confirmed memory errors by type and severity;
repeated navigation actions;
-
termination reason.
Trial-level result template Condition Trial Deepest milestone Progress points Cost Actions Decisions Memory errors Repeat overhead End reason Model A 1 Not measured Not measured Not measured Not measured Not measured Not measured Not measured Not measured Model B 1 Not measured Not measured Not measured Not measured Not measured Not measured Not measured Not measured
For each condition, publish the median and full range across trials. With only five trials, avoid strong significance claims. Show every trial and emphasize effect size: additional milestones reached, actions saved per milestone, cost saved per point, and change in material memory-error rate.
summary = {
"completion_rate": completed_trials / valid_trials,
"median_progress_points": median(progress_points),
"median_cost_usd": median(total_cost_usd),
"median_actions": median(accepted_actions),
"median_cost_per_point": median(
total_cost_usd / progress_points
for trials where progress_points > 0
),
"median_repeat_overhead": median(
repeated_navigation_actions / accepted_actions
)
}
Do not assign infinite cost-per-point values to zero-progress trials and then silently discard them. Report zero-progress count separately. Completion rate and deepest milestone preserve information that ratios cannot.
Verification checklist
Before accepting the comparison, verify all of the following:
The ROM, initial save state, emulator configuration, prompts, schemas, and budgets have matching hashes across conditions.
The first captured frame is identical in every valid trial.
No trial begins with memory, conversation, or cache state from a previous trial.
Every accepted controller action appears in both the trace and emulator input log.
Every model call—including retries—appears in cost and latency totals.
Every milestone has frame, video, or hidden-oracle evidence.
Every confirmed memory error cites an earlier fact and the later contradictory decision.
Required backtracking is excluded from repeated-route penalties.
Annotators were blinded until labels were frozen.
Raw trial rows reconcile with summary statistics.
sha256sum --check SHA256SUMS succeeds.
Failure cases and how to classify them
The agent walks into a wall repeatedly
If it misread collision geometry, classify the primary cause as perception. If it correctly recorded the obstruction and later repeats the same plan without new evidence, add a memory-use failure. One event may have a primary cause and a secondary memory consequence, but it must not be counted twice in the same aggregate category.
The agent forgets whether dialogue finished
Check whether the last frame was captured after sufficient settle frames. If the harness sampled mid-transition, this is an instrumentation failure. If a stable post-dialogue frame was supplied and the event was previously verified, it may be an omission.
The model emits a long controller macro
Reject sequences beyond the declared maximum. Log the invalid output, charge the call, and apply the same single retry policy. Never truncate silently because truncation creates an unrequested action policy.
The agent saves and reloads the game
Disallow emulator save/load controls entirely. If START is allowed for the in-game menu, distinguish it from emulator hotkeys. The agent must not create private rollback branches.
The rival battle diverges
Battle randomness and model choices can change health and timing. This is part of the episode unless the emulator or initial state differs. Report battle actions separately so navigation efficiency is not obscured.
A provider call times out
Apply the predeclared retry policy and include timeout latency and any billed usage. If the provider returns no usage record, mark cost unknown rather than zero. Repeated infrastructure failures should be reported separately from agent quality.
The model updates memory correctly but ignores it
This is not successful memory behavior. Record the retrieval and classify the contradictory plan as a memory-use failure. Storage quality and use quality should be reported independently.
The hidden state probe disagrees with the screen
Pause scoring and inspect emulator timing. A probe taken before an action and a screenshot taken after it are not synchronized evidence. Associate both with a shared frame index before using either as ground truth.
Interpretation
A useful conclusion is specific:
Under the frozen FireRed opening-task protocol, one condition reached a deeper median milestone with fewer accepted actions and fewer material memory errors, while total cost was measured from all logged calls.
A weak conclusion overreaches:
Model A has better long-term memory and is the best game-playing agent.
The experiment supports the first form only after real measurements are inserted. Even then, inspect causal patterns. Lower cost may come from shorter outputs rather than better memory. Fewer actions may come from riskier macros. Higher progress may come from stronger visual recognition. The trace should show whether memory errors actually precede repeated routes and stalled objectives.
Limitations
One opening mission does not represent the full game, long battles, puzzle areas, item management, or strategy over many hours.
Pokémon FireRed contains stochastic events, so identical initial state and inputs do not make model-generated episodes identical.
Model comparison does not isolate memory from vision, planning, language understanding, or action-format compliance.
Hidden emulator state improves scoring but may depend on version-specific addresses and can be implemented incorrectly.
Human memory-error annotation requires judgment. Blinding and dual annotation reduce, but do not remove, subjectivity.
Provider pricing and model implementations can change. Preserve the price schedule and exact model identifiers used on the test date.
Persistent memory may add cost while improving progress. The correct trade-off depends on the application’s cost of delay and failure.
A fixed action budget favors concise control policies; a fixed wall-time budget also reflects provider latency. Publish both.
ROM possession and emulation laws differ by jurisdiction. This protocol does not provide legal advice or copyrighted game files.
What a complete experiment package contains
A reader should be able to audit the comparison without trusting the summary. Publish the protocol version, redacted manifest, emulator version, hashes that do not expose copyrighted content, prompts, action schema, memory policy, milestone rubric, error rubric, trace schema, analysis code, trial-level results, annotation disagreements, and integrity file. Do not publish the ROM, secrets, private provider payloads, or copyrighted assets.
The final report should answer four questions side by side: how far each condition progressed, what that progress cost, how many actions it required, and which memory failures created avoidable work. When those four quantities are visible, a cheap but forgetful agent can no longer look efficient merely because each individual call is inexpensive.
Final takeaway
Memory matters when it changes behavior over time. In this test, the evidence is not an eloquent self-report or a plausible plan. It is a verified parcel collected once, a completed objective not restarted, a return route taken for a reason, and a Pokédex reached within fixed budgets.
Freeze the environment, separate observations from ground truth, count controller inputs rather than only model calls, charge every retry, annotate memory errors against prior evidence, and publish trial-level results. That turns Pokémon FireRed from an entertaining demo into a reproducible test of agent progress efficiency.
← Explore practical guides
·
Open the Agent Lab glossary →
Agent Lab Journal
Real experiments. Verifiable conclusions.
Top comments (0)